multithreading - pulling out from a text file and do some process in the minimal time Python -


i have text file billion lines. in each line there integer. try pull them out , make processing takes long time - need push items read queue , make process maximum threads cpu can handle. in addition, before script starts, there user input tells how integers want read file in 1 second. example: if user wants read 1000 lines(integers) per second, want sleeping (1/1000 second - visulaizing delay in reading outer file disk). tried solve through next example found here on 1 of questions no success: (i work pyscripter)

def worker(): while true:     item = q.get()     do_work(item)     q.task_done()  q = queue() in range(num_worker_threads):  t = thread(target=worker)  t.daemon = true  t.start()  item in source(): q.put(item)  q.join()       # block until tasks done 


Comments

Popular posts from this blog

How has firefox/gecko HTML+CSS rendering changed in version 38? -

javascript - Complex json ng-repeat -

jquery - Cloning of rows and columns from the old table into the new with colSpan and rowSpan -