multithreading - pulling out from a text file and do some process in the minimal time Python -
i have text file billion lines. in each line there integer. try pull them out , make processing takes long time - need push items read queue , make process maximum threads cpu can handle. in addition, before script starts, there user input tells how integers want read file in 1 second. example: if user wants read 1000 lines(integers) per second, want sleeping (1/1000 second - visulaizing delay in reading outer file disk). tried solve through next example found here on 1 of questions no success: (i work pyscripter)
def worker(): while true: item = q.get() do_work(item) q.task_done() q = queue() in range(num_worker_threads): t = thread(target=worker) t.daemon = true t.start() item in source(): q.put(item) q.join() # block until tasks done
Comments
Post a Comment