python - Generate large file and send it -

i have rather large .csv file (up 1 million lines) want generate , send when browser requests it.

the current code have (except don't generate same data):

class csvhandler(tornado.web.requesthandler):    def get(self):     self.set_header('content-type','text/csv')     self.set_header('content-disposition','attachement; filename=dump.csv')       self.write('linenumber,measure\r\n') # file header     line in range(0,1000000):        self.write(','.join([str(line),random.random()])+'\r\n') # mock data  app = tornado.web.application([(r"/csv",csvhandler)]) app.listen(8080) 

the problems have method above are:

• the web browser doesn't directly start downloading chunks sent. hangs while webserver seems prepare whole content.
• the web server blocked while processes request , makes other clients hang.

by default, data buffered in memory until end of request can replaced error page if exception occurs. send response incrementally, handler must asynchronous (so can interleaved both writing of response , other requests on ioloop) , use requesthandler.flush() method.

note "being asynchronous" not same "using @tornado.web.asynchronous decorator"; in case recommend using @tornado.gen.coroutine instead of @asynchronous. allows use yield operator every flush:

class csvhandler(tornado.web.requesthandler):      @tornado.gen.coroutine     def get(self):         self.set_header('content-type','text/csv')         self.set_header('content-disposition','attachment; filename=dump.csv')           self.write('linenumber,measure\r\n') # file header         line in range(0,1000000):              self.write(','.join([str(line),random.random()])+'\r\n') # mock data             yield self.flush() 

self.flush() starts process of writing data network, , yield waits until data has reached kernel. lets other handlers run , helps manage memory consumption (by limiting how far ahead of client's download speed can get). flushing after every line of csv file little expensive, may want flush after every 100 or 1000 lines.

note if there exception once download has started, there no way show error page client; can cut download off partway through. try validate request , fail before first call flush().