Groups   3 of 99+      julia-users › What is the fastest way to perform 100k blocking IO operations in parallel? 15 posts by 6 authors      Andrei Zh    8 22 15   I'm writing a kind of a web scanner that should retrieve and analyze about 100k URLs as fast as possible. Of course, it will take time anyway, but I'm looking for how to utilize my CPUs and network as much as possible.   My initial approach was to add all available processors, pack urls into tasks and run these tasks in parallel:       using Requests urls   ...  time  sync  parallel for url in urls     resp   get url      println  Status: $ resp.status    end  My assumption was that 100k tasks would be created, each task would execute GET request and, since this is IO operation, free current thread for other tasks. From logs, however, I see that each worker executes tasks one by one, every time waiting for GET request to finish.   So how do I start 100k requests in parallel?    100k is here just for example, I can easily split then into chunks of 10k, for example, so system limits and overused CPU network are not an issue; issue is in their underutilization .   Thanks      Andrei Zh    8 22 15   BTW, I'm not sure I use  parallel correctly, so I tried to start tasks manually with  async too:    time  sync for url in urls      async begin          resp   get url          println  Status: $ resp.status        end end  But I didn't notice any difference in performance.       Andrei Zh    8 23 15   More generally, is there anything like futures callbacks or async IO in Julia? I couldn't find anything, but maybe I just don't know the right keywords.      Steven G. Johnson    8 23 15    parallel in Julia is for executing separate parallel processes  true multi-tasking, with separate address spaces .   async is for  tasks , which are  green threads  and represent cooperative multitasking  within the same process and the same address space .  io in Julia is asynchronous — while one task is blocked waiting on io, another task will wake up and start running.   This is based on the libuv library, which is designed for high-performance asynchronous io.   The first question is whether you want to fetch URLs in separate OS processes, or you want to use green threads within the same process.  It sounds like you want the latter, in which case  async is the right thing.  The second question is whether something about the Requests-jl package is serializing things somehow; for that you might file an issue at Requests-jl.     Andrei Zh    8 23 15   Hi Steven,   thanks for your answer! It turns out I misunderstood  async long time ago, assuming it also makes a remote call to other processes and thus introduces true multi-tasking. So now I need to rethink my approach before going further.   Just to clarify: my goal is to perform as many requests as possible at the same time, so I want to use both - multiple processes  to start several requests at several cores in parallel  and tasks  to launch new requests while old ones are still waiting for IO to complete .   So I will update my approach and come back with results or new questions.       Jonathan Malmaud    8 24 15   As one of the maintainers of Requests-jl, I'm especially interested in its use for high-performance applications so don't hesitate to file an issue if it gives you any performance problems.    This message has been deleted.    Andrei Zh    8 24 15   Jonathan, thanks for your support. So far I noticed that DNS gives pretty large delay. E.g. resolving IP addresses for 1000 URLs took 80 seconds in serial code and 26 seconds in muli-task code:    Serial execution:   julia   time for url in urls                begin                    Base.getaddrinfo URI url .host                 end            end elapsed time: 80.071810293 seconds  732400 bytes allocated    Multitask execution:   julia   time  sync for url in urls             async begin                                                           Base.getaddrinfo URI url .host             end        end  elapsed time: 26.241893516 seconds  4277968 bytes allocated   So I'll try to pre-resolve IPs and test again.       Jonathan Malmaud    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  Thanks for the report, Andrei - Would you mind filing this is an issue at https:  github.com JuliaWeb Requests-jl?       Seth    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  Name resolution delays are generally an issue with network latency. Trying to resolve 1000 uncached names will take a while on any system:  seth schroeder: $ time host www.julialang.org www.julialang.org is an alias for julialang.github.io. julialang.github.io is an alias for github.map.fastly.net. github.map.fastly.net has address 23.235.47.133   real 0m3.268s user 0m0.003s sys 0m0.020s  Unless your server is localhost, you're going to have delays.      Uwe Fechner    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  Did you try the DNS servers from Google, e.g. 8.8.8.8 ? I never saw a reply that needs more than one second.  Well, in our university network.       Seth    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  Uni networks are typically high bandwidth and low latency and are optimally peered. My home network, unfortunately, exhibits none of those traits :   In any case, even a 70-millisecond resolution time results in 70 seconds of delay for 1000 uncached resolutions.      Jameson    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  If you are doing a lot of parallel dns queries, you may want to try increasing the number that can be run simultaneously but setting the UV_THREADPOOL_SIZE environment variable before starting julia to something larger  default is 4, max is 128 .      Andrei Zh    8 24 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?   Jameson: setting UV_THREADPOOL_SIZE to 32 seems to reduce DNS resolution time twice  from 26 to 12 seconds on my latest tests , so thank you.   However, DNS seems to be not the only root of the problem: I noticed that with large number of URLs first ones get result very quickly  300-800ms , but then latency begins to grow until timeout is exceeded  I'm seen more then 100 seconds for some requests .   I will try to set up stable test and post it here as well as to issues in Request-jl.       Andrei Zh    8 25 15  Re:  julia-users  Re: What is the fastest way to perform 100k blocking IO operations in parallel?  I submitted a new issue to Requests-jl describing the problem and my observations. In short, each separate request takes a reasonable time, but when I launch a lot of them in tasks, they become very slow.