Groups   71 of 99+      julia-users › Optimizing embarrassingly parallel computation with SharedArrays and custom network topology 1 post by 1 author      johnhold... gmail.com    Apr 19   I have an embarrassingly parallel problem with the following features: I need to run a pure function on different inputs a lot of times I need to record the results of  every  call to this function I need to pass a lot of information as input to this function, and in particular the result needs to match to the particular input. But suppose temporarily that all the inputs are fixed except just for say just a number indexing which computation I'm doing  see example below . I have successfully run examples of solving these problems on a single machine and on a multi-machine cluster with passwordless SSH using three distinct approaches.  Apologies for lack of MWEs here; I will add if necessary, but should be simple enough and applicable to any example where pure_function below is doing a very small amount of work.   In the single machine case, use a SharedArray, and do   sync  parallel for i in 1:N     my_shared_array i    pure_function i,  other fixed inputs   end  In the multi machine case, I can't use a SharedArray. So I can do essentially the identical thing with a DistributedArray, or Essentially the same thing where instead of using  parallel for, I just use pmap.  Also another version where I essentially tried manually to recreate  parallel for by sending remote calls, waiting for workers to finish, and then feeding them the next task, but this didn't outperform the others and anyway I'd prefer to stick to the realm of canonical    safe  approaches.  When I time these  I'll provide specific benchmarks if necessary  on my local machine ONLY,  1 is way faster than either  2 or  3. The slowness of  3 in particular is consistent with the documentation, since pure_function isn't doing much work--I have relatively a high number of calls, and relatively a small amount of work being done in each call--so  parallel should be the better solution than pmap.   Interlude--I've also tried another approach, say  1A, where I don't use a SharedArray at all, but just use  vcat  as the reduction for  parallel for, as suggested e.g. here. But this is of course slower than all of these approaches when the array is long enough, IIUC b c calling reduce on vcat is doing a lot of unnecessary traversals instead of just  smartly  aggregating within processes and then pasting together the completed results of the processes. BTW, I've also tried to write a modified  parallel that doesn't try to reduce but just collects the values, but I failed to get it to work.  Benchmarking these results is slighty complicated by my uncertainty over how much of the message-passing overhead is due to passing the inputs to the processes versus how much is due to the method of allocating the work to available workers versus in turn the overhead of collecting all the results.   Based on  a  comparing the speed on my local machine of approach  1 to approaches  2  3, I believe SharedArrays are ideal for work on a single machine.  Based on  b  comparing the speed of  2    3 on my local machine only to the speed of  2  3 when I have multiple machines, using both the all_to_all and master_slave topology, it seems that there is a lot of additional overhead in sending information across the network to each of the processes on the remote  machines . Note I don't have independent evidence of this; perhaps Julia is already  cleverly  in the background deciding that it only needs to communicate across the network once per machine rather than once per process. But IIUC this is not so, and either way the overhead of communicating with additional processes on remote machines is so far so high that I basically get no gain from adding them.   Consider these observations, it seems that the ideal approach would look something like this:  Setup a custom network topology where the master process has connections  only  to single  lieutenant  processes on worker machines, and  Per-machine, the lieutenant process  including the original master process  creates a SharedArray, adds  local  processes, and distributes work to all the  local-only  nodes Therefore the initial inputs and the per-machine SharedArrays are the only entities passed across the network  minimize network communication, which IIUC is the slowest inter-process communication , and then on a given machine as little as possible is passed between processes.  My questions are:  Is this really the ideal approach? Or is there a better way? If that really is the ideal approach, how can I do it? I've tried tinkering with the  spawn lieutenant-worker processes on other machines and then make them spawn their own worker processes  idea, but I get errors because the remote lieutenants aren't the master process and don't seem to be  allowed  to spawn more workers; this suggested to me that I must be thinking about the problem the wrong way, since I probably shouldn't have to modify functions in Base in order to get this to work. So  A  literally how can I do it, and  B  is there a way to do it without digging too deep into the implementation, but just solving the problem generically   at a high level? Has no one else encountered this particular issue? I've seen so many excellent examples of distributed computing in these forums and elsewhere, yet they all stick either with distributed arrays for multi-machine work or SharedArrays for local work, i.e. thinking either about parallelizing across some local processes or setting up a network of machines and parallelizing across a bunch of possibly remote processes, but nothing directly tackling the issue of discriminating among worker processes depending on whether they're local or on a remote machine.   Addendum...here is a subset of things I've read on the topic, many of which have proved useful food for thought, despite comment  3 above.   http:  docs.julialang.org en latest manual parallel-computing   https:  github.com JuliaLang julia pull 11665 https:  github.com JuliaLang julia issues 3655  https:  github.com proflage 2015-julia-hands-on  https:  groups.google.com d topic julia-users 1sNXYtIbS1Q discussion https:  groups.google.com d topic julia-users E8fGIiDwckc discussion https:  groups.google.com d topic julia-users W5QIfE7f0O4 discussion https:  groups.google.com d topic julia-users WJBIAYzrZgg discussion https:  groups.google.com d topic julia-users b4tEzOOOnJI discussion https:  groups.google.com d topic julia-users fMTwlMJKNVw discussion https:  groups.google.com d topic julia-users fe1yZawvvi0 discussion https:  groups.google.com d topic julia-users u4tJSCX4jxw discussion https:  groups.google.com d topic julia-users vUpyqIxz3Hw discussion https:  groups.google.com d topic julia-users vt2hS9h36a0 discussion