Groups 6 of 99+ julia-users › Running multiple scripts in parallel Julia sessions 9 posts by 5 authors Ritchie Lee Jan 22 Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", ...., "script10.jl"] and I would like to run them in parallel in separate Julia sessions, but 4 at a time (since I only have 4 cores on my machine). Is there any way to do this programmatically? I tried doing this: addprocs(4) pmap(include, scripts) or addprocs(4) @parallel for s in scripts include(s) end However, this seems to reuse the same session, so all the global consts in the script file are colliding. I would like to make sure that the namespaces are completely separate. Thanks! Stefan Karpinski Jan 23 Any reason not to run them all as separate processes? - show quoted text - Ritchie Lee Jan 23 Do you mean using @spawn, success, or just from the command prompt? The scripts are long-running experiments where I am also tracking things like CPU time. I would like them queued and run 4 at a time, i.e., if 1 finishes ahead of others, then the next one will start running on the free processor. Is there a way to schedule into separate processes? - show quoted text - Tim Holy Jan 24 pmap? --Tim On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote: > Do you mean using @spawn, success, or just from the command prompt? > > The scripts are long-running experiments where I am also tracking things > like CPU time. I would like them queued and run 4 at a time, i.e., if 1 > finishes ahead of others, then the next one will start running on the free > processor. Is there a way to schedule into separate processes? > > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski wrote: > > Any reason not to run them all as separate processes? > > > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee > - show quoted text - Ritchie Lee Jan 24 The scripts contain a lot of global consts and other things. pmap seems to mix the namespaces for all scripts executed on the same processor. - show quoted text - Tim Holy Jan 24 You could write your own pmap-like function (see its source code) that starts a new worker and then shuts it down again when done. Or, just wrap all your scripts inside modules? module Script1 # code that was in script1.jl end --Tim - show quoted text - Kenta Sato Jan 24 I think GNU parallel is the best tool for that purpose. http://www.gnu.org/software/parallel/ You can pass -j option to control the number of maximum jobs at a time. michae...@gmail.com Jan 25 It would be easy to do this using the MPI.jl package. That's what I would do, but only because I'm familiar with how it works. The native parallel methods of Julia may work just as well, though. - show quoted text - Ritchie Lee Jan 26 Thanks for the suggestions everyone! - show quoted text -