Groups 37 of 99+ julia-users › good approach to run julia scripts in parallel in a PC 3 posts by 2 authors Charles Santana Jan 12 Hi julians, Parallel computing is not my focus, but I would like to run a julia script in parallel in my PC in order to save time. The PC has an 8-core processor so I would like to run at least 4 replicates of my script (test1.jl) at the same time, each one with one different parameter. So far I was working on a cluster and I was using qsub in order to submit my jobs. I didn't need to configure anything. I had a quota of 20 jobs, so if I launched 200 jobs 20 of them would run and the others would be in a queue waiting for their moment to run. Now I would like to run my scripts in my machine and I would like to do something similar. For me it is fine if I wait for the current jobs to finish running in order to launch another bunch of jobs. As well as it is fine if I launch a new job for each experiment that finishes. The easiest option the best :) My script does not receive any file as input, but it writes the outputs to unique files (the file names are unique for each job). Do you recommend a good way to do it in Julia? Let's say my script is called "test1.jl". So far I was trying the following code to call my jobs: function main() parameters = collect(0.001:0.001:0.6); @parallel for(p in parameters) command1 = `/home/cdesantana/Downloads/julia -p 4 /home/cdesantana/Experiments/test1.jl $p`; run(command1); end end main(); I really don't understand what is happening here, but I am sure it is not working 4 jobs of my scripts... :( cdesantana@c-de-santana:~/Data/Dendritics/poster2016$ ps aux|grep julia cdesant+ 1964 3.7 1.3 9214696 107356 pts/10 Sl 15:20 0:01 /home/cdesantana/Downloads/julia/julia script.jl cdesant+ 1994 92.2 2.7 9489844 214388 pts/10 Rl 15:20 0:31 /home/cdesantana/Downloads/julia/julia -p4 /home/cdesantana/Experiments/test1.jl 1 0.001 cdesant+ 2013 8.7 1.5 9189424 120340 ? Ssl 15:20 0:02 /home/cdesantana/Downloads/julia/usr/bin/julia -Cnative -J/home/cdesantana/Downloads/julia/usr/lib/julia/sys.so --bind-to 192.168.89.174 --worker cdesant+ 2016 9.2 1.6 9288520 131600 ? Ssl 15:20 0:03 /home/cdesantana/Downloads/julia/usr/bin/julia -Cnative -J/home/cdesantana/Downloads/julia/usr/lib/julia/sys.so --bind-to 192.168.89.174 --worker cdesant+ 2018 6.7 1.6 9353972 131772 ? Ssl 15:20 0:02 /home/cdesantana/Downloads/julia/usr/bin/julia -Cnative -J/home/cdesantana/Downloads/julia/usr/lib/julia/sys.so --bind-to 192.168.89.174 --worker cdesant+ 2023 7.7 1.6 9419496 131604 ? Ssl 15:20 0:02 /home/cdesantana/Downloads/julia/usr/bin/julia -Cnative -J/home/cdesantana/Downloads/julia/usr/lib/julia/sys.so --bind-to 192.168.89.174 --worker cdesant+ 2159 0.0 0.0 11716 892 pts/10 S+ 15:21 0:00 grep --color=auto julia I was expecting that it would be running my script 4 times, each one with one different value of $p (0.001, 0.002, 0.003, 0.004). Any suggestion? My machine has ubuntu installed so I am fine if I need to combine Linux programs with Julia. Many thanks in advance for any help! Best, Charles -- Um axé! :) -- Charles Novaes de Santana, PhD https://github.com/cndesantana Greg Plowman Jan 13 You might be able to use pmap: http://docs.julialang.org/en/release-0.4/manual/parallel-computing/#parallel-map-and-loops http://docs.julialang.org/en/release-0.4/stdlib/parallel/?highlight=pmap#Base.pmap Perhaps something like: @everywhere function test1(p) println("doing stuff with $p") return 2*p end function main() parameters = collect(0.001:0.001:0.6) results = pmap(test1, parameters) end main() - show quoted text - Charles Santana Jan 13 Re: [julia-users] Re: good approach to run julia scripts in parallel in a PC Absolutely amazing!!! Thanks a lot, Greg! It works like a charm! Best, Charles - show quoted text -