Groups   43 of 99+      julia-users › Help with parallel computing for optimization. 2 posts by 2 authors      Nsep    Apr 11   Hello World,  I am relatively new with Julia. I wrote an optimization model  MIP  that I need to run many, many time for sensitivity analysis. I am working on a cluster that uses SLURM. I wrote my model as a Julia Module.   Basically, the process that I want to do is to have a file with all the different cases; and using a  for-loop  I want each case to be solved by a different node  all cores in the node solving the same MIP case .   I have tried two different ways:  1  using the machinefile option of Julia  see .sh file code below .   ! bin bash   SBATCH --uid nsep   SBATCH --job-name  juliaTest   SBATCH --partition newnodes  SBATCH --output  juliaTest. j. N.out   SBATCH --error  juliaTest. j. N.err   SBATCH --time 1:0:0   SBATCH -N 3   SBATCH -n 20  SBATCH --export ALL  export SLURM_NODEFILE `generate_pbs_nodefile`  .  etc profile.d modules.sh module add engaging julia 0.4.3 module add engaging gurobi 6.5.1  julia --machinefile $SLURM_NODEFILE   Cases-jl  Using this method I get an error when loading MyModule  the model    everywhere push! LOAD_PATH,   home nsep Test     everywhere using MyModule  everywhere using DataFrames   everywhere inpath   home nsep Test Input   everywhere outpath   home nsep Test Results  mysetup Dict     config. options for MyModule   everywhere mysetup  casepath   home nsep Test   everywhere cases_in_data   readtable  $casepath Cases_Control.csv , header true    parallel for c in 1:size cases_in_data,1     loading general inputs   myinputs   Load_inputs mysetup,inpath     creating output directory   mkdir  $outpath Case$c     case_outpath  $outpath Case$c     case-specific inputs   myinputs  pMaxCO2   1  cases_in_data :Emissions  c    myresults   solve_model mysetup,myinputs    write_outputs mysetup,case_outpath,myresults,myinputs  end  The error that I get is:  WARNING: replacing module MyModule WARNING: replacing module MyModule WARNING: replacing module MyModule  signal  11 : Segmentation fault-jl_module_using at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  unknown function  ip: 0x2aaaaae0def9  unknown function  ip: 0x2aaaaae0e1e5  unknown function  ip: 0x2aaaaae0de3d  unknown function  ip: 0x2aaaaae0e77c -jl_load_file_string at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  include_string at loading-jl:266-jl_apply_generic at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  include_from_node1 at . loading-jl:307-jl_apply_generic at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  unknown function  ip: 0x2aaaaadf92a3  unknown function  ip: 0x2aaaaadf8639  unknown function  ip: 0x2aaaaae0daac -jl_toplevel_eval_in at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  eval at . sysimg-jl:14-jl_apply_generic at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  anonymous at multi-jl:1364-jl_f_apply at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  anonymous at multi-jl:910 run_work_thunk at multi-jl:651 run_work_thunk at multi-jl:660-jlcall_run_work_thunk_21367 at   unknown line -jl_apply_generic at  cm shared engaging julia julia-a2f713dea5 bin .. lib julia libjulia.so  unknown line  anonymous at task-jl:58 unknown function  ip: 0x2aaaaadff514  unknown function  ip:  nil   sh: line 1: 18358 Segmentation fault       cm shared engaging julia julia-a2f713dea5 bin julia --worker Worker 2 terminated. ERROR  unhandled task failure : EOFError: read end of file  in read at stream-jl:911  in message_handler_loop at multi-jl:868  in process_tcp_streams at multi-jl:857  in anonymous at task-jl:63 ERROR: LoadError: ProcessExitedException    in yieldto at . task-jl:71  in wait at . task-jl:371  in wait at . task-jl:286  in wait at . channels-jl:63  in take! at . channels-jl:53  in take! at . multi-jl:809  in remotecall_fetch at multi-jl:735  in remotecall_fetch at multi-jl:740  in anonymous at multi-jl:1386  ...and 1 other exceptions.   in sync_end at . task-jl:413  in anonymous at multi-jl:1395  in include at . boot-jl:261  in include_from_node1 at . loading-jl:304  in process_options at . client-jl:280  in _start at . client-jl:378 while loading  home nsep Cases-jl, in expression starting on line 3   2  The other method I tried was using ClusterManagers-jl, .sh file below.   ! bin bash   SBATCH --uid nsep   SBATCH --job-name  juliaTest   SBATCH --partition newnodes  SBATCH --output  juliaTest. j. N.out   SBATCH --error  juliaTest. j. N.err   SBATCH --time 0:2:0   SBATCH -N 4  SBATCH --export ALL  .  etc profile.d modules.sh module add engaging julia 0.4.3 module add engaging gurobi 6.5.1  julia   julia_cluster-jl  and then in the Julia code I tried to run the SLURM:example in the ClusterManagers page.  using ClusterManagers    Arguments to the Slurm srun 1  command can be given as keyword   arguments to addprocs.  The argument name and value is translated to   a srun 1  command line argument as follows:   1  If the length of the argument is 1     -arg value ,      e.g. t  0:1:0      -t 0:1:0    2  If the length of the argument is   1     --arg value       e.g. time  0:1:0      --time 0:1:0    3  If the value is the empty string, it becomes a flag value,      e.g. exclusive        --exclusive    4  If the argument contains  _ , they are replaced with  - ,      e.g. mem_per_cpu 100     --mem-per-cpu 100   addprocs SlurmManager 4 , partition  newnodes , t  00:2:00    hosts      pids      for i in workers       host, pid   fetch  spawnat i  gethostname  , getpid         push! hosts, host      push! pids, pid  end    The Slurm resource allocation is released when all the workers have   exited for i in workers       rmprocs i  end   But I get this error:  Error launching Slurm job: MethodError length, :all_to_all,    If anyone could help figure out  1  what is wrong in my code when passing MyModule,  2  waht I am doing wrong when trying ClusterManagers, that would be AWESOME!         Jiahao Chen    Apr 12   Looks like the bug report ClusterManagers-jl 31. Can you try  Pkg.checkout  ClusterManagers     and see if that works for you?