Groups   13 of 99+      julia-users › parallel and PyCall 9 posts by 4 authors      Yakir Gagnon    10 25 15     How can I use pyimport with multiple processes? So say I want  taken from here   using PyCall  pyimport math math.sin math.pi   4  - sin pi   4     returns 0.0 How exactly can I make this work  after launching julia with say 2 processes :  r    spawn math.sin math.pi   4  fetch r  - sin pi   4     returns 0.0 Currently I’m getting this error:  WARNING: Module __anon__ not defined on process 2 fatal error on 2: ERROR: UndefVarError: __anon__ not defined  in deserialize at serialize-jl:500  in handle_deserialize at serialize-jl:461  in deserialize at serialize-jl:480  in handle_deserialize at serialize-jl:461  in deserialize at serialize-jl:536  in handle_deserialize at serialize-jl:461  in deserialize at serialize-jl:694  in deserialize_datatype at serialize-jl:647  in handle_deserialize at serialize-jl:461  in message_handler_loop at multi-jl:847  in process_tcp_streams at multi-jl:836  in anonymous at task-jl:63 RemoteRefWorker 2 terminated.  Channel ERROR  unhandled task failure : EOFError: read end of file Any   2,1,11      Matthew Pearce    10 26 15   Thought I had an idea about this, I was wrong:  ```julia  julia   everywhere using PyCall  julia   everywhere  pyimport pylab  julia  remotecall_fetch pylab.cumsum, 5, collect 1:10   ERROR: cannot serialize a pointer   inlined code  from error-jl:21  in serialize at serialize-jl:420   inlined code  from dict-jl:372  in serialize at serialize-jl:428  in serialize at serialize-jl:310  in serialize at serialize-jl:420  repeats 2 times   in serialize at serialize-jl:302  in serialize at serialize-jl:420   inlined code  from dict-jl:372  in serialize at serialize-jl:428  in serialize at serialize-jl:310  in serialize at serialize-jl:420  repeats 2 times   in serialize at serialize-jl:302  in serialize at serialize-jl:420   inlined code  from dict-jl:372  in send_msg_ at multi-jl:222   inlined code  from multi-jl:177  in remotecall_fetch at multi-jl:728   inlined code  from multi-jl:368  in remotecall_fetch at multi-jl:734  julia  pylab.cumsum collect 1:10   10-element Array Int64,1 :   1   3   6  10  15  21  28  36  45  55  ```  0  0    Yakir Gagnon    10 26 15   Yea, right? So what’s the answer? How can we if at all do any PyCalls parallely?    0  0    Yakir Gagnon    10 29 15    Matthew: did you find a solution?    0  0    Matthew Pearce    10 30 15    So I got something working for my pylab example.   julia  import PyCall  julia  PyCall. pyimport pylab  julia   everywhere import PyCall  julia   everywhere PyCall. pyimport pylab  julia   everywhere A   pylab.cumsum collect 1:10   1.  julia  fetch  spawnat remotes 1  A  10-element Array Float64,1 :   1.0   3.0   6.0  10.0  15.0  21.0  28.0  36.0  45.0  55.0     No luck with the math module I'm afraid. Two different types of errors depending on style:  julia   spawnat remotes 1  PyCall. pyimport math as pymath RemoteRef Channel Any   2,1,305   julia  fetch  spawnat remotes 1   pymath.sin pymath.pi   4  - sin pymath.pi   4     ERROR: On worker 2: UndefVarError: pymath not defined  in anonymous at multi-jl:1330  in anonymous at multi-jl:889  in run_work_thunk at multi-jl:645  in run_work_thunk at multi-jl:654  in anonymous at task-jl:54  in remotecall_fetch at multi-jl:731    inlined code  from multi-jl:368   in call_on_owner at multi-jl:776  in fetch at multi-jl:784  julia   everywhere PyCall. pyimport math as pymath  julia  fetch  spawnat remotes 1   pymath.sin pymath.pi   4  - sin pymath.pi   4     Worker 2 terminated.srun: error: mrc-bsu-tesla1: task 0: Exited with exit code 1 ERROR: ProcessExitedException    in yieldto at . task-jl:67  in wait at . task-jl:367  in wait at . task-jl:282  in wait at . channels-jl:97  in take! at . channels-jl:84  in take! at . multi-jl:792  in remotecall_fetch at multi-jl:729    inlined code  from multi-jl:368   in call_on_owner at multi-jl:776  in fetch at multi-jl:784    ERROR  unhandled task failure : EOFError: read end of file     0  0    Yakir Gagnon    10 30 15   OK, I found a horrible work around that might be good enough  for me : Here is a mock python script:  import math, time x   math.sin math.pi 4  time.sleep 5    mock local work print x    spit it out to julia and here is the julia code that runs it:  r    spawn readall `python example.py`  sleep 2    mock local work wait r    wait on python t   fetch r  x   parse t  x - sin pi 4    not zero but it works... This seems to work no matter what. It’s horrid, but better than nothing.    0  0    thr    12 6 15   I think I found a way to use pycall in parallel:  To get around the  cannot serialize pointer problem , I declared the corpus delicti everywhere and wrapped it inside a julia function that is also declared everywhere: It's also kind of clumsy and I'm not sure if it applies to your problem, but this worked for me:   everywhere using PyCall  everywhere  pyimport a_python_module   everywhere pyfun   a_python_function    everywhere function wrap_pyfun x      return pyfun x   end  r1   remotecall 2, wrap_pyfun, arg1  r2   remotecall 3, wrap_pyfun, arg2   It also works if you have a python object and you need to call one of it's methods  everywhere pyobj   a_python_object    everywhere function wrap_pymethod x      return pyobj :method  x  end  hope this helps   ...  0  0    Yakir Gagnon    12 6 15  Re:  julia-users  Re: parallel and PyCall  OMG.... I went through so much round about ways to avoid that error... This would be amazing if it works!!! Thanks!   Yakir Gagnon The Queensland Brain Institute  Building  79  The University of Queensland Brisbane QLD 4072 Australia  cell +61  0 424 393 332 work +61  0 733 654 089   0  0    Michael Wojnowicz    Jan 14  Re:  julia-users  Re: parallel and PyCall   thr, your solution of writing a Julia wrapper worked for me.  thank you. i wouldn't have thought of that.  how did you think to do that, by the way?   0  0