Groups   163 of 99+      julia-users › non-responsive REPL with PyCall 10 posts by 4 authors      Yakir Gagnon    10 12 15   I'm using PyCall to retrieve a vector of floats from an external spectrometer  a gizmo that measures light intensities per wavelength . I have it all set up in scheduled tasks with channels as the containers for the intensities and the python module I'm using is from  Andreas Poehlmann  https:  github.com ap-- python-seabreeze   see below for code .   One important piece of information is the integration time  similar to the shutter speed in a camera : after I set the integration time the spectrometer start sampling the spectra at that frequency. When I try to retrieve the intensities it will spit them out only when one cycle ends. This means that when I try to run the function that retrieves the intensities it can take anything from 0 to integration-time seconds.   Here's the weird thing:  When I run my code the REPL becomes non-responsive for integration-time seconds, so if I try to type some text, the letters get typed in only one letter at an integration-time  note that CPU usage is less than 3  ... But, if I replace the function that retrieves the intensities with some mock function that `sleep`s for a random amount of time and returns a  equally long  vector of random floats, the REPL jitter is gone..!   I've stipped down my code to the following but the above described behavior is still happening:   ```Julia using PyCall  pyimport seabreeze seabreeze.use  pyseabreeze    pyimport seabreeze.spectrometers as sb devices   sb.list_devices   S   sb.Spectrometer devices 1   S :integration_time_micros  300000  c0   Channel Array Float64,1   1  function fetchI    while true   y   S :intensities      put! c0,y   end end function plotit    while true   take! c0   end end  schedule fetchI    schedule plotit   ``` and if I replace the 11th line  `y   S :intensities   ` in `fetchI  `  with `y   mockfunction  `, where: ```julia function mockfunction    sleep rand   IT 1e-6   return rand 2048  end ``` then the REPL becomes responsive again.  Other than understanding what is going on here, I'd love to know how to get my snappy REPL back...      Steven G. Johnson    10 12 15     On Monday, October 12, 2015 at 1:59:59 AM UTC-4, Yakir Gagnon wrote:  One important piece of information is the integration time  similar to the shutter speed in a camera : after I set the integration time the spectrometer start sampling the spectra at that frequency. When I try to retrieve the intensities it will spit them out only when one cycle ends. This means that when I try to run the function that retrieves the intensities it can take anything from 0 to integration-time seconds.   Here's the weird thing:  When I run my code the REPL becomes non-responsive for integration-time seconds, so if I try to type some text, the letters get typed in only one letter at an integration-time  note that CPU usage is less than 3  ... But, if I replace the function that retrieves the intensities with some mock function that `sleep`s for a random amount of time and returns a  equally long  vector of random floats, the REPL jitter is gone..!   Julia io functions, and functions like sleep t , use the libuv library for asynchronous cooperative multitasking.  That means that when one task is waiting on io, another task  e.g. the REPL  can wake up if there is something for it to do.  However, Python io does not use libuv, so when the Python io task is waiting to finish reading something then it just blocks, and nothing else in Julia can run.     Yakir Gagnon    10 12 15  Re:  julia-users  Re: non-responsive REPL with PyCall  I see, thanks for the great explanation!  So there's nothing I can do. Would Escher get around it? I guess I'd need to implement that python code in Julia...       Mohammed El-Beltagy    10 13 15  Re:  julia-users  Re: non-responsive REPL with PyCall  Short of doing a reimplementation, you could possibly run your python code in another process via a remotecall  as described in the manual http:  julia.readthedocs.org en latest manual parallel-computing  . In that case you REPL would be responsive as the io is done in another process.       Stefan Karpinski    10 13 15  Re:  julia-users  Re: non-responsive REPL with PyCall  I'm working on fixing up the API to Amit's PR here that allows you to call a Clanguage function in another thread. That could also potentially be used for this.      Yakir Gagnon    10 13 15  Re:  julia-users  Re: non-responsive REPL with PyCall  I'll try the remotecall option  next week , hopefully the fetch from that won't get things stuck.  I imagine there are a lot of other Python libraries that don't use libuv and get Julia's coroutines stuck...   Thanks for all the attention!    Yakir Gagnon The Queensland Brain Institute  Building  79  The University of Queensland Brisbane QLD 4072 Australia  cell +61  0 424 393 332 work +61  0 733 654 089      Stefan Karpinski    10 13 15  Re:  julia-users  Re: non-responsive REPL with PyCall  Yeah, this is a fundamental problem, not only with Python but any other Clanguage library that uses blocking calls. At one point, when wrapping such a Clanguage library, I had the slightly insane notion of using LD_PRELOAD to replace all the blocking functions in libc with our own versions that do the same thing but using Julia's yielding API.      Steven G. Johnson    10 13 15  Re:  julia-users  Re: non-responsive REPL with PyCall  Note that PyCall supports passing Julia io streams to Python.  So, if you can just monkey-patch the Python library to use a Julia io stream to talk to the spectrometer, then it will yield to Julia's event loops.     Yakir Gagnon    10 25 15  Re:  julia-users  Re: non-responsive REPL with PyCall  Thanks for the suggestion! I get an deserialize error. Not really sure how to make a S available on all processes:  using Winston, PyCall, Reactive  pyimport seabreeze seabreeze.use  pyseabreeze    pyimport seabreeze.spectrometers as sb devices   sb.list_devices   S   sb.Spectrometer devices 1   To bad really, because that seems like a really nice and easy solution.       Yakir Gagnon    10 25 15  Re:  julia-users  Re: non-responsive REPL with PyCall  Finally had some time. Thanks for all the suggestions! I wouldn't know how to change stuff in that other python library  I only know some matlab and julia . But thanks.     On Wednesday, October 14, 2015 at 5:49:32 AM UTC+10, Steven G. Johnson wrote: Note that PyCall supports passing Julia io streams to Python.  So, if you can just monkey-patch the Python library to use a Julia io stream to talk to the spectrometer, then it will yield to Julia's event loops.