Groups 163 of 99+ julia-users › non-responsive REPL with PyCall 10 posts by 4 authors Yakir Gagnon 10/12/15 I'm using PyCall to retrieve a vector of floats from an external spectrometer (a gizmo that measures light intensities per wavelength). I have it all set up in scheduled tasks with channels as the containers for the intensities and the python module I'm using is from [Andreas Poehlmann](https://github.com/ap--/python-seabreeze) (see below for code). One important piece of information is the integration time (similar to the shutter speed in a camera): after I set the integration time the spectrometer start sampling the spectra at that frequency. When I try to retrieve the intensities it will spit them out only when one cycle ends. This means that when I try to run the function that retrieves the intensities it can take anything from 0 to integration-time seconds. Here's the weird thing: When I run my code the REPL becomes non-responsive for integration-time seconds, so if I try to type some text, the letters get typed in only one letter at an integration-time (note that CPU usage is less than 3%)... But, if I replace the function that retrieves the intensities with some mock function that `sleep`s for a random amount of time and returns a (equally long) vector of random floats, the REPL jitter is gone..! I've stipped down my code to the following but the above described behavior is still happening: ```Julia using PyCall @pyimport seabreeze seabreeze.use("pyseabreeze") @pyimport seabreeze.spectrometers as sb devices = sb.list_devices() S = sb.Spectrometer(devices[1]) S[:integration_time_micros](300000) c0 = Channel{Array{Float64,1}}(1) function fetchI() while true y = S[:intensities]() put!(c0,y) end end function plotit() while true take!(c0) end end @schedule fetchI() @schedule plotit() ``` and if I replace the 11th line (`y = S[:intensities]()` in `fetchI()`) with `y = mockfunction()`, where: ```julia function mockfunction() sleep(rand()*IT*1e-6) return rand(2048) end ``` then the REPL becomes responsive again. Other than understanding what is going on here, I'd love to know how to get my snappy REPL back... Steven G. Johnson 10/12/15 On Monday, October 12, 2015 at 1:59:59 AM UTC-4, Yakir Gagnon wrote: One important piece of information is the integration time (similar to the shutter speed in a camera): after I set the integration time the spectrometer start sampling the spectra at that frequency. When I try to retrieve the intensities it will spit them out only when one cycle ends. This means that when I try to run the function that retrieves the intensities it can take anything from 0 to integration-time seconds. Here's the weird thing: When I run my code the REPL becomes non-responsive for integration-time seconds, so if I try to type some text, the letters get typed in only one letter at an integration-time (note that CPU usage is less than 3%)... But, if I replace the function that retrieves the intensities with some mock function that `sleep`s for a random amount of time and returns a (equally long) vector of random floats, the REPL jitter is gone..! Julia I/O functions, and functions like sleep(t), use the libuv library for asynchronous cooperative multitasking. That means that when one task is waiting on I/O, another task (e.g. the REPL) can wake up if there is something for it to do. However, Python I/O does not use libuv, so when the Python I/O task is waiting to finish reading something then it just blocks, and nothing else in Julia can run. Yakir Gagnon 10/12/15 Re: [julia-users] Re: non-responsive REPL with PyCall I see, thanks for the great explanation! So there's nothing I can do. Would Escher get around it? I guess I'd need to implement that python code in Julia... - show quoted text - Mohammed El-Beltagy 10/13/15 Re: [julia-users] Re: non-responsive REPL with PyCall Short of doing a reimplementation, you could possibly run your python code in another process via a remotecall (as described in the manual http://julia.readthedocs.org/en/latest/manual/parallel-computing/). In that case you REPL would be responsive as the I/O is done in another process. - show quoted text - Stefan Karpinski 10/13/15 Re: [julia-users] Re: non-responsive REPL with PyCall I'm working on fixing up the API to Amit's PR here that allows you to call a C function in another thread. That could also potentially be used for this. - show quoted text - Yakir Gagnon 10/13/15 Re: [julia-users] Re: non-responsive REPL with PyCall I'll try the remotecall option (next week), hopefully the fetch from that won't get things stuck. I imagine there are a lot of other Python libraries that don't use libuv and get Julia's coroutines stuck... Thanks for all the attention! Yakir Gagnon The Queensland Brain Institute (Building #79) The University of Queensland Brisbane QLD 4072 Australia cell +61 (0)424 393 332 work +61 (0)733 654 089 - show quoted text - Stefan Karpinski 10/13/15 Re: [julia-users] Re: non-responsive REPL with PyCall Yeah, this is a fundamental problem, not only with Python but any other C library that uses blocking calls. At one point, when wrapping such a C library, I had the slightly insane notion of using LD_PRELOAD to replace all the blocking functions in libc with our own versions that do the same thing but using Julia's yielding API. - show quoted text - Steven G. Johnson 10/13/15 Re: [julia-users] Re: non-responsive REPL with PyCall Note that PyCall supports passing Julia I/O streams to Python. So, if you can just monkey-patch the Python library to use a Julia I/O stream to talk to the spectrometer, then it will yield to Julia's event loops. Yakir Gagnon 10/25/15 Re: [julia-users] Re: non-responsive REPL with PyCall Thanks for the suggestion! I get an deserialize error. Not really sure how to make a S available on all processes: using Winston, PyCall, Reactive @pyimport seabreeze seabreeze.use("pyseabreeze") @pyimport seabreeze.spectrometers as sb devices = sb.list_devices() S = sb.Spectrometer(devices[1]) To bad really, because that seems like a really nice and easy solution. - show quoted text - Yakir Gagnon 10/25/15 Re: [julia-users] Re: non-responsive REPL with PyCall Finally had some time. Thanks for all the suggestions! I wouldn't know how to change stuff in that other python library (I only know some matlab and julia). But thanks. On Wednesday, October 14, 2015 at 5:49:32 AM UTC+10, Steven G. Johnson wrote: Note that PyCall supports passing Julia I/O streams to Python. So, if you can just monkey-patch the Python library to use a Julia I/O stream to talk to the spectrometer, then it will yield to Julia's event loops.