isready   blocks  7006 Closed GaborOszlanyi opened this Issue on May 28, 2014 · 15 comments Projects None yet Labels parallel Milestone No milestone Assignees No one assigned 4 participants  GaborOszlanyi  ViralBShah  amitmurthy  JeffBezanson Notifications    GaborOszlanyi GaborOszlanyi commented on May 28, 2014  isready RemoteRef  is there for checking whether a long calculation is ready.  However, the following call to isready   blocks until the calculation is ready:  julia  r    spawnat 2  n 4000; a rand n,n ; b inv a ;  julia  isready r  true  This makes some of my parallel code useless.  tested using 0.3 prerelease on both 32-bit Windows and Ubuntu Nightly   ViralBShah The Julia Language member ViralBShah commented on May 28, 2014  Yes, isready should really just check and return a Bool immediately.  ViralBShah ViralBShah added the parallel label on May 28, 2014  amitmurthy The Julia Language member amitmurthy commented on May 28, 2014  The issue is that if the other worker is computation bound, no other code can execute, even the isready call. Till we have a background thread, at least for such checks  and also maybe heartbeats , the workaround is to have a local RemoteRef and have the computation populate that. The following should work.  r RemoteRef    async put! r, remotecall_fetch 2,   -  n 4000; a rand n,n ; inv a     isready r    amitmurthy The Julia Language member amitmurthy commented on May 28, 2014  Or just have isready take a timeout value.  Just trolling  StefanKarpinski  :-   GaborOszlanyi GaborOszlanyi commented on May 28, 2014  Thank you very much for the workaround, Amit!   At the same time, I think the issue is still open.   JeffBezanson The Julia Language member JeffBezanson commented on May 28, 2014  isready is a race condition and should not exist  by the time you get its answer, it might not be correct anymore . This should be done by waiting for the computation in an async task as  amitmurthy shows, but you can do anything you need to when it finishes: pinging another RemoteRef as shown, or setting a flag, printing a message, whatever.  GaborOszlanyi GaborOszlanyi commented on May 28, 2014  So you plan to remove isready completely ?  The suggested workaround is already subtle and slightly verbose for me, what is the best practice that is brief and works without isready ?  Maybe a new macro ?  JeffBezanson The Julia Language member JeffBezanson commented on May 28, 2014  Well, isready does not really fit with our model, which is event-driven. You should get notifications when something happens, not ask whether it has happened. Perhaps say a bit more about how you use it, and why you need it.  GaborOszlanyi GaborOszlanyi commented on May 28, 2014  I am not arguing for isready, I started using it because it was there. I did not understand that its use means a race condition and is considered bad practice. Now that I learned it, I will try to switch from polling to something event driven.  I would greatly appreciate good examples. The use case? Fairly simple: hundreds of calculations taking different times on several cores, trying to keep all cores busy by noticing when a given core is available.  JeffBezanson The Julia Language member JeffBezanson commented on May 28, 2014  For that, I recommend the dynamic scheduling pattern used by pmap. You have a task per worker, each of which waits for a worker to finish its current task and then gives it the next item.  I think for this release we should deprecate isready.  GaborOszlanyi GaborOszlanyi commented on May 28, 2014  Thank you for the pmap suggestion.  As for the fate of isready let the experts decide. It seemed just so simple and straightforward.  amitmurthy The Julia Language member amitmurthy commented on May 29, 2014  I am not for deprecating isready. The event-driven model is fine when everything is working fine. But when tasks take too long - for whatever reason - bugs, external dependencies, etc,., we do need a mechanism that makes it easy to detect the same.  The race condition with isready exists only when there more than 2 readers writers on the same remoteref. The application just needs to be aware of the same.  As and when  6741 is ready, we should actually have IO  libuv  running in its own thread and all computation in their own. Like ZeroMQ does - it starts an independent internal io thread and all application 0MQ sockets communicate with it via an internal mechanism.  GaborOszlanyi GaborOszlanyi commented on May 29, 2014  So because my use case of isready was a single reader single writer of the same RemoteRef, I did not do anything wrong?  amitmurthy The Julia Language member amitmurthy commented on May 29, 2014  Yes. A simple example of the possible race condition with isready  isready r       say, returns true in process 1 isready r       say, returns true in process 2 take! r         gets value immediately in process 1 take! r         blocks in process 2, since between the time of isready and take! in process 2, it has already been taken by process 1.   JeffBezanson The Julia Language member JeffBezanson commented on May 29, 2014  Ok, we can keep it. We can warn in the docs that it might block, and suggest async waiting instead.  ViralBShah The Julia Language member ViralBShah commented on May 29, 2014  I like keeping it and warning in the docs. How about putting the recipe described here in the manual for future users with the same question.  JeffBezanson JeffBezanson added a commit that closed this issue on May 29, 2014   JeffBezanson  improve docs for isready  . closes  7006    5cd7754  JeffBezanson JeffBezanson closed this in 5cd7754 on May 29, 2014  ViralBShah ViralBShah referenced this issue on May 29, 2014 Closed Implement S I::AbstractVector Bool   and S I::AbstractVector  for sparse X  7023