Groups   48 of 99+      julia-users › Parallel file access 3 posts by 2 authors      Zachary Roth    Oct 10   Hi, everyone,  I'm trying to save to a single file from multiple worker processes, but don't know of a nice way to coordinate this.  When I don't coordinate, saving works fine much of the time.  But I sometimes get errors with reading writing of files, which I'm assuming is happening because multiple processes are trying to use the same file simultaneously.  I tried to coordinate this with a queue channel of `Condition`s managed by a task running in process 1, but this isn't working for me.  I've tried to simiplify this to track down the problem.  At least part of the issue seems to be writing to the channel from process 2.  Specifically, when I `put!` something onto a channel  or `push!` onto an array  from process 2, the channel array is still empty back on process 1.  I feel like I'm missing something simple.  Is there an easier way to go about coordinating multiple processes that are trying to access the same file?  If not, does anyone have any tips?  Thanks for any help you can offer.  Cheers, ---Zachary     Ralph Smith    Oct 11   You can do it with 2  e.g. integer  channels per worker  requests and replies  and a task for each pair in the main process. That's so ugly I'd be tempted to write an  interface to named system semaphores. Or just use a separate file for each worker.   0  0    Zachary Roth    3:19 PM  8 minutes ago    Thanks for the reply and suggestion, Ralph.  I tried to get this working with semaphores mutexes locks etc.  But I've not been having any luck.  Here's a simplified, incomplete version of what I'm trying to do.  I'm hoping that someone can offer a suggestion if they see some sample code.    function localfunction       files   listfiles       locks    Threads.SpinLock   for _ in files      ranges   getindexranges length files        pmap pairs ranges   do rows_and_cols         rows, cols   rows_and_cols         workerfunction files, locks, rows, cols      end end  function workerfunction files, locks, rows, cols      data   kindofexpensive ...      pairs   pairs rows, cols            sync for idx in unique  rows; cols            async begin             lock locks idx               updatefile files idx , data idx               unlock locks idx           end     end end  This  obviously  does not work.  I think that the problem is that the locks are being copied when the function is spawned on each process.  I've tried wrapping the locks semaphores in Futures RemoteChannels, but that also hasn't worked for me.  I found that I could do the sort of coordination that I need by starting Tasks on the local process.  More specifically, each file would have an associated Task to handle the coordination between processes.  But this only worked for me in a simplified situation with the Tasks being declared globally.  When I tried to implement this coordination within localfunction above, I got an error  really a bunch of errors  that said that a running Task cannot be serialized.  Sorry for the long post, but I'm really hoping that someone can help me out.  I have a feeling that I'm missing something pretty simple.  ---Zachary   0  0