Groups


48 of 99+  



julia-users ›
Parallel file access
3 posts by 2 authors  



Zachary Roth 	

Oct 10


Hi, everyone,

I'm trying to save to a single file from multiple worker processes, but don't know of a nice way to coordinate this.  When I don't coordinate, saving works fine much of the time.  But I sometimes get errors with reading/writing of files, which I'm assuming is happening because multiple processes are trying to use the same file simultaneously.

I tried to coordinate this with a queue/channel of `Condition`s managed by a task running in process 1, but this isn't working for me.  I've tried to simiplify this to track down the problem.  At least part of the issue seems to be writing to the channel from process 2.  Specifically, when I `put!` something onto a channel (or `push!` onto an array) from process 2, the channel/array is still empty back on process 1.  I feel like I'm missing something simple.  Is there an easier way to go about coordinating multiple processes that are trying to access the same file?  If not, does anyone have any tips?

Thanks for any help you can offer.

Cheers,
---Zachary
 


Ralph Smith 	

Oct 11


You can do it with 2 (e.g. integer) channels per worker (requests and replies) and a task for each pair in the main process. That's so ugly I'd be tempted to write an
 interface to named system semaphores. Or just use a separate file for each worker.
- show quoted text -

0 
0



Zachary Roth 	

3:19 PM (8 minutes ago)


Thanks for the reply and suggestion, Ralph.  I tried to get this working with semaphores/mutexes/locks/etc.  But I've not been having any luck.

Here's a simplified, incomplete version of what I'm trying to do.  I'm hoping that someone can offer a suggestion if they see some sample code.  

function localfunction()
    files = listfiles()
    locks = [Threads.SpinLock() for _ in files]
    ranges = getindexranges(length(files))

    pmap(pairs(ranges)) do rows_and_cols
        rows, cols = rows_and_cols
        workerfunction(files, locks, rows, cols)
    end
end

function workerfunction(files, locks, rows, cols)
    data = kindofexpensive(...)
    pairs = pairs(rows, cols)
    
    @sync for idx in unique([rows; cols])
        @async begin
            lock(locks[idx])
            updatefile(files[idx], data[idx])
            unlock(locks[idx])
        end
    end
end

This (obviously) does not work.  I think that the problem is that the locks are being copied when the function is spawned on each process.  I've tried wrapping the locks/semaphores in Futures/RemoteChannels, but that also hasn't worked for me.

I found that I could do the sort of coordination that I need by starting Tasks on the local process.  More specifically, each file would have an associated Task to handle the coordination between processes.  But this only worked for me in a simplified situation with the Tasks being declared globally.  When I tried to implement this coordination within localfunction above, I got an error (really a bunch of errors) that said that a running Task cannot be serialized.

Sorry for the long post, but I'm really hoping that someone can help me out.  I have a feeling that I'm missing something pretty simple.

---Zachary
- show quoted text -

0 
0