functions are serialised at every batch with `pmap` and `pgenerate` #16345
Closed
oxinabox opened this Issue on May 12 · 8 comments
Projects
None yet
Labels
parallel
performance
Milestone
No milestone
Assignees
No one assigned
3 participants
@oxinabox
@amitmurthy
@JeffBezanson
Notifications

You’re not receiving notifications from this thread.
@oxinabox
oxinabox commented on May 12 • edited

So I was trying to workout why my parallel code was taking so long.
After-all I only sent the big datastructures once, though a closure as the function that was mapped over.
That should happen, once (I thought), since the function is constant
Not so

MWE:

addprocs(5)

immutable Foo
    bar::Char
end

#HACK: lets debug what is being serialised by overloading the calls
function Base.serialize(s::Base.SerializationState, x::Foo)
    tic()
    Base.Serializer.serialize_any(s,x)
    tt=toq()
    open("ser_log.txt","a") do fp
        println(fp, tt)
    end
end

function test()
    st = Foo(rand('a':'z')) 
    pmap(r->string(st.bar)^r, 1:100)
    #Base.pgenerate(default_worker_pool(), r->string(st.bar)^r, 1:100) |> collect
end

Then running the function and counting the lines in the log:

test()
run(`wc -l ser_log.txt`)
OUT> 17

So it was serialized 17 times for pmap.
If is switch to pgenerate is it 18 times (so about the same).
I believe that after the batchsplit step is done that is once serialisation of the closure, per batch that was sent.
It only need to be serialized once.

(in my nonMWE, it is happening millions of times, and takes 6 seconds a piece...)

I suspect this is already known, but I can't find an issue for it, so maybe not.

see also:

    #11938
    #16322

versioninfo()
Julia Version 0.5.0-dev+3928
Commit fdc4a85 (2016-05-06 04:46 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: AMD Opteron 63xx class CPU
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Piledriver)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, bdver2

@oxinabox
oxinabox commented on May 13
Work Arounds

So goal here is to cut the serialisation down to once per worker.
Ideally it would be once ever, but that requires a delving down deeper.
Presending (Does Not Work, and IDK why)

Idea is we should first send the function into remote Futures.
Then replace the function we send with pgenerate with a lightweight function that does a lookup for the right (local) Future.
Instead of each time sending the orginal function (which could be a closure haulling tons of data).
Just send a call to fetch the function.

I really thought this would work, but it does not.
And that might be a bug in and of itself; or the cause of the bug reported above.

function _pgenerate(f,c)
    worker_ids = workers()
    r_fs = Dict{Int64,Base.AbstractRemoteRef}()
    for id in worker_ids
        r_fs[id]=Future(id)
        put!(r_fs[id], f)                
    end

    worker_pool = WorkerPool(worker_ids)

    #Give everyone the dict of functions on each worker
    #get them to retieve theirs
    #and execute it    
    Base.pgenerate(worker_pool, x->fetch(r_fs[myid()])(x), c)  
end

Does not help. Infact makes it worse, cos one extra call initially, per worker.
Global Variable Hack (Works, but leaks memory and is and scary)

This is kinda scary.
Instead of sending the function intailly, in a remote Future, and then fetching it.
Send a command to create a global variable with the function it it.
Then just use the function that is now declared remotely.

function _pgenerate_gh(f,c, mname=Symbol("_func_genmagic_hack"*string(rand(1:1024)))::Symbol)
    #Reusing the `mname` in subsequent called can A.) Reclaim memory, B.) Violate certain concurrency expectations
    worker_ids = workers()
    for id in worker_ids
        remotecall_wait(id, mname,f) do mname_i, f_i
            eval(Expr(:global, Expr(Symbol("="), mname_i, f_i)))
        end
    end

    worker_pool = WorkerPool(worker_ids)

    #Give send a function telling them to look up the function locally
    Base.pgenerate(worker_pool, x->eval(mname)(x), c)  
end

I feel unclean.
@amitmurthy
The Julia Language member
amitmurthy commented on May 13

For this particular scenario we could support a batch_size=:max in #15975 which would result in only a single call to each worker.

Other alternatives (along the lines you are exploring)

    create a global variable with only the data and use that. You will need to remember to clear it after the map though.
    https://github.com/amitmurthy/ClusterDicts.jl has a simple approach of replicating data on all workers and storing them in a global dict and use a pre-defined named function that accesses data off the global dict. Still need to remember to clear data off the dict though.

cc: @samoconnor
@oxinabox
oxinabox commented on May 13 • edited

A very different work around is what I thought was a good idea a couple of days ago.
Before I found pgenerate

http://codereview.stackexchange.com/questions/128058/using-remotechannels-for-an-parallel-iterable-map
(That code has problems, like lost of small sends)

The idea being that when you start you create on each worker a task that infinite loops, while reading one channel and writing to another. The infinite loop ends when a channel is closed.
Then the constant data portion - i.e. the expensive closure is only serialised in the first step when the loops on the workers are created.

Problem with batch_size=:max is that it assumed you can hold the whole Iterable you are running over in memory. Or at least a very substantial portion of it.
And that is not generally true.
(Although as I have just found out. I personally can afford to whole the whole Gigaword Corpus in RAM. Most people can't.)
@JeffBezanson
The Julia Language member
JeffBezanson commented on May 13

We should probably remember which functions we have sent to which workers and automatically avoid duplicating.
@amitmurthy
The Julia Language member
amitmurthy commented on May 13

Not a bad idea. The mapping function is typically an anonymous function. pmap should cache it on each worker only for the duration of the call. Will need to store in a remote reference and manage its lifetime efficiently.
@JeffBezanson
The Julia Language member
JeffBezanson commented on May 13

Thinking in more detail, closures might be a separate issue. Internal to the serializer, we could avoid re-sending the code for functions, which is not very big but is slow to serialize. Avoiding re-sending closure data is harder, since it can be arbitrarily big and will need a cache management strategy as you point out.
@JeffBezanson JeffBezanson added performance parallel labels on May 13
@oxinabox
oxinabox commented on May 18

So I improved my hack around the issue, to actually only serialize once.
By doing the serialization entirely outside the remote_calls, and just sending the serialised data.
It is annoying that the serialization code is stateful (rather than pure), which makes it hard to logic around -- I was trying to do this at a lower, more general level.

This, as with the last, is twice as fast as pmap, in the above test.
I can't using the MWE above detect a significant difference.
But I suspect that when it comes to my actual code.
Which hauls >10MB of data around within the closures, and has to send to 32 workers,
this will be pretty huge.

function _pgenerate_gh_sc(f,c, mname=Symbol("_func_genmagic_hack"*string(rand(UInt64)))::Symbol)
    #Reusing the `mname` in subsequent called can A.) Reclaim memory, B.) Violate certain concurrency expectations
    worker_ids = workers()

    s_f_buff = IOBuffer()
    serialize(s_f_buff, f)
    s_f = s_f_buff.data

    function make_global(mname_i, s_f_i)
        eval(Expr(:global, 
                    Expr(Symbol("="), mname_i, 
                        Expr(:call,:deserialize,
                            Expr(:call, :IOBuffer, s_f_i)
                            )
                        )
                 )
            )
    end

    @sync for id in worker_ids
        @async remotecall_wait(make_global, id, mname, s_f)
    end

    worker_pool = WorkerPool(worker_ids)

    #Give send a function telling them to look up the function locally
    Base.pgenerate(worker_pool, x->eval(mname)(x), c)  
end

This was referenced on May 20
Merged
Configure pmap behavior via keyword args #15975
Open
rename SerializationState to Serializer, make official, document #16558
Closed
RFC/WIP : cache mapped function remotely for pmap. [ci skip] #16695
Merged
Introduce AbstractWorkerPool, CachingPool #16808
@amitmurthy
The Julia Language member
amitmurthy commented on Jul 21

closed by #16808
@amitmurthy amitmurthy closed this on Jul 21