Groups


173 of 99+  



julia-users ›
Still confused about how to use pmap with sharedarrays in Julia
7 posts by 3 authors  



Graydon Barz 	

Aug 10


What is the best way to modify the following Julia code:

my_output = pmap(x -> my_function(x, my_shared_array), x_list)

so that 'output' is consistent with 

my_output = map(x -> my_function(x, myarray), x_list)?

My naive attempt above resulted in a an error:  
RemoteException(pid#, CapturedException(UndefVarError: my_shared_array),...  which I understand from other posts is because the other processes do not know of my_shared_array.

However, attempts to use various permutations of @sync @async have resulted in output = Task (done) @x0007 etc., and not the desired 'output' list. 


 


Greg Plowman 	

Aug 10


I have also found the combination of shared arrays, anonymous functions and parallel constructs confusing.

StackOverflow question helped me Shared array usage in Julia

In essence, "although the underlying data is shared to all workers, the declaration is not. You will still need to pass in the reference to [the shared array]"

S = SharedArray(Int,3,4)
S[:] = 1:length(S)
@everywhere f(S,i) = (println("S[$i] = $(S[i])"); S[i])

output1 = pmap(i -> f(S,i), 1:length(S))    # error
output2 = pmap((S,i) -> f(S,i), fill(S,length(S)), 1:length(S))
output3 = pmap(f, fill(S,length(S)), 1:length(S))


In seems then, in version1, the reference S is local to the worker, but S is not defined on the worker -> error.
In versions 2&3 the local S is passed as argument to worker and all works as expected.

 


Graydon Barz 	

Aug 11


Thank your for the suggestions as they enabled me to solve the issue - though for a different reason than I thought.

I had seen indeed seen the StackArray post you mentioned, but my initial attempt based on that post did not work. Specifically, following that post, I had tried:

output = pmap((x,y) -> f(x, y), ["strg1", "strg2", "strg_3"] , [S,S,S])

and received an error message for each of the three components of the output starting with: RemoteException(pid#,CapituredException(MethodError(getindex,(0.000115947...  

However, modifying it per your suggestion by defining SVector= fill(S, 3) and then running

output = pmap((x,y)->f(x,y), ["strg1", "strg2", "strg3"] , SVector) 

worked.

The fact that meaning of  "," changes depending on what is in placed in [a,b,c] seems have been the source of the issue.

(As an aside, this inconsistency seems to me to be somewhat of a less-than-desireable feature in Julia.)
- show quoted text -
 


Greg Plowman 	

Aug 11



The fact that meaning of  "," changes depending on what is in placed in [a,b,c] seems have been the source of the issue.

(As an aside, this inconsistency seems to me to be somewhat of a less-than-desireable feature in Julia.)


I think this is changing in v0.5. See https://github.com/JuliaLang/julia/blob/master/NEWS.md#language-changes-1

Square brackets and commas (e.g. [x, y]) no longer concatenate arrays, and always simply construct a vector of the provided values. If x and y are arrays, [x, y] will be an array of arrays (#3737, #2488, #8599).

 


Greg Plowman 	

Aug 12


pmap inside a function also seems to work:

@everywhere f(A,i) = (println("A[$i] = $(A[i])+1"); A[i] += 1)
wrapped_pmap(A) = pmap(i -> f(A,i), 1:length(A))

S = SharedArray(Int,10)
S[:] = 1:length(S)

output1 = pmap(i -> f(S,i), 1:length(S)) #error
output2 = wrapped_pmap(S) #seems to work


I'm quite confused about why second version with pmap wrapped in a function works.
Perhaps someone can explain?
 

 


Amit Murthy 	

Aug 12

Re: [julia-users] Re: Still confused about how to use pmap with sharedarrays in Julia

Since S is global, the closure i -> f(S,i) only captures a reference to S. And since this binding is not defined on the workers we see an error.

For example,

function foo()
    S = SharedArray(Int,10)
    S[:] = 1:length(S)

    output1 = pmap(i -> f(S,i), 1:length(S))
end

foo()

works as expected.

I agree that this is confusing.

- show quoted text -
 


Amit Murthy 	

Aug 12

Re: [julia-users] Re: Still confused about how to use pmap with sharedarrays in Julia

This would work too

let S=S
    output1 = pmap(i -> f(S,i), 1:length(S))
end
- show quoted text -