Groups


198 of 99+  



julia-users ›
@everywhere and memory allocation
3 posts by 2 authors  



Pieterjan Robbe 	

12/1/15


does the @everywhere macro allocate extra memory to make local copies of a matrix for every processor?

A = sprandn(10000,10000,0.7)
@time A = sprandn(10000,10000,0.7)

gives 2.422259 seconds (23 allocations: 1.565 GB, 3.77% gc time)

@everywhere A = sprandn(10000,10000,0.7)
@time @everywhere A = sprandn(10000,10000,0.7)

gives 16.495639 seconds (1.31 k allocations: 1.565 GB, 6.14% gc time).

However, I know that there are local copies of the matrix on each processor:

@everywhere println(A[1,1])


-1.2751101862102039

	From worker 5:	0.0

	From worker 4:	0.0

	From worker 2:	0.853669869355948

	From worker 3:	0.0


Is there a way to use the @everywhere macro without allocating extra memory? Suppose A was created using

A = speye(10000,10000,0.7)

is there also a copy of the matrix A for all of the workers?
 


Andre Bieler 	

12/1/15


I think you are looking for shared arrays?
http://docs.julialang.org/en/latest/manual/parallel-computing/#id2
- show quoted text -
 


Pieterjan Robbe 	

12/1/15


No, not really. SharedArrays do only support bitstypes. I have an application where the function to be executed in parallel depends on some fixed data (some constants, a dict, some arrays, etc.) I would like to replace my @parallel for loop by @everywhere (because of the reuse of my cholesky factorization), but I'm worried that this will explode my memory usage.