Groups


21 of 99+  



julia-users ›
@parallel and HDF5 is posible to work together?
5 posts by 3 authors  



paul analyst 	

9/1/15



@parallel  and HDF5 is posible to work together?

julia> addprocs(6)
6-element Array{Any,1}:
 2
 3
 4
 5
 6
 7

julia> @parallel for i=1:l
       dset=fid["punkts"*string(i)]
       f=vec(h5read("F.h5","F",(:,i))); 
       .........
       end

WARNING: Module HDF5 not defined on process 2
WARNING: Module HDF5 not defined on process 3
julia> close(fid)

julia> fatal error on 3: WARNING: Module HDF5 not defined on process 6
WARNING: Module HDF5 not defined on process 7
WARNING: Module HDF5 not defined on process 4
fatal error on 6: fatal error on fatal error on 74: : WARNING: Module HDF5 not defined on process 5
fatal error on 5: ERROR: HDF5 not defined
 in deserialize at serialize.jl:377
 in handle_deserialize at serialize.jl:352
 in deserialize at serialize.jl:506
 in handle_deserialize at serialize.jl:352

Paul
 


Nils Gudat 	

9/1/15


As the warning says, the module is not defined on the workers. Do

addprocs()
import HDF5
@everywhere using HDF5
 


paul analyst 	

9/4/15

Re: [julia-users] Re: @parallel and HDF5 is posible to work together?

Thanak lot Nils, 

Unfortunatly like below: 

julia> addprocs(7) 
7-element Array{Any,1}: 
  2 
  3 
  4 
  5 
  6 
  7 
  8 

julia> import HDF5 

julia> @everywhere using HDF5 

julia> @parallel for i=1:l 
        dset=fid["punkty"*string(i)] 
        f=vec(h5read("F.h5","F",(:,i))); 
... 
        println(i) 
        end 
julia> close(fid) 

julia> toc() 
elapsed time: 18.801164342 seconds 
18.801164342 

julia> exception on 2: ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
exception on 4: exception on 6: exception on 5: exception on exception 
on 3: exception on 8: 7: ERROR: File or o 
ect has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 
ERROR: File or object has been closed 
  in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 
(repeats 2 times) 







W dniu 2015-09-01 o 22:14, Nils Gudat pisze: 
- show quoted text -
 


paul analyst 	

9/4/15

Re: [julia-users] Re: @parallel and HDF5 is posible to work together?

what wrong?

fid = h5open("data.h5","r+")
this code work :


import HDF5
@everywhere using HDF5
@parallel for i=1:l
#dset=fid["punkty"*string(i)]

f=vec(h5read("F.h5","F",(:,i))); 
#dset[:,:]=f*rand()
println(i)
end
close(fid) 


this code do not work:

fid = h5open("data.h5","r+")

@parallel for i=1:l
       dset=fid["punkty"*string(i)]
       f=vec(h5read("F.h5","F",(:,i)));
       #dset[:,:]=f*rand()
       println(i)
       end

ERROR: File or object has been closedn exception on 236: 4:

 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
: ERROR: File or object has been closed
 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)

 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
ERROR: File or object has been closed
 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
julia> close(fiexception on dexception on 58: ):  ERROR: File or object has been closed

 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
ERROR: File or object has been closed
 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
7: julia> ERROR: File or object has been closed

 in o_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:696 (repeats 2 times)
close(fid)


Paul


W dniu 2015-09-04 o 19:44, Paul Analyst pisze:
- show quoted text -
 


René Donner 	

9/4/15


Only process number 1 has the actual underlying file handle. Therefore all other workers, when trying to read, report that the file is unaccessible. 

You could try to designate one worker for the reads, i.e. use something along the lines of 

  @fetchfrom 1 dset=fid["punkty"*string(i)] 

You could alternatively try to open the file on all workers - which should be ok if all accesses are read only. 
- show quoted text -
Topic options