Groups


8 of 99+  



julia-users ›
Why does enumerate fail in parallel?
8 posts by 5 authors  



Nils Gudat 	

8/19/15


I just rewrote one of my parallel loops and was surprised by this:

> list = ["a", "b", "c"]

> for i in enumerate(list)
>   println(i)
> end

(1,"a") 
(2,"b") 
(3,"c")

> addprocs(2)

> @sync @parallel for i in enumerate(list)
>   println(i)
> end

ERROR: `getindex` has no method matching getindex(::Enumerate{Array{ASCIIString,1}}, ::UnitRange{Int64})

Am I doing something wrong here? Is this expected behaviour?
 


Ismael Venegas Castelló 	

8/19/15


Enumerate is an iterator, you need to collect the items first:

julia> @parallel for i in collect(enumerate(list))
           println(i)
       end

julia>  From worker 2:  (1,"a")
    From worker 2:  (2,"b")
    From worker 3:  (3,"c")
- show quoted text -
 


Ismael Venegas Castelló 	

8/19/15


Well that works but it's indeed odd, can you open a new issue for this?
- show quoted text -
 


ele...@gmail.com 	

8/20/15




On Thursday, August 20, 2015 at 4:50:49 AM UTC+10, Ismael VC wrote:
Well that works but it's indeed odd, can you open a new issue for this?

Not really odd, @parallel needs to divide the set of values between multiple processes, so it needs the whole set of values.
- show quoted text -
 


John Brock 	

8/20/15


This seems issue-worthy if the most recent nightly have the same problem. It looks like Enumerate supports the length property, so the underlying code for @parallel should be able to check the length of the enumerator and figure out how many jobs to assign to each worker. 

And regardless of whether it makes sense for @parallel to support enumerate, that error message is pretty opaque -- it doesn't make it obvious to the programmer that parallel doesn't support enumerate, which is a pretty natural thing to try.
- show quoted text -
 


John Brock 	

8/20/15


This seems issue-worthy if the most recent nightly has the same problem. It looks like Enumerate supports the length property, so the underlying code for @parallel should be able to check the length of the enumerator and figure out how many jobs to assign to each worker. 

And regardless of whether it makes sense for @parallel to support enumerate, that error message is pretty opaque -- it doesn't make it obvious to the programmer that parallel doesn't support enumerate, which is a pretty natural thing to try.

On Thursday, August 20, 2015 at 3:44:49 AM UTC-7, ele...@gmail.com wrote:
- show quoted text -
 


ele...@gmail.com 	

8/20/15


Certainly the error message could be more useful.

If it is possible to detect that the argument is an iterator then @parallel could do the collect itself though.
- show quoted text -
 


Michael Ohlrogge 	

Feb 13


Yes, I just lost about an hour trying to figure out what was going on here.  I didn't find it intuitive that the parallel should not work for enumeration, and the error message did little to point me in the right direction.


On Thursday, August 20, 2015 at 7:46:37 PM UTC-7, ele...@gmail.com wrote:
Certainly the error message could be more useful.