Groups   29 of 99+      julia-users › Complex parallel computing implementation 4 posts by 2 authors      Adrian Salceanu    Sep 12   I was wondering if anybody can point me towards a tutorial or a large code base using parallel computing. Everything that is discussed so far in the docs and books is super simple - take a function, run it in parallel, the end.   To explain, I'm working on a full stack MVC web framework - so think many functions and a few types grouped in maybe 20 modules. Plus more or less 20 other external modules. The workflow that I'm after is:   1. bootstrap - load the necessary components to start up the framework  parsing command line args, loading configuration, setting up include paths, etc  2. start an instance of HttpServer and listen to a port 3. when the server receives a request it invokes a function of the Router module which is the entry point into the MVC stack 4. once the Router gets the request, it's pushed up the MVC stack and at the end a HttpServer.Response instance is returned  That being said,  a. my strategy is simple: for each request, spawn the function call at  3 to a worker b. imagine that what's at  4 represents 80  of the app, with a multitude of functions being invoked across a multitude of modules  ORM, controller, Models, Loggers, Databases, Caching, Sessions, Authentication, etc .   Everything works great single process, but getting the stack to run on multiple workers it's a nightmare. I got it to the point where at  3 I can invoke a simple function call  something like returning a string or a date  and run it on multiple workers. But when I try to invoke the Router and snowball the framework, I end up in an avalanche of unknown references.   The codebase is now littered with calls to  everywhere that add a lot of noise. But up to this point I still wasn't able to make it work. The errors come from deep within the Julia stack, the workers crash saying that a module or function can't be found but there's no stack trace to point me towards the location so it's really trial end error commenting and uncommenting  include  and  using  statements to see what gives, I get errors from within external modules, etc.   I guess what I'm trying to say is that my experience with parallel computing applied to a large codebase is very frustrating. And I was wondering if anybody has done a parallel implementation in a larger codebase and has any tips on how to attack it.    P.S. I might be spoiled by Elixir, but ideally I'd like to be able to spawn processes and functions whenever I need, and have the compiler take care of making the code available across workers.  Another thing that would be very useful, also inspired by Elixir, is the supervisor design pattern, to look over and restart the workers when they crash.      Adrian Salceanu    Sep 12   This is a random example of an error - not really sure how to debug this, seems to crash within the Postgres library. The dump is long but not really helpful.   12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 5 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 3 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 2 fatal error on fatal error on 3: ERROR: attempt to send to unknown socket 4: ERROR: attempt to send to unknown socket 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 5 fatal error on 3: ERROR: attempt to send to unknown socket 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 fatal error on 2: ERROR: attempt to send to unknown socket ERROR: LoadError: On worker 2: LoadError: On worker 2: LoadError: LoadError: LoadError: LoadError: LoadError: ProcessExitedException    in yieldto at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in wait at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  repeats 3 times   in take! at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  repeats 2 times   in remotecall_fetch at multi-jl:745  in remotecall_fetch at multi-jl:750  in anonymous at multi-jl:1396  ...and 1 other exceptions.   in include_string at loading-jl:282  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in require at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in include_string at loading-jl:282  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in require at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in eval at  Users adrian Dropbox Projects jinnie lib Genie src Database-jl:1  in include_string at loading-jl:282  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in require at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in include_string at loading-jl:282  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in eval at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in anonymous at multi-jl:1394  in run_work_thunk at multi-jl:661  in remotecall_fetch at multi-jl:734  in remotecall_fetch at multi-jl:750  in anonymous at multi-jl:1396 while loading  Users adrian .julia v0.4 PostgreSQL src .. deps build-jl, in expression starting on line 9 while loading  Users adrian .julia v0.4 PostgreSQL src PostgreSQL-jl, in expression starting on line 8 while loading  Users adrian Dropbox Projects jinnie lib Genie database_adapters PostgreSQLDatabaseAdapter-jl, in expression starting on line 3 while loading  Users adrian Dropbox Projects jinnie lib Genie src Database-jl, in expression starting on line 7 while loading  Users adrian Dropbox Projects jinnie lib Genie src commands-jl, in expression starting on line 11  in remotecall_fetch at multi-jl:735  in remotecall_fetch at multi-jl:750  in anonymous at multi-jl:1396  ...and 1 other exceptions.   in sync_end at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in anonymous at multi-jl:1405  in include_string at loading-jl:282  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in require at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in eval at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in anonymous at multi-jl:1394  in anonymous at multi-jl:923  in run_work_thunk at multi-jl:661   inlined code  from multi-jl:923  in anonymous at task-jl:63 while loading  Users adrian Dropbox Projects jinnie lib Genie src Genie-jl, in expression starting on line 1  in remotecall_fetch at multi-jl:747  in remotecall_fetch at multi-jl:750  in anonymous at multi-jl:1396  ...and 4 other exceptions.   in sync_end at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in anonymous at multi-jl:1405  in include at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in include_from_node1 at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in process_options at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib  in _start at  usr local Cellar julia 0.4.6_1 lib julia sys.dylib while loading  Users adrian Dropbox Projects jinnie genie-jl, in expression starting on line 16      dnm    Sep 12   Have you tried Dagger-jl to set up a DAG of computations you need performed?      Adrian Salceanu    Sep 13   Thanks - haven't thought of using Dagger  don't have previous experience with it either . I've read the docs now but I'm not sure how using it would help, maybe I'm missing something?   My problem is really about running deeply nested function calls across multiple modules, in parallel. Making the code available to the workers involves a lot of back tracking, module by module and function by function, prepending  everywhere, well... everywhere. Cherry picking function calls in a few thousands lines codebase to make it load across processes is a task for computers, not people. Is this the idiomatic Julia way, because it really feels like I'm misusing the language? Am I supposed to end up with tens of calls to  everywhere? - it doesn't feel right...