Groups   148 of 99+      julia-users › static compilation 23 posts by 7 authors      Ján Adamčák    May 2   Hello,  Trying to create an executable from Julia source, there are questions that raised... The process of exporting functions wasn't successful, function s  was not exported to dll, and even the generated dll cant be loaded in c++ code. Below is the process explained in details:  Can anybody bring more insight, how to build a standalone executable from Julia source?   Following resources were used for experiments below:  http:  juliacomputing.com blog 2016 02 09 static-julia.html  http:  juliacomputing.com blog 2016 03 10 j2c-announcement.html   Assume userimg-jl, which contains exported function s :   user functionality for DLL  Base.ccallable foo x::Int    x+1   Now, create a dll: julia.exe  build_sysimg-jl   . Julia Compile bin   native  julia_dll-jl  --force    File build_sysimg-jl on line 77 contains following command flags:            run `$julia -C $cpu_target --output-ji $sysimg_path.ji --output-o            $sysimg_path.o --sysimage .. .. .. lib julia sys.dll -J $inference_path.ji            --startup-file no sysimg-jl --compile all --eval nothing`   The resulting dll has 29MB.   The next step is to use the dll in c++ project:  include  iostream   include  windows.h   typedef int  fnc_foo   int ;  int main      HINSTANCE hDLL   LoadLibraryA  test_dll.dll  ;  auto a   GetLastError  ;  if  hDLL    nullptr     throw std::exception  test_dll.dll not found!  ;   fnc_foo foo    fnc_foo GetProcAddress hDLL,  foo  ;  if !foo     throw std::exception  foo in test_dll.dll not found!  ;   std::cout    foo 10      \n ;   FreeLibrary hDLL ;   return 0;    In the c++ code above the dll cannot be loaded, hDLL   nullptr, and the first exception is thrown.  Expecting the dll with Dependency Walker even an exported function foo   cannot be found.  Thanks for any suggestions. Jan     Ján Adamčák    May 6  Translate message to English    Nobody???  Dňa pondelok, 2. mája 2016 11:30:11 UTC+2 Ján Adamčák napísal -a :      Andreas Lobinger    May 6   Hello colleague,  this topic is still seen as experimental and not that many of julia users could be considered expert on this... If the recipe given  long time ago i tried to follow this on a linux installation which i general has more tooling to get shared libraries and compilation working...  doesn't work, you could raise a concrete issue on github - there you get more audience with julia internal know-how.  Wishing a happy day,         Andreas    On Friday, May 6, 2016 at 8:51:47 AM UTC+2, Ján Adamčák wrote: Nobody???  Dňa pondelok, 2. mája 2016 11:30:11 UTC+2 Ján Adamčák napísal -a :  Hello,  Trying to create an executable from Julia source, there are questions that raised... The process of exporting functions wasn't successful, function s  was not exported to dll, and even the generated dll cant be loaded in c++ code. Below is the process explained in details:  Can anybody bring more insight, how to build a standalone executable from Julia source?      Tom Breloff    May 6  Re:  julia-users  Re: static compilation  Also, I think there's even fewer people that are experts on the Windows build process.  I think the community tends to be more linux osx focused.      Jeff Bezanson    May 6  Re:  julia-users  Re: static compilation  That command line in build_sysimg-jl:77 looks strange. Actually I  can't find that line anywhere; our build_sysimg and BuildExecutable-jl  now have something slightly different.       Ján Adamčák    May 6  Re:  julia-users  Re: static compilation  Sorry, my fault.   During last week I created build_sysimg2-jl from julia 0.4.5 build_sysimg-jl and when I run it, at line 86  build_sysimg2-jl   run `$julia -C $cpu_target --output-o sysimg_all.o --sysimage $sysimg_path.$ Libdl.dlext  --startup-file no --compile all --eval nothing`   I got this error:  fatal: error thrown and no exception handler available. ErrorException  Task cannot be serialized  -jl_unprotect_stack at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_throw at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_error at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_save_system_image_to_stream at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_create_system_image at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line -jl_atexit_hook at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll  unknown line  unknown function  ip: 00000000004028B5  unknown function  ip: 000000000040140C  unknown function  ip: 000000000040153B  BaseThreadInitThunk at C:\WINDOWS\system32\KERNEL32.DLL  unknown line  RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll  unknown line  ERROR: LoadError: failed process: Process `'C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\julia' -C native --output-o sysimg_all.o --sysimage my_img.dll --startup-file no --compile all --eval nothing`, ProcessExited 1    1   in run at process-jl:531 while loading c:\Users\Adam\AppData\Local\Julia-0.4.5\share\julia\build_sysimg2-jl, in expression starting on line 191   Before this error, julia wrote some warnings:  WARNING: could not attach metadata for  simd loop.  Same error I got on Ubuntu 16.04...     Dňa piatok, 6. mája 2016 15:48:19 UTC+2 Jeff Bezanson napísal -a :  Attachments  1   build_sysimg2-jl 8 KB     Download     Ján Adamčák    May 10  Re:  julia-users  Re: static compilation  Hello guys,   Thank you for your comments, though we were more optimistic...   Dňa piatok, 6. mája 2016 16:27:28 UTC+2 Ján Adamčák napísal -a :      Jameson    May 10  Re:  julia-users  Re: static compilation  The compile-all flag is only partially functional on v0.4. I think it's best to just leave it off. I tested on master and fixed a bug with emitting ` ccallable`, but that's unrelated. From the command line below, it looks like you are not adding any code to the system image  `--eval nothing`  which would also means there are no `ccallable` declarations being emitted into the current compile.  Other than that, I don't see anything wrong with that command. You shouldn't see that error unless you tried to make a Task or use ` async` from the compile host. It's ambiguous how that would be serialized, so it's simply an error and any parallel workers should be created   started by an `__init__` method.      Ján Adamčák    May 12  Re:  julia-users  Re: static compilation  Thanks  Jameson,  I am a bit confused about  you are not adding any code to the system image  `--eval nothing`  . According to your blog http:  juliacomputing.com blog 2016 02 09 static-julia.html , I think that this is a crucial point to obtain a small sized dll. Am I right?  What is then the right way to emit  ccallable  declarations in order to export julia function s ?  foo in our example from the original post in this thread   Is it okay to work with current version of julia 0.4.5. or I have to switch to another version; If yes, to which one?  Thanks in advance.  Dňa utorok, 10. mája 2016 22:13:57 UTC+2 Jameson napísal -a :      Jameson    May 12  Re:  julia-users  Re: static compilation  We have been working on a number of simplifications on master, so some of the best practices and extra steps aren't necessary anymore. But it should still work on v0.4.  There are a few different goals that can be accomplished by invoking the Julia compiler directly, so it was a bit difficult to write that blog post talking about them all generically. Since it touches on several of the optimization options, I structured it in part to show how these layers can build on each other. But I decided to leave out demonstrations of how mixing various layers and options can be used to create other products.  Since most of these steps are already configured in the Julia build system, one of the easiest ways to augment it is to simply drop a userimg-jl file into base   This will then get incorporated into the usual build and become part of the pre-defined system image.  The `-e nothing` stage is there because you have to give it something to evaluate  a file, stdin, or `-e`, etc. , or it will pop open the REPL and wait for the user to enter commands. This is actually also a valid way to create an executable and can be fun to play with as a development exercise  I still do this on occasion to test out how it is handling odd cases .  To get a ccallable declaration to show up in the binary, the only condition is that you must declare it ccallable in the same execution step as the final output.  -jameson      Ján Adamčák    May 13  Re:  julia-users  Re: static compilation  Thanks  Jameson,   I have successfully built .so with  --compile all  flag on 0.5-dev on ubuntu 16.04, but my .so library is 110MB. The same compilation on Win10 crashes. I can call my function in c++ code using-jl_init and-jl_get_function.  My primary goal is to compile a dll with my own exported function, which I can call without-jl_init  . I think it has something to do with ccallable, which you have mentioned. I'm stacked at this point. Could you please explain in more details, or point me to Julia code, how to move forward?  Thanks in advance. -jan  Dňa piatok, 13. mája 2016 4:30:41 UTC+2 Jameson napísal -a :      Jameson    May 13  Re:  julia-users  Re: static compilation    without-jl_init    That is not implemented at this time, although patches are welcome.     it has something to do with ccallable  Yes, it also is orthogonal to compile-all. It is possible that compile-all is non-functional on v0.4 on Windows, I know master has many enhancements, which may have included more stable Windows support. I suggest playing with these two options independently before jumping into combining them.       Ján Adamčák    May 18  Re:  julia-users  Re: static compilation  Thanks  Jameson  Do you have an idea, how the compilation will work on v0.5 release? I have tried with recent commits of master, but  ccallable doesn't work.    Dňa sobota, 14. mája 2016 1:17:45 UTC+2 Jameson napísal -a :      Jameson    May 18  Re:  julia-users  Re: static compilation  You might need to be more specific than  doesn't work ; I tried it last week and it worked for me. The usage is essentially the same  the macro in v0.5 takes an extra argument of the return type, but the deprecation warning should note this already .      Ján Adamčák    May 19  Re:  julia-users  Re: static compilation  Thanks  Jameson  The problem with  ccallable is, I get following error Version 0.5.0-dev+4124  2016-05-16 22:35 UTC  :  julia   Base.ccallable  Int64, foo x::Int64    x+1  ERROR: expected method definition in  ccallable  in eval ::Module, ::Any  at .\boot-jl:226  Thanks  Dňa streda, 18. mája 2016 18:46:31 UTC+2 Jameson napísal -a :      Isaiah    May 19  Re:  julia-users  Re: static compilation  It looks like there might be a parser ambiguity when the arguments are given as a tuple  the whole thing is parsed as keyword argument to the macro call .  Either of the following will work right now:       Base.ccallable Int64 foo x::Int64    x+1      Base.ccallable Int64, function foo x::Int64  x+1 end   Please file a bug report with the signature you tried.      Jameson    May 22  Re:  julia-users  Re: static compilation  I tried building a new system image on v0.4 starting from an existing sys.so library  as you tried below , and found that it failed with the same  Task cannot be serialized  error.  So I guess that use case must have been fixed during v0.5 development. The workaround is to start with inference.ji and load all of the extra code through userimg-jl. That worked for me  and is just a bit slower for incremental development , so I didn't investigate further.      Ján Adamčák    May 23  Re:  julia-users  Re: static compilation  Thanks  Jameson,  Another error I'm getting while compiling c example is  C:  Users  Adam  AppData  Local  Julia-0.5.0-dev  start_func.c:111: undefined reference to `__imp-jl_tls_states' collect2.exe: error: ld returned 1 exit status  This error is caused by JL_GC_PUSH1 &x ;   Is there some workaround or is it just a bug?  Thanks.  Dňa pondelok, 23. mája 2016 4:56:52 UTC+2 Jameson napísal -a :      Yichao Yu    May 23  Re:  julia-users  Re: static compilation   On May 23, 2016 10:30 AM,  Ján Adamčák   jada... gmail.com  wrote:     Thanks  Jameson,     Another error I'm getting while compiling c example is     C:  Users  Adam  AppData  Local  Julia-0.5.0-dev  start_func.c:111: undefined reference to `__imp-jl_tls_states'   collect2.exe: error: ld returned 1 exit status     This error is caused by JL_GC_PUSH1 &x ;   Looks like you are missing threading enabling macro while building your code       Ján Adamčák    May 23  Re:  julia-users  Re: static compilation  Thanks Yichao Yu,  Can you tell me which macro did you mean and where to place it?  Thanks  Dňa 23.5.2016 17:37 používateľ  Yichao Yu   yyc... gmail.com  napísal:      Yichao Yu    May 23  Re:  julia-users  Re: static compilation  On Mon, May 23, 2016 at 12:09 PM, Ján Adamčák  jada... gmail.com  wrote:    Thanks Yichao Yu,       Can you tell me which macro did you mean and where to place it?   You need -DJULIA_ENABLE_THREADING 1 when compiling the Clanguage Cpp code if  julia is built with threading enabled  which is the default now .       Ján Adamčák    May 24  Re:  julia-users  Re: static compilation  Thanks Yichao Yu,  Your macro works fine.  So in the next step I tried the following flag:  --compile all  during system image compilation.  My userimg-jl:   Base.ccallable Int64-jl_foo x::Int64     x+1   with the following Clanguage code:   include  julia.h   include  stdio.h   include  math.h   include  time.h   int main int argc, char  argv        -jl_options.compile_enabled   JL_OPTIONS_COMPILE_OFF;   -jl_options.startupfile   JL_OPTIONS_STARTUPFILE_OFF;   -jl_init_with_image NULL,  libmyimg_all.dll  ;   -jl_function_t  func  -jl_get_function-jl_main_module, -jl_foo  ;   -jl_value_t  argument  -jl_box_int64 2 ;   -jl_value_t  ret  -jl_call1 func, argument ;    if -jl_is_int64 ret            int64_t retInt64  -jl_unbox_int64 ret ;        printf -jl_foo 2  in C:  d\n , retInt64 ;          system  PAUSE  ;    int ret   0;   -jl_atexit_hook ret ;    return ret;      I got this message:  C:\\Users\\Adam\\AppData\\Local\\Julia-0.5.0-dev1\\bin\\my_prog.exe code missing for Base.   Base.   , Base.Cstring, Ptr Void    sysimg may not have been built with --compile all code missing for Base.   Base.   , Ptr UInt8 , Ptr Void    sysimg may not have been built with --compile all code missing for Base.pointer Base. pointer, Array UInt8, 1 , UInt64   sysimg may not have been built with --compile all code missing for Base.unsafe_convert Base. unsafe_convert, Type Ptr UInt8  , Array UInt8, 1    sysimg may not have been built with --compile all code missing for Base.- Base. -, UInt64, Int64   sysimg may not have been built with --compile all code missing for Base.promote Base. promote, UInt64, Int64   sysimg may not have been built with --compile all code missing for Base.elsize Base. elsize, Array UInt8, 1    sysimg may not have been built with --compile all code missing for Base.+ Base. +, Ptr UInt8 , UInt64   sysimg may not have been built with --compile all code missing for Base.rem Base. rem, UInt64, Type UInt64    sysimg may not have been built with --compile all code missing for Base.oftype Base. oftype, Ptr UInt8 , UInt64   sysimg may not have been built with --compile all code missing for Base.convert Base. convert, Type Ptr UInt8  , UInt64   sysimg may not have been built with --compile all code missing for Base.convert Base. convert, Type Ptr Int32  , Ptr UInt8    sysimg may not have been built with --compile all code missing for Base.SparseArrays.CHOLMOD.set_print_level Base.SparseArrays.CHOLMOD. set_print_level, Array UInt8, 1 , Int64   sysimg may not have been built with --compile all code missing for Base.unsafe_store! Base. unsafe_store!, Ptr Int32 , Int64   sysimg may not have been built with --compile all-jl_foo 2  in C: 3 Press any key to continue . . .  Execution of my function was successful, but I think I found bug in --compile all implementation:  code missing for ...  sysimg may not have been built with --compile all  What do you think???  Thanks.   Dňa pondelok, 23. mája 2016 18:42:20 UTC+2 Yichao Yu napísal -a :      Ján Adamčák    May 26  Re:  julia-users  Re: static compilation  I reported it as bug.   16595