Groups


148 of 99+  



julia-users ›
static compilation
23 posts by 7 authors  



Ján Adamčák 	

May 2


Hello,

Trying to create an executable from Julia source, there are questions that raised... The process of exporting functions wasn't successful, function(s) was not exported to dll, and even the generated dll cant be loaded in c++ code. Below is the process explained in details:

Can anybody bring more insight, how to build a standalone executable from Julia source?


Following resources were used for experiments below:

http://juliacomputing.com/blog/2016/02/09/static-julia.html

http://juliacomputing.com/blog/2016/03/10/j2c-announcement.html


Assume userimg.jl, which contains exported function(s):
# user functionality for DLL
@Base.ccallable foo(x::Int) = x+1


Now, create a dll:
julia.exe "build_sysimg.jl" "./Julia/Compile/bin/" native "julia_dll.jl" --force



File build_sysimg.jl on line 77 contains following command flags:
           run(`$julia -C $cpu_target --output-ji $sysimg_path.ji --output-o
           $sysimg_path.o --sysimage ../../../lib/julia/sys.dll -J $inference_path.ji
           --startup-file=no sysimg.jl --compile=all --eval nothing`)

The resulting dll has 29MB.


The next step is to use the dll in c++ project:
#include <iostream>
#include <windows.h>

typedef int(*fnc_foo) (int);

int main()
{
 HINSTANCE hDLL = LoadLibraryA("test_dll.dll");
 auto a = GetLastError();
 if (hDLL == nullptr)
   throw std::exception("test_dll.dll not found!");

 fnc_foo foo = (fnc_foo)GetProcAddress(hDLL, "foo");
 if(!foo)
   throw std::exception("foo in test_dll.dll not found!");

 std::cout << foo(10) << "\n";

 FreeLibrary(hDLL);

 return 0;
}

In the c++ code above the dll cannot be loaded, hDLL = nullptr, and the first exception is thrown.

Expecting the dll with Dependency Walker even an exported function foo() cannot be found.

Thanks for any suggestions.
Jan
 


Ján Adamčák 	

May 6

Translate message to English  

Nobody???

Dňa pondelok, 2. mája 2016 11:30:11 UTC+2 Ján Adamčák napísal(-a):
- show quoted text -
 


Andreas Lobinger 	

May 6


Hello colleague,

this topic is still seen as experimental and not that many of julia users could be considered expert on this...
If the recipe given (long time ago i tried to follow this on a linux installation which i general has more tooling to get shared libraries and compilation working...) doesn't work, you could raise a concrete issue on github - there you get more audience with julia internal know-how.

Wishing a happy day,

       Andreas



On Friday, May 6, 2016 at 8:51:47 AM UTC+2, Ján Adamčák wrote:
Nobody???

Dňa pondelok, 2. mája 2016 11:30:11 UTC+2 Ján Adamčák napísal(-a):

Hello,

Trying to create an executable from Julia source, there are questions that raised... The process of exporting functions wasn't successful, function(s) was not exported to dll, and even the generated dll cant be loaded in c++ code. Below is the process explained in details:

Can anybody bring more insight, how to build a standalone executable from Julia source?

 


Tom Breloff 	

May 6

Re: [julia-users] Re: static compilation

Also, I think there's even fewer people that are experts on the Windows build process.  I think the community tends to be more linux/osx focused.
- show quoted text -
 


Jeff Bezanson 	

May 6

Re: [julia-users] Re: static compilation

That command line in build_sysimg.jl:77 looks strange. Actually I 
can't find that line anywhere; our build_sysimg and BuildExecutable.jl 
now have something slightly different. 
- show quoted text -
 


Ján Adamčák 	

May 6

Re: [julia-users] Re: static compilation

Sorry, my fault. 

During last week I created build_sysimg2.jl from julia 0.4.5 build_sysimg.jl and when I run it, at line 86 (build_sysimg2.jl)

run(`$julia -C $cpu_target --output-o sysimg_all.o --sysimage $sysimg_path.$(Libdl.dlext) --startup-file=no --compile=all --eval nothing`)

I got this error:

fatal: error thrown and no exception handler available.
ErrorException("Task cannot be serialized")
jl_unprotect_stack at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_throw at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_error at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_compress_ast at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_save_system_image_to_stream at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_create_system_image at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
jl_atexit_hook at C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\libjulia.dll (unknown line)
unknown function (ip: 00000000004028B5)
unknown function (ip: 000000000040140C)
unknown function (ip: 000000000040153B)
BaseThreadInitThunk at C:\WINDOWS\system32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
ERROR: LoadError: failed process: Process(`'C:\Users\Adam\AppData\Local\Julia-0.4.5\bin\julia' -C native --output-o sysimg_all.o --sysimage my_img.dll --startup-file=no --compile=all --eval nothing`, ProcessExited(1)) [1]
 in run at process.jl:531
while loading c:\Users\Adam\AppData\Local\Julia-0.4.5\share\julia\build_sysimg2.jl, in expression starting on line 191

 Before this error, julia wrote some warnings:

WARNING: could not attach metadata for @simd loop.

Same error I got on Ubuntu 16.04...
 


Dňa piatok, 6. mája 2016 15:48:19 UTC+2 Jeff Bezanson napísal(-a):
- show quoted text -
Attachments (1)

build_sysimg2.jl
8 KB     Download
 


Ján Adamčák 	

May 10

Re: [julia-users] Re: static compilation

Hello guys, 

Thank you for your comments, though we were more optimistic...


Dňa piatok, 6. mája 2016 16:27:28 UTC+2 Ján Adamčák napísal(-a):
- show quoted text -
 


Jameson 	

May 10

Re: [julia-users] Re: static compilation

The compile-all flag is only partially functional on v0.4. I think it's best to just leave it off. I tested on master and fixed a bug with emitting `@ccallable`, but that's unrelated. From the command line below, it looks like you are not adding any code to the system image (`--eval nothing`) which would also means there are no `ccallable` declarations being emitted into the current compile.

Other than that, I don't see anything wrong with that command. You shouldn't see that error unless you tried to make a Task or use `@async` from the compile host. It's ambiguous how that would be serialized, so it's simply an error and any parallel workers should be created / started by an `__init__` method.
- show quoted text -
 


Ján Adamčák 	

May 12

Re: [julia-users] Re: static compilation

Thanks @Jameson,

I am a bit confused about "you are not adding any code to the system image (`--eval nothing`)". According to your blog http://juliacomputing.com/blog/2016/02/09/static-julia.html , I think that this is a crucial point to obtain a small sized dll. Am I right?

What is then the right way to emit "ccallable" declarations in order to export julia function(s)? (foo in our example from the original post in this thread)

Is it okay to work with current version of julia 0.4.5. or I have to switch to another version; If yes, to which one?

Thanks in advance.

Dňa utorok, 10. mája 2016 22:13:57 UTC+2 Jameson napísal(-a):
- show quoted text -
 


Jameson 	

May 12

Re: [julia-users] Re: static compilation

We have been working on a number of simplifications on master, so some of the best practices and extra steps aren't necessary anymore. But it should still work on v0.4.

There are a few different goals that can be accomplished by invoking the Julia compiler directly, so it was a bit difficult to write that blog post talking about them all generically. Since it touches on several of the optimization options, I structured it in part to show how these layers can build on each other. But I decided to leave out demonstrations of how mixing various layers and options can be used to create other products.

Since most of these steps are already configured in the Julia build system, one of the easiest ways to augment it is to simply drop a userimg.jl file into base/ 
This will then get incorporated into the usual build and become part of the pre-defined system image.

The `-e nothing` stage is there because you have to give it something to evaluate (a file, stdin, or `-e`, etc.), or it will pop open the REPL and wait for the user to enter commands. This is actually also a valid way to create an executable and can be fun to play with as a development exercise (I still do this on occasion to test out how it is handling odd cases).

To get a ccallable declaration to show up in the binary, the only condition is that you must declare it ccallable in the same execution step as the final output.

-jameson
- show quoted text -
 


Ján Adamčák 	

May 13

Re: [julia-users] Re: static compilation

Thanks @Jameson,


I have successfully built .so with "--compile=all" flag on 0.5-dev on ubuntu 16.04, but my .so library is 110MB. The same compilation on Win10 crashes. I can call
my function in c++ code using jl_init and jl_get_function.

My primary goal is to compile a dll with my own exported function, which I can call without jl_init(). I think it has something to do with ccallable, which you have mentioned. I'm stacked at this point. Could you please explain in more details, or point me to Julia code, how to move forward?

Thanks in advance.
-jan

Dňa piatok, 13. mája 2016 4:30:41 UTC+2 Jameson napísal(-a):
- show quoted text -
 


Jameson 	

May 13

Re: [julia-users] Re: static compilation

> without jl_init()

That is not implemented at this time, although patches are welcome. 

> it has something to do with ccallable

Yes, it also is orthogonal to compile-all. It is possible that compile-all is non-functional on v0.4 on Windows, I know master has many enhancements, which may have included more stable Windows support. I suggest playing with these two options independently before jumping into combining them. 
- show quoted text -
 


Ján Adamčák 	

May 18

Re: [julia-users] Re: static compilation

Thanks @Jameson

Do you have an idea, how the compilation will work on v0.5 release? I have tried with recent commits of master, but @ccallable doesn't work.



Dňa sobota, 14. mája 2016 1:17:45 UTC+2 Jameson napísal(-a):
- show quoted text -
 


Jameson 	

May 18

Re: [julia-users] Re: static compilation

You might need to be more specific than "doesn't work"; I tried it last week and it worked for me. The usage is essentially the same (the macro in v0.5 takes an extra argument of the return type, but the deprecation warning should note this already).
- show quoted text -
 


Ján Adamčák 	

May 19

Re: [julia-users] Re: static compilation

Thanks @Jameson

The problem with @ccallable is, I get following error(Version 0.5.0-dev+4124 (2016-05-16 22:35 UTC)):

julia> @Base.ccallable (Int64, foo(x::Int64) = x+1)
ERROR: expected method definition in @ccallable
 in eval(::Module, ::Any) at .\boot.jl:226

Thanks

Dňa streda, 18. mája 2016 18:46:31 UTC+2 Jameson napísal(-a):
- show quoted text -
 


Isaiah 	

May 19

Re: [julia-users] Re: static compilation

It looks like there might be a parser ambiguity when the arguments are given as a tuple (the whole thing is parsed as keyword argument to the macro call).

Either of the following will work right now:

    @Base.ccallable Int64 foo(x::Int64) = x+1
    @Base.ccallable(Int64, function foo(x::Int64) x+1 end)

Please file a bug report with the signature you tried.
- show quoted text -
 


Jameson 	

May 22

Re: [julia-users] Re: static compilation

I tried building a new system image on v0.4 starting from an existing sys.so library (as you tried below), and found that it failed with the same "Task cannot be serialized" error.  So I guess that use case must have been fixed during v0.5 development. The workaround is to start with inference.ji and load all of the extra code through userimg.jl. That worked for me (and is just a bit slower for incremental development), so I didn't investigate further.
- show quoted text -
 


Ján Adamčák 	

May 23

Re: [julia-users] Re: static compilation

Thanks @Jameson,

Another error I'm getting while compiling c example is

C://Users//Adam//AppData//Local//Julia-0.5.0-dev//start_func.c:111: undefined reference to `__imp_jl_tls_states'
collect2.exe: error: ld returned 1 exit status

This error is caused by JL_GC_PUSH1(&x); 

Is there some workaround or is it just a bug?

Thanks.

Dňa pondelok, 23. mája 2016 4:56:52 UTC+2 Jameson napísal(-a):
- show quoted text -
 


Yichao Yu 	

May 23

Re: [julia-users] Re: static compilation


On May 23, 2016 10:30 AM, "Ján Adamčák" <jada...@gmail.com> wrote:
>
> Thanks @Jameson,
>
> Another error I'm getting while compiling c example is
>
> C://Users//Adam//AppData//Local//Julia-0.5.0-dev//start_func.c:111: undefined reference to `__imp_jl_tls_states'
> collect2.exe: error: ld returned 1 exit status
>
> This error is caused by JL_GC_PUSH1(&x); 

Looks like you are missing threading enabling macro while building your code

- show quoted text -
 


Ján Adamčák 	

May 23

Re: [julia-users] Re: static compilation

Thanks Yichao Yu,

Can you tell me which macro did you mean and where to place it?

Thanks

Dňa 23.5.2016 17:37 používateľ "Yichao Yu" <yyc...@gmail.com> napísal:
- show quoted text -
 


Yichao Yu 	

May 23

Re: [julia-users] Re: static compilation

On Mon, May 23, 2016 at 12:09 PM, Ján Adamčák <jada...@gmail.com> wrote: 
> Thanks Yichao Yu, 
> 
> Can you tell me which macro did you mean and where to place it? 

You need -DJULIA_ENABLE_THREADING=1 when compiling the C/C++ code if 
julia is built with threading enabled (which is the default now). 
- show quoted text -
 


Ján Adamčák 	

May 24

Re: [julia-users] Re: static compilation

Thanks Yichao Yu,

Your macro works fine.

So in the next step I tried the following flag: "--compile=all" during system image compilation.

My userimg.jl:

@Base.ccallable Int64 jl_foo(x::Int64) = (x+1)

with the following C code:

#include <julia.h>
#include <stdio.h>
#include <math.h>
#include <time.h>

int main(int argc, char *argv[])
{
   jl_options.compile_enabled = JL_OPTIONS_COMPILE_OFF;
   jl_options.startupfile = JL_OPTIONS_STARTUPFILE_OFF;
   jl_init_with_image(NULL, "libmyimg_all.dll");
   jl_function_t *func = jl_get_function(jl_main_module, "jl_foo");
   jl_value_t* argument = jl_box_int64(2);
   jl_value_t* ret = jl_call1(func, argument);
   if (jl_is_int64(ret)) {
       int64_t retInt64 = jl_unbox_int64(ret);
       printf("jl_foo(2) in C: %d\n", retInt64);
    }
   system("PAUSE");
   int ret = 0;
   jl_atexit_hook(ret);
   return ret;
}
	

I got this message:

C:\\Users\\Adam\\AppData\\Local\\Julia-0.5.0-dev1\\bin\\my_prog.exe
code missing for Base.==(Base.#==, Base.Cstring, Ptr{Void})  sysimg may not have been built with --compile=all
code missing for Base.==(Base.#==, Ptr{UInt8}, Ptr{Void})  sysimg may not have been built with --compile=all
code missing for Base.pointer(Base.#pointer, Array{UInt8, 1}, UInt64)  sysimg may not have been built with --compile=all
code missing for Base.unsafe_convert(Base.#unsafe_convert, Type{Ptr{UInt8}}, Array{UInt8, 1})  sysimg may not have been built with --compile=all
code missing for Base.-(Base.#-, UInt64, Int64)  sysimg may not have been built with --compile=all
code missing for Base.promote(Base.#promote, UInt64, Int64)  sysimg may not have been built with --compile=all
code missing for Base.elsize(Base.#elsize, Array{UInt8, 1})  sysimg may not have been built with --compile=all
code missing for Base.+(Base.#+, Ptr{UInt8}, UInt64)  sysimg may not have been built with --compile=all
code missing for Base.rem(Base.#rem, UInt64, Type{UInt64})  sysimg may not have been built with --compile=all
code missing for Base.oftype(Base.#oftype, Ptr{UInt8}, UInt64)  sysimg may not have been built with --compile=all
code missing for Base.convert(Base.#convert, Type{Ptr{UInt8}}, UInt64)  sysimg may not have been built with --compile=all
code missing for Base.convert(Base.#convert, Type{Ptr{Int32}}, Ptr{UInt8})  sysimg may not have been built with --compile=all
code missing for Base.SparseArrays.CHOLMOD.set_print_level(Base.SparseArrays.CHOLMOD.#set_print_level, Array{UInt8, 1}, Int64)  sysimg may not have been built with --compile=all
code missing for Base.unsafe_store!(Base.#unsafe_store!, Ptr{Int32}, Int64)  sysimg may not have been built with --compile=all
jl_foo(2) in C: 3
Press any key to continue . . .

Execution of my function was successful, but I think I found bug in --compile=all implementation:

code missing for ...  sysimg may not have been built with --compile=all

What do you think???

Thanks.


Dňa pondelok, 23. mája 2016 18:42:20 UTC+2 Yichao Yu napísal(-a):
- show quoted text -
 


Ján Adamčák 	

May 26

Re: [julia-users] Re: static compilation

I reported it as bug.

#16595