Groups 100 of 99+ julia-users › What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. 14 posts by 6 authors Páll Haraldsson 9/23/15 [I have friends who might benefit from Julia.] Say for, quants/finance and bio (genome sequencing and systems biology). And for me, for web use (what is recommended, Escher; Mux, isn't that Sinatra-style? Nothing available similar to Django/Rails? Or needed? Would Python/Django and Julia work well together? Anyone tried?) [I just do not want to be "promoting" Julia too much if the libraries are hopelessly broken/immature, and I can't judge myself in very specific cases. Even is they could help, but better to know then. Pointers to good libraries in these three areas that you trust would be good.] I pretty much think I know the strengths of Julia - as a language, but people want to get work done, and libraries matter. I'm not just asking for existence, I'm aware of a lot of libraries in these and other areas, I'm thinking what is the coverage of those areas and are they beta quality (and help needed?)? I don't know maybe the questions about these areas need to be more targeted? Should I ask the what they do/need as I'm ignorant of that..? I'm also just curious in general how big the landscape of powerful libraries is.. I understand e.g the optimization libraries are best in class.. [My quant (physicist friend used Python/Numpy/C++, and the systems biology one MATLAB/C++. I guess if there are gaps you can easily use the other languages together to fill them, less so with MATLAB?] Thanks, -- Palli Randy Zwitch 9/23/15 Julia is as capable as any of the languages you have mentioned as far as I'm concerned. When I read "people want to get work done", I read that as "people want SOMEONE ELSE to do the work". Julia probably isn't the place for them now, if they are looking for someone else to have already provided them with every package with every functionality they would ever want. R, Python, MATLAB, C/C++, Java, Scala, PHP, JavaScript are all viable substitutes in different ways. But if someone is the type of person who likes to build things and collaborate with smart people, then the Julia community will welcome you with open arms. - show quoted text - Páll Haraldsson 9/23/15 On Wednesday, September 23, 2015 at 12:35:47 PM UTC, Randy Zwitch wrote: Julia is as capable as any of the languages you have mentioned as far as I'm concerned. When I read "people want to get work done", I read that as "people want SOMEONE ELSE to do the work". And you would be absolutely right. I tried to phrase the question in a positive way with "and help needed?" [For me, that would be mostly non-math stuff*, and I've submitted some trivial/beginner.. fixes.] I'm ok with that as I am just tinkering. Imagine Julia had no libraries, as at first then I would have been as exited about the language. It is a language that makes me think differently and try new paradigms I haven't tried before (multiple dispatch). I might have tried to build a website (and web server from scratch). Some people do not want to be early adopters. I can understand that. I'm not so sure you would be by now. I'm asking about the "ecosystem" not the language per se. I know about JuliaQuant, BioJulia, GPU stuff in Julia JuliaWeb etc. I am so grateful for what has already been done with the language - and the libraries from what I can see. If there where my fields, I think I would jump on Julia right now. I'm not sure why people are reluctant, I want to tell them you do not only have basic building blocks (linear algebra/matrix multiplication, FFT etc. stuff in Base), but also these libraries that (mostly) work, and if not you can help fix/contribute. I do not want to oversell Julia, so I keep quiet (mostly) about stuff I'm ignorant about.. * I knew about say, Morsel (Sinatra-like), then Mux is recommended over it. I'm not sure, it seems to be a replacement/also Sinatra-style. I've never used "full web frameworks". PHP isn't my favorite language and while I'm sure Python (or Ruby) is nice for web stuff I'm willing to use Julia even if there is (short term) pain/learning experience.. I would want to be able to do what is needed in pure Julia. Even knowing about future direction is helpful, there might be some duplication of effort and you might end up fixing the wrong package.. A list of packages to use/focus on for helping with would be helpful in this and the other two areas. > Julia probably isn't the place for them now Are at least some of the packages ready and used in production already? -- Palli. Andrei Zh 9/23/15 If you are looking for a best in the class libraries, you probably won't find many. This is implied by a simple fact that most such libraries had already been created in other languages by the time Julia was born. However, if you want something comparable to such best libraries, then I would stress the following areas (from my experience and highly subjectively, of course): * image processing (e.g. Images.jl, ImageView.jl), which still changes, but has quite impressive functionality already * deep learning (e.g. Mocha.jl, Strada.jl, Boltzmann.jl) - fast, full functional, easy to use and modify libraries (compare to frameworks in C++ or Theano, for example) * concurrent, parallel and distributed programming (core Julia) - far behind Python or R, probably comparable with Erlang * GPU computing (see JuliaGPU organization) - pretty convenient, especially combined with Julia's compilation to native code * symbolic and metaprogramming (macros, Calculus.jl) - like Lisp with infix notation or SymPy, built in the language I also expect that Julia will become more popular with development of new areas for which there are no good libraries at all and Julia may become perfect solution. At the same time, to keep people involved, we not only need to add more strengths, but also remove weaknesses. And Julia's web stack seems to be one of the biggest weaknesses, so if you are interested and wish to contribute, please, do it. - show quoted text - David Anthoff 9/23/15 RE: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. Optimization should definitely be on this list. The JuMP package is just phenomenal, in my mind a much better overall experience for many problems than any existing alternative. From: julia...@googlegroups.com [mailto:julia...@googlegroups.com] On Behalf Of Andrei Zh Sent: Wednesday, September 23, 2015 1:58 PM To: julia-users Subject: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. - show quoted text - Jonathan Malmaud 9/23/15 The webstack has seen considerable improvement lately. Mux is the most mature and supported webapp framework at this point. - show quoted text - Andrei Zh 9/24/15 It's great that webstack has got many improvements recently, but as far as I can see even more job is still to be done. E.g. for me 2 kinds of web apps that I need most often are web UIs and high-performance web services. I'm not sure about performance (last time I tested HttpServer it was quite moderate, but maybe it has changed already), but for web UIs I miss at least following features (taking Mux as the basis): - template engine: Mustache.jl can probably be used, but so far Google knows about zero common occurrences of Mustache.jl and Mux.jl except for very general lists of frameworks - serving static files: possible to do in pure Julia, of course, but it's another several hours to implement - sessions: middleware is exactly for this kind of things, but again, it's better to have it out of the box, than write everything yourself - authentication and security: how to set up HTTPS? how to restrict access to certain pages? - stability: I've just knew that Morsel.jl is now deprecated, if I had applications using it, I would need to migrate them now, and I'm really not sure Mux.jl won't be deprecated during next year too This means that if you want to provide users with a nice interface to your Julia application, you should either spend a couple of days adding missing stuff (and probably not in way suitable for other users) or just use, say, Python and do the job in a couple of hours. - show quoted text - Jonathan Malmaud 9/24/15 Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. I agree with all that - there isn't a web framework for Julia that is at the level of something Django or RoR. It seems totally reasonable to use those mature tools for the frontend of your webapp, which could in term communicate with a Julia backend. I just meant that some of the lower levels of the stack, like implementation for the full HTTP spec, proper handling of unicode and binary data at the HTTP level, solid SSL support, etc is good now. HttpServer.jl includes examples of setting up HTTPS. - show quoted text - Páll Haraldsson 9/24/15 On Wednesday, September 23, 2015 at 8:58:01 PM UTC, Andrei Zh wrote: If you are looking for a best in the class libraries, you probably won't find many. This is implied by a simple fact that most such libraries had already been created in other languages by the time Julia was born. However, if you want something comparable to such best libraries, then I would stress the following areas (from my experience and highly subjectively, of course): * image processing (e.g. Images.jl, ImageView.jl), which still changes, but has quite impressive functionality already * deep learning (e.g. Mocha.jl, Strada.jl, Boltzmann.jl) - fast, full functional, easy to use and modify libraries (compare to frameworks in C++ or Theano, for example) * concurrent, parallel and distributed programming (core Julia) - far behind Python or R, probably comparable with Erlang I find this statement highly surprising.. wander if you meant to reverse this.. My quant friend who had worked for years in Python had trouble parallelizing Python code (may be resolved now..). I'm not familiar with R, but Python has the GIL and associated problems. I also thought Erlang was best-in-class (for concurrent, not "parallel").. @Malmaud About "Mux is the most mature and supported webapp framework at this point." I should check out, but got excited about http://escher-jl.org/ Are you saying that Escher may just be immature at this point? I understood it was high/higher level [than Mux], but compatibility between browsers (e.g. to Safari) not a given yet. I assume, that is a JavaScript generation issue, while Mux doesn't even have that(?) and assume you need to provide all your client-side "content" yourself? @Anthoff "The JuMP package is just phenomenal", yes, that I heard and JuMP is what had in mind. And I understand it's pure Julia, while it "currently supports a number of open-source and commercial solvers". Back to "best in the class libraries, you probably won't find many". In many cases at least, people do not care if pure Julia libraries are available. Often just having libraries for relevant things/wrappers for libraries in C/C++ etc., that they can build on is great. From that perspective, I'm very optimistic Julia is very usable for all kinds of stuff already. For those in the Java/Scala world, I'm less sure about reusing that, I know you can with JavaCall.jl, but understand there are bugs and limitations to it. - show quoted text - Páll Haraldsson 9/24/15 Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. On fim 24.sep 2015 13:25, Jonathan Malmaud wrote: I agree with all that - there isn't a web framework for Julia that is at the level of something Django or RoR. It seems totally reasonable to use those mature tools for the frontend of your webapp, which could in term communicate with a Julia backend. Sounds good. I wasn't really sure if using with Python/Django was recommended. Python is well supported with PyCall.jl but frameworks call you (the "Hollywood principle": "don't call use, we'll call you") unlike regular libraries, so I guess you have to mess with callbacks. OR you use use Python as your main language and call Julia from it with: https://github.com/JuliaLang/pyjulia [that uses PyCall indirectly, that you have to install first] I wander which is preferred (is pyjulia no longer buggy/inferior to PyCall? Much [less] used?) and if you use the second option you end up not using Julia much and might never migrate fully to Julia.. [That could be ok though.] About Ruby on Rails you can use that and call Julia, with RoR_julia_eg (haven't heard of calling in the other direction). It may not be as good as using Django, as Python will work in the same address space as Julia and allows zero-copy. It however may not be too important as: are (individual) web page that speed-critical, (and would have to share that much data between the languages)? I just meant that some of the lower levels of the stack, like implementation for the full HTTP spec, proper handling of unicode and binary data at the HTTP level, solid SSL support, etc is good now. HttpServer.jl includes examples of setting up HTTPS. On Thu, Sep 24, 2015 at 9:16 AM, Andrei Zh wrote: It's great that webstack has got many improvements recently, but as far as I can see even more job is still to be done. E.g. for me 2 kinds of web apps that I need most often are web UIs and high-performance web services. I'm not sure about performance (last time I tested HttpServer it was quite moderate, https://en.wikipedia.org/wiki/Julia_(programming_language)#For_web_use "HttpServer.jl, has low latency 0.5 ms and high throughput (latency on the same order of Python's Flask and Scala's Spray mature frameworks, and throughput also comparable[82])." I'm not sure if Julia's web page is missing something with the most important web (or other..) libraries. Possibly session management, if it exists: but maybe it has changed already), but for web UIs I miss at least following features (taking Mux as the basis): - template engine: Mustache.jl can probably be used, but so far Google knows about zero common occurrences of Mustache.jl and Mux.jl except for very general lists of frameworks - serving static files: possible to do in pure Julia, of course, but it's another several hours to implement Wouldn't that be kind of trivial, since Julia already acts as a web server, to load files from disk and forward? - sessions: middleware is exactly for this kind of things, but again, it's better to have it out of the box, than write everything yourself I thought I had seen something related to sessions, but may misremember, maybe it was to save REPL sessions.. or related to web/IJulia (only things I find now..). - authentication and security: how to set up HTTPS? how to restrict access to certain pages? mbedTLS.jl? A good replacement for GnuTLS.jl that seem on the way out. - stability: I've just knew that Morsel.jl is now deprecated, if I had applications using it, I would need to migrate them now, and I'm really not sure Mux.jl won't be deprecated during next year too I didn't check if this works the same, or just similarly? Or even not that..? Morsel was Sinatra-like and that seems to be a hot thing and often the way to go. Python and others reimplemented Ruby's Sinatra micro framework with Flask. I'm not sure Julia and Flask would make sense as by being "micro", this style of framework is already "complete" in Julia?] This means that if you want to provide users with a nice interface to your Julia application, you should either spend a couple of days adding missing stuff (and probably not in way suitable for other users) or just use, say, Python and do the job in a couple of hours. -- Palli. P.S.: I replied by e-mail.. And got "Delivery Status Notification (Failure)". "You might have spelled or formatted the group name incorrectly." Does this work for others from Thunderbird? E-mail works for GitHub, that is nice.. and I do not know about Discourse, the proposed new forum software.. Andrei Zh 9/24/15 I find this statement highly surprising.. wander if you meant to reverse this.. My quant friend who had worked for years in Python had trouble parallelizing Python code (may be resolved now..). I'm not familiar with R, but Python has the GIL and associated problems. I also thought Erlang was best-in-class (for concurrent, not "parallel"). Oops, you are right, it's exactly the opposite. Consider it a result of quick answer between 2 meetings. Julia's capabilities are much better for concurrent and parallel programming than these in Python or R. For those in the Java/Scala world, I'm less sure about reusing that, I know you can with JavaCall.jl, but understand there are bugs and limitations to it. Unfortunately, that's true. @aviks has made a great work connecting Julia to JVMs via Java Native Interface, but as far as I can see, JNI is shitty by itself, and it's very hard to to make integration between Julia and Java really stable. "HttpServer.jl, has low latency 0.5 ms and high throughput (latency on the same order of Python's Flask and Scala's Spray mature frameworks, and throughput also comparable[82])." That's funny, because specified reference leads to an issue on performance that was opened by me and incorrectly interpreted by the author of Wikipedia page. At that test Julia outperformed Flask, but was still 3-6 times slower than Spray. Seems like I need to repeat my tests to get latest results. I didn't check if this works the same, or just similarly? They have totally different APIs and approaches in general. The main point is that if you want to keep it working with updated version of Julia and libraries, you have to adapt web application too. And it's really not funny to come back to a code written half a year ago just because libraries it used are now deprecated. Wouldn't that be kind of trivial, since Julia already acts as a web server, to load files from disk and forward? To tell the truth, I have no idea. I know that for each static file (e.g. image or CSS) there's a specific MIME type, specific headers, etc. There is probably some caching policy and specific way to write bytes to a socket. But I don't know for sure and just want to provide web interface to my code, doing it as simple as possible. (please, don't consider it as a criticism of Julia web stack - I'm really thankful to all the people involved and pretty impressed by the work they've done - but rather as a TODO list for future development). Páll Haraldsson 9/25/15 On Thursday, September 24, 2015 at 11:07:56 PM UTC, Andrei Zh wrote: I find this statement highly surprising.. wander if you meant to reverse this.. My quant friend who had worked for years in Python had trouble parallelizing Python code (may be resolved now..). I'm not familiar with R, but Python has the GIL and associated problems. I also thought Erlang was best-in-class (for concurrent, not "parallel"). Oops, you are right, it's exactly the opposite. Consider it a result of quick answer between 2 meetings. Julia's capabilities are much better for concurrent and parallel programming than these in Python or R. But "probably comparable with Erlang" was not an error, taken conservatively, you are probably talking about what pure benchmark numbers of concurrency (only) might indicate. I believe Erlang has more (than speed) currently not in Julia, while not better for non-concurrency related. For those in the Java/Scala world, I'm less sure about reusing that, I know you can with JavaCall.jl, but understand there are bugs and limitations to it. Unfortunately, that's true. @aviks has made a great work connecting Julia to JVMs via Java Native Interface, but as far as I can see, JNI is shitty by itself, and it's very hard to to make integration between Julia and Java really stable. Even if this would work perfectly, I'm not sure (most) Julia (or Java?) users would want to mix (I guess ok for truly good/critical libraries..) at least for GUI/enterprise applications.. At least for me, it seems involving a JVM or any VM, is outdated.. Yes, Julia has LLVM, but it's not the same. "HttpServer.jl, has low latency 0.5 ms and high throughput (latency on the same order of Python's Flask and Scala's Spray mature frameworks, and throughput also comparable[82])." That's funny, because specified reference leads to an issue on performance that was opened by me and incorrectly interpreted by the author of Wikipedia page. At that test Julia outperformed Flask, but was still 3-6 times slower than Spray. Good throughput, but very high latency https://github.com/JuliaWeb/HttpServer.jl/issues/48 "At the same time, with even very naive test in Python (using requests library) I was able to get 1.3ms, and with Julia (Requests.jl) it was even lower - about 0.5ms per request." Then the issue was closed. It seems Julia can have low latency (are Flask and Splay [numbers, for it and throughput] considered best-in-class, at least for those language?). Splay has "< 1 ms" so unclear how much better, it may be. It seems you do not dispute the low latency claim of Julia [on the WP page] (any more)? Do you only disagree, then only, with the "high throughput" of Julia/[HttpServer.jl] (and would only change high->good)? [With 1000 threads] Julia beats Flask, "comparable", while a third of Splay. Maybe within an order of magnitude is not comparable.. Not sure about the 6 times slower number. I haven't looked to closely at Requests.jl, so I do not know the differences between it and HttpServer.jl are other libraries. It really only matters that one library gets good throughput and then you can use that one? Seems like I need to repeat my tests to get latest results. It would be interesting to know if the numbers have improved.. I didn't check if this works the same, or just similarly? They have totally different APIs and approaches in general. The main point is that if you want to keep it working with updated version of Julia and libraries, you have to adapt web application too. And it's really not funny to come back to a code written half a year ago just because libraries it used are now deprecated. Yes, see your point.. I guess the stuff still works it it ever did.. I see for plots that there is a generic package that is supposed to hide the differences of six (by now) different plotting packages.. You say the API is totally different, I wander how difficult or stupid it is to support old APIs here in some way.. Jameson 9/25/15 Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. From browsing issues, it looks like the HttpServer.jl performance issue you referenced below should be now fixed by https://github.com/JuliaWeb/HttpServer.jl/pull/59. - show quoted text - Andrei Zh 9/25/15 Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned.. From browsing issues, it looks like the HttpServer.jl performance issue you referenced below should be now fixed by https://github.com/JuliaWeb/HttpServer.jl/pull/59. Yes, this issue has been fixed. It seems Julia can have low latency Yes, on my later tests latency was pretty good (are Flask and Splay [numbers, for it and throughput] considered best-in-class, at least for those language?) Spray is pretty good, Flask is here just for comparison. I don't think Flask has been developed with performance in mind. Do you only disagree, then only, with the "high throughput" As you can see from results in #40, HttpServer.jl already has pretty good performance, just I believe Julia has no blockers not to get even more (comparable to best-in-class). I wander how difficult or stupid it is to support old APIs here in some way.. In most cases it's unreasonable to keep outdated API for many reasons. But "probably comparable with Erlang" was not an error, taken conservatively, you are probably talking about what pure benchmark numbers of concurrency (only) might indicate. I believe Erlang has more (than speed) currently not in Julia, while not better for non-concurrency related. When processing large number of small requests (e.g. text messages in social network or requests to API), blocking network operations become a bottleneck. E.g. if you have blocking "get()" operation, you either need to waste processing time until request is complete or switch processor context. Both of these are quite expensive and set a limit to a maximum throughput. Erlang has actor-based concurrency, which means that you can switch between tasks without switching processor context and get high utilization of CPU. Scala's Spray is built on Akka, which is an implementation of the same actor model, and gets very good performance. Julia has similar concurrency model using Tasks, and that's why I'm so optimistic regarding future of the web frameworks in Julia. Even if this would work perfectly, I'm not sure (most) Julia (or Java?) users would want to mix (I guess ok for truly good/critical libraries..) at least for GUI/enterprise applications.. Consider, for example, Apache Spark and the whole Hadoop infrastructure. There's nothing even close to it in non-JVM world, and that's why other languages (including Python and R) build their interfaces to JVM. - show quoted text -