Move DArrays from Base  10333 Closed ViralBShah opened this Issue on Feb 25, 2015 · 26 comments Projects None yet Labels parallel Milestone No milestone Assignees No one assigned 10 participants  ViralBShah  jakebolewski  andreasnoack  amitmurthy  tknopp  mwil  jiahao  timholy  tkelman  kmsquire Notifications    ViralBShah The Julia Language member ViralBShah commented on Feb 25, 2015  We need a ton of functionality to be developed on DArrays, and perhaps even experimentation with layouts and such. I feel it is best to move the DArray functionality out of Base into the JuliaParallel organization  https:  github.com JuliaParallel  and develop it there.  jakebolewski The Julia Language member jakebolewski commented on Feb 25, 2015  :+1:, I can do this.  ViralBShah The Julia Language member ViralBShah commented on Feb 25, 2015  Cc:  andreasnoack  amitmurthy  alanedelman  ViralBShah ViralBShah added the parallel label on Feb 25, 2015  andreasnoack The Julia Language member andreasnoack commented on Feb 25, 2015  I'm fine with this.  jakebolewski The Julia Language member jakebolewski commented on Feb 25, 2015  What about SharedArrays?  amitmurthy The Julia Language member amitmurthy commented on Feb 25, 2015  I am fine too.  I think SharedArrays should continue to be in Base, at least till we get threading support in.  Only concern - do we have any idea how many real users of DArrays's are there? I know 0.4 is all about breaking changes, but we should generally try and minimize stuff like name changes, package changes, etc in the user community.  ViralBShah The Julia Language member ViralBShah commented on Feb 25, 2015  I think SharedArrays should stay. Even after threading, we will want them, but perhaps then they can be in a package.  There are probably a few users of DArrays, but they are likely to be cutting edge, since the capabilities do not compose very well. The current APIs could be a lot better too.  ViralBShah The Julia Language member ViralBShah commented on Feb 25, 2015   jakebolewski is fast:  https:  github.com JuliaParallel DistributedArrays-jl  jakebolewski jakebolewski referenced this issue on Feb 26, 2015 Merged Remove `DArray`'s from Base  10334  tknopp tknopp commented on Feb 26, 2015  great! It seems  5155 is slowly happening...  jakebolewski jakebolewski closed this in  10334 on Feb 27, 2015  mwil mwil commented on Mar 2, 2015  I was using DArrays for a few computations ... How do you actually use the packages in the new location? DistributedArrays is unknown and Pkg.add  JuliaParallel  , Pkg.add  DistributedArrays   or similar is not working.  jiahao The Julia Language member jiahao commented on Mar 2, 2015   mwil if you want to stay on 0.4, you can try Pkg.clone  https:  github.com JuliaParallel DistributedArrays-jl.git  . But most users should be using 0.3 instead.  timholy The Julia Language member timholy commented on Mar 2, 2015  Why hasn't the package been registered?  tkelman The Julia Language member tkelman commented on Mar 2, 2015  Can we please not leave users out in the cold, and actually do this correctly, making sure things are usable  registered, tested, working on both 0.3 and 0.4  before we take them out of base? This won't be the last time we do this.  jiahao The Julia Language member jiahao commented on Mar 2, 2015   tkelman Nobody knows how to migrate anything out of Base correctly. I doubt very much that we can maintain backwards compatibility. Nobody figured out how to do it for Base.Graphics, and then there was the whole StatsBase débâcle before that.  tkelman The Julia Language member tkelman commented on Mar 2, 2015  Whatever happened with StatsBase was well over a year ago, I don't know what went wrong there. Now we're stumbling over wrong ways to do this by taking things out very hastily before replacements are working correctly, let's not keep repeating this pattern.  Are there any specific challenges to making DistributedArrays-jl work well on both 0.3 and 0.4?  Some good first steps would be broadly announcing on the mailing lists once there's consensus that some functionality should be migrated to a package, getting current users of that functionality to migrate to the package version before merging the PR that deletes it from base, and getting things to a working state with at least the obvious kinks worked out first.  jiahao The Julia Language member jiahao commented on Mar 2, 2015  I agree that removing a module from Base in a dev version at the very least merits a mention on julia-dev. However, we need the flexibility of being able to break code in order to get things done. In this case the breakage is necessary because:      once a type foo is exported from a module and that module is loaded, you are stuck with having that type in the Main namespace and no other module can re-export foo. So you can't have two versions of the same module coexist without breakage, and you can't change the definition of foo without breaking backwards compatibility.      once a function bar is exported from a module A and another module B exports the same foo, any conflicting methods will get overwritten in B. But that means that you cannot change anything in B whilst guaranteeing the same behavior as A by re-exporting bar. So it would be impossible to make breaking changes in B, thus defeating the entire purpose of this use case.  The only solution is to rename everything, but then you have to deal with the pain of synchronizing the name change globally, which was the case in the Stats -  StatsBase rename.  kmsquire The Julia Language member kmsquire commented on Mar 2, 2015  Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...  Are there problems with this?  tkelman The Julia Language member tkelman commented on Mar 2, 2015      at the very least merits a mention on julia-dev  Probably julia-users at this point. It's been six months since 0.3.0, suggesting people stay away from 0.4 isn't going to be a viable response for much longer.      However, we need the flexibility of being able to break code in order to get things done.  When there's a clear immediate upside and good reason for the breakage, of course it needs to happen occasionally. Here code was just deleted, the upside is in modularity and development constraints of being inside vs outside of base but there's no urgency on those points. It's entirely possible to recognize those particular upsides before even removing any code or causing any breakage.      The only solution is to rename everything, but then you have to deal with the pain of synchronizing the name change globally, which was the case in the Stats -  StatsBase rename.  A migration is being forced anyway. Getting various other packages to use the new Graphics version was done quickly enough, but in retrospect it should not have used the same name as the base module.      Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...  Sounds worth trying. Especially if it can be done using only 0.3 features, without needing one of  6195  6884 or  7449.  jiahao The Julia Language member jiahao commented on Mar 2, 2015      Here code was just deleted, the upside is in modularity and development constraints of being inside vs outside of base but there's no urgency on those points.  Unlike the Base.Graphics removal, at this point we do have a critical mass of applications waiting to be written  like parallel linear algebra  and the labor to do it  those cc'ed in the OP  and lots of potentially breaking changes  like user-customizable transports  to try out.      Getting various other packages to use the new Graphics version was done quickly enough, but in retrospect it should not have used the same name as the base module.  The problems with Graphics-jl are not so simple that they can be solved by renaming the module. Please read what I wrote above about importing conflicting definitions more carefully. The current situation is metastable and will collapse the moment a breaking change in Graphics-jl is made since all the newly transitioned code implicitly requires backward compatibility.      Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...  I don't see how Compat-jl will solve the problem of breaking backward compatibility. At best it can only proffer the kind of feature freeze which can also be expressed by putting julia 0.4-dev+1234 in the REQUIRE file of the newly externalized package. Furthermore, it appears that Compat macros do not compose with macros or other applications requiring compile-time code generation.  I couldn't figure out how to use Compat in an expression like BinDeps. install  :lib    :symbol . BinDeps. install  compat Dict :lib    :symbol  does not work, but I haven't looked more closely at how to fix this.   tkelman The Julia Language member tkelman commented on Mar 2, 2015      at this point we do have a critical mass of applications waiting to be written  like parallel linear algebra  and the labor to do it  those cc'ed in the OP   I find this hard to take seriously when the package is still not registered or passing its tests. better now, but this work really should've been done before removing code from base.      Please read what I wrote above about importing conflicting definitions more carefully. The current situation is metastable and will collapse the moment a breaking change in Graphics-jl is made since all the newly transitioned code implicitly requires backward compatibility.  Short-term solution, why not leave things unexported from the under-development replacement versions until they've been tested and evaluated as working? Should be easy enough to test things fully-qualified at first, or only import them into local modules to avoid the conflicting imports into the main namespace. Clearly we need a better long-term solution for this problem, and to be more careful about touching the global namespace.  Should we consider un-exporting some lesser-used modules within base to lessen this transition pain?      I couldn't figure out how to use Compat in an expression like BinDeps. install  :lib    :symbol   It sounds from JuliaIO HDFfive-jl 216 like  compat BinDeps. install Dict :lib    :symbol  might actually work for that one, but it does need more testing.  jiahao The Julia Language member jiahao commented on Mar 2, 2015      Short-term solution, why not leave things unexported from the under-development replacement versions until they've been tested and evaluated as working? Should be easy enough to test things fully-qualified at first, or only import them into local modules to avoid the conflicting imports into the main namespace.  I think that's the only viable solution right now - keep previously exported identifiers untouched and develop new replacements in a different namespace or with different identifiers.  tknopp tknopp commented on Mar 2, 2015  I am with  tkelman that it would be good to first create and register the package before removing it. But other than that I cannot see whats the issue. Master is not !  for users and I cannot see the point of discussing these things on the julia-users mailing list. A half year since the last release is not an argument, we still should not encourage using 0.4dev for users. This leads to the same situation as in 0.3 where we had a rolling release in the end.  Personally I do not fully understand the point of the Compat-jl package since IMHO packages should not developed against master. Better make more regular Julia releases instead if the features are really that important.  tkelman The Julia Language member tkelman commented on Mar 2, 2015      Better make more regular Julia releases instead if the features are really that important.  This is the point. Features are important, staged functions and performance improvements and other changes are resulting in a group of users  and that big gray area between users and contributors  who do work against master. That's inherently risky of them, but letting them know things will be moving ahead of time is the best way to get their help.  More regular major releases is a good goal, but that's going to need feature definition, stabilization, and wide testing for the RC stage. But anyway we're way off topic.  tknopp tknopp commented on Mar 2, 2015  I think it needs destabilizing on master to get features in. And justifications on julia mailing lists while the primary development platform is github seems redundant.  Don't think that this is off-topic at all since you were not happy how this got merged.  ViralBShah The Julia Language member ViralBShah commented on Mar 5, 2015  I believe the DistributedArrays-jl package has to still be added to METADATA. Any reason not to do it yet?  jakebolewski The Julia Language member jakebolewski commented on Mar 5, 2015  The package is registered and tagged now that all the functionality is working again.  timholy The Julia Language member timholy commented on Mar 5, 2015  :+1:  jiahao jiahao referenced this issue on Mar 5, 2015 Open Shrinking Base and Introducing a Standard Library  5155