Errors when loading parallel packages on same machine #16788 Closed RossBoylan opened this Issue on Jun 6 · 25 comments Projects None yet Labels parallel Milestone No milestone Assignees No one assigned 5 participants @RossBoylan @tkelman @wildart @StefanKarpinski @vtjnash Notifications You’re not receiving notifications from this thread. @RossBoylan RossBoylan commented on Jun 6 See also #16778. That problem involves multiple machines; I get very similar errors using, e.g., -p 4 on a single machine. Originally I think the packages/cache were a bit out of date, and when I ran in parallel things went wrong and rebuilt everywhere. If they were all rebuilding in the same spot, as they seemed to, it's easy to imagine that would cause problems. I also had a single CPU copy of julia running at the same time (idle) throughout. I was running under ESS. After that I started up a single cpu version, did a Pkg.update(), and ran my code. Everything seemed OK. However, I then ran a parallel julia, and continued to get errors. The included code is all in a module and connect to a Postgresql DB to get some data. Here are excerpts from the sessions, first my initial run | | | | | | |/ _' | | | | |_| | | | (_| | | Version 0.4.6-pre+28 (2016-04-22 00:59 UTC) _/ |\__'_|_|_|\__'_| | Commit 022917e (44 days old release-0.4) |__/ | x86_64-linux-gnu WARNING: Terminal not fully functional julia> julia> nprocs() 5 julia> now() 2016-06-05T00:04:52 # Pacific time US julia> @everywhere include("trouble.jl") INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Debug.ji for module Debug. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/PDMats.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module PDMats uuid did not match cache file This is likely because module PDMats does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsFuns.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: Module ArrayViews uuid did not match cache file This is likely because module ArrayViews does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji WARNING: Module ArrayViews uuid did not match cache file This is likely because module ArrayViews does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ArrayViews.ji WARNING: Module ArrayViews uuid did not match cache file This is likely because module ArrayViews does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Distributions.ji for module Distributions. WARNING: Module ArrayViews uuid did not match cache file This is likely because module ArrayViews does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/StatsBase.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/PDMats.ji for module PDMats. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji WARNING: Module SortingAlgorithms uuid did not match cache file This is likely because module SortingAlgorithms does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji WARNING: Module SortingAlgorithms uuid did not match cache file This is likely because module SortingAlgorithms does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji WARNING: Module SortingAlgorithms uuid did not match cache file This is likely because module SortingAlgorithms does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji WARNING: Module SortingAlgorithms uuid did not match cache file This is likely because module SortingAlgorithms does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DataArrays.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/StatsFuns.ji for module StatsFuns. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/GZip.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Docile.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ArrayViews.ji for module ArrayViews. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/StatsBase.ji for module StatsBase. WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/JSON.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ImmutableArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Codecs.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Colors.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/ColorTypes.ji WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "autocor"; uses of it in module DataFrames must be qualified WARNING: both DataArrays and StatsBase export "inverse_rle"; uses of it in module DataFrames must be qualified WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Iterators.ji WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/Distributions.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Compose.ji WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DataFrames.ji for module DataFrames. WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji WARNING: Module FixedPointNumbers uuid did not match cache file This is likely because module FixedPointNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Cairo.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: both DataArrays and StatsBase export "rle"; uses of it in module DataFrames must be qualified WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Graphics.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DataArrays.ji for module DataArrays. WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji WARNING: Module Iterators uuid did not match cache file This is likely because module Iterators does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Showoff.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Hexagons.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Contour.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Loess.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji WARNING: Module WoodburyMatrices uuid did not match cache file This is likely because module WoodburyMatrices does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji WARNING: Module WoodburyMatrices uuid did not match cache file This is likely because module WoodburyMatrices does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji WARNING: Module WoodburyMatrices uuid did not match cache file This is likely because module WoodburyMatrices does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/KernelDensity.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Calculus.ji WARNING: Module Calculus uuid did not match cache file This is likely because module Calculus does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji WARNING: Module Calculus uuid did not match cache file This is likely because module Calculus does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji WARNING: Module Calculus uuid did not match cache file This is likely because module Calculus does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/DualNumbers.ji WARNING: Module DualNumbers uuid did not match cache file This is likely because module DualNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji WARNING: Module DualNumbers uuid did not match cache file This is likely because module DualNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji WARNING: Module DualNumbers uuid did not match cache file This is likely because module DualNumbers does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Optim.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji WARNING: Module Compat uuid did not match cache file This is likely because module Compat does not support precompilation but is imported by a module that does. WARNING: deserialization checks failed while attempting to load cache from /home/ross/.julia/lib/v0.4/Grid.ji INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/GZip.ji for module GZip. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/SortingAlgorithms.ji for module SortingAlgorithms. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Docile.ji for module Docile. WARNING: Module Docile uuid did not match cache file This is likely because module Docile does not support precompilation but is imported by a module that does. WARNING: Module Docile uuid did not match cache file This is likely because module Docile does not support precompilation but is imported by a module that does. WARNING: Module Docile uuid did not match cache file This is likely because module Docile does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: Module Docile uuid did not match cache file This is likely because module Docile does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/DataFrames.ji WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module DBI INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/JSON.ji for module JSON. WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ImmutableArrays.ji for module ImmutableArrays. WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Gadfly.ji for module Gadfly. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Codecs.ji for module Codecs. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/FixedPointNumbers.ji for module FixedPointNumbers. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Colors.ji for module Colors. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/ColorTypes.ji for module ColorTypes. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Iterators.ji for module Iterators. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Compose.ji for module Compose. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Cairo.ji for module Cairo. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Graphics.ji for module Graphics. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Showoff.ji for module Showoff. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Hexagons.ji for module Hexagons. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Contour.ji for module Contour. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/FixedSizeArrays.ji for module FixedSizeArrays. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Loess.ji for module Loess. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/WoodburyMatrices.ji for module WoodburyMatrices. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/KernelDensity.ji for module KernelDensity. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Calculus.ji for module Calculus. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/DualNumbers.ji for module DualNumbers. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Optim.ji for module Optim. INFO: Recompiling stale cache file /home/ross/.julia/lib/v0.4/Grid.ji for module Grid. WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 2 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: node state is inconsistent: node 3 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 4 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji WARNING: Module StatsFuns uuid did not match cache file This is likely because module StatsFuns does not support precompilation but is imported by a module that does. WARNING: node state is inconsistent: node 5 failed to load cache from /home/ross/.julia/lib/v0.4/Gadfly.ji ERROR: On worker 5: LoadError: LoadError: LoadError: LoadError: UndefVarError: CairoSurface not defined in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in require at ./loading.jl:259 in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in require at ./loading.jl:259 in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in eval at ./sysimg.jl:14 in anonymous at multi.jl:1394 in anonymous at multi.jl:923 in run_work_thunk at multi.jl:661 [inlined code] from multi.jl:923 in anonymous at task.jl:63 while loading /home/ross/.julia/v0.4/Compose/src/cairo_backends.jl, in expression starting on line 45 while loading /home/ross/.julia/v0.4/Compose/src/Compose.jl, in expression starting on line 17 while loading /home/ross/.julia/v0.4/Gadfly/src/Gadfly.jl, in expression starting on line 9 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 10 in remotecall_fetch at multi.jl:747 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 in sync_end at ./task.jl:413 in anonymous at multi.jl:1405 julia> include("/home/ross/PCORI/trouble.jl") WARNING: replacing module RB RB julia> @everywhere include("trouble.jl") WARNING: replacing module RB WARNING: replacing module RB WARNING: replacing module RB WARNING: replacing module RB WARNING: replacing module RB WARNING: could not import Gadfly.plot into RB And after the cleanup with one CPU: julia> nprocs() 4 julia> @everywhere include("trouble.jl") WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module DataFrames WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: replacing module DataFrames WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: replacing module DataFrames WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module Gadfly WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. WARNING: replacing module Gadfly WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. WARNING: replacing module Gadfly WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. ERROR: On worker 2: LoadError: MethodError: `prepare` has no method matching prepare(::PostgreSQL.PostgresDatabaseHandle, ::ASCIIString) Closest candidates are: prepare(::PostgreSQL.PostgresDatabaseHandle, ::AbstractString) prepare(::DBI.DatabaseHandle, ::AbstractString) in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in eval at ./sysimg.jl:14 in anonymous at multi.jl:1394 in anonymous at multi.jl:923 in run_work_thunk at multi.jl:661 [inlined code] from multi.jl:923 in anonymous at task.jl:63 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 15 in remotecall_fetch at multi.jl:747 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 in sync_end at ./task.jl:413 in anonymous at multi.jl:1405 @RossBoylan RossBoylan referenced this issue on Jun 6 Closed Errors when loading parallel packages with ssh tunnel #16778 @tkelman The Julia Language member tkelman commented on Jun 6 See also #13684. You should probably load everything and do recompilation in serial first, before trying to load that code on parallel workers. @RossBoylan RossBoylan commented on Jun 6 Thanks for the reference. Note that the second log in my original report was after I loaded and recompiled in serial; there were still problems. @vtjnash vtjnash closed this on Jun 12 @RossBoylan RossBoylan commented on Jun 13 • edited This seems to have been closed with the message The binary build on each client needs to be the same, not just the source checkout. But the bug title and the very first line of my report says that these errors happened while running on a single machine. So I think this bug should be re-opened. @tkelman The Julia Language member tkelman commented on Jun 14 I'd look into what's causing the module replacement and method overwrite warnings, resolve those and see if you still have an issue. @RossBoylan RossBoylan commented on Jun 15 How? The warnings only occur in parallel mode; when I ran on a single process immediately before there were no complaints (that is, once I did the package update). @tkelman tkelman added the needs-more-info label on Jun 15 @RossBoylan RossBoylan commented on Jun 16 If the needs-more-info tag is directed at me, I don't understand what more info I should provide. @tkelman The Julia Language member tkelman commented on Jun 16 Self-contained code that allows others to reproduce the problem. @RossBoylan RossBoylan commented on Jun 16 The errors appear to concern that state of my package cache; are you saying I should upload the cache? The sequence that led to these problems began running parallel jobs when the cache needed to be updated, and reproducing that would require an update to the external packages. I suppose with some fiddling I could create a local "upstream" package and tweak it. Apart from the question of origins, I'm currently in this weird state in which Pkg.update() runs fine as a single process, but loading in parallel fails. Ross … @wildart wildart commented on Jun 16 Pkg.update is not designed to run in parallel. Running it in parallel mode could result in unidentified state of the global package cache. @tkelman The Julia Language member tkelman commented on Jun 16 Sounds like you need to clear out your cache, have it rebuild in serial, and if the problems go away there's not really an actionable bug right now, other than perhaps the feature request of implementing a package-cache lock file. @RossBoylan RossBoylan commented on Jun 16 How do I clear the cache? I don't see a command in http://docs.julialang.org/en/release-0.4/manual/packages/. Do I just deleted the whole ~/.julia/v0.4, or maybe ~/julia/v0.4/.cache and the directory it points to? BTW, I did not explicitly run Pkg.update() in parallel, but, as the first transcript in my original report shows, some kind of update operation was triggered when I loaded my module in parallel. I'm guessing it was basically a Pkg.update(). @tkelman The Julia Language member tkelman commented on Jun 16 delete everything under ~/.julia/lib/v0.4 @RossBoylan RossBoylan commented on Jun 16 ~/.julia/lib/v0.4/.cache is a symlink to ~/.julia/.cache. Should I delete ~/.julia/.cache as well, or perhaps the material underneath it? I'd hate to delete the cache without actually deleting it:) @tkelman The Julia Language member tkelman commented on Jun 16 .cache is actually a git cache, not what we're referring to here, and you shouldn't need to touch it - we mean just the .ji package precompilation "cache," sorry for the unclear wording @RossBoylan RossBoylan commented on Jun 17 • edited Short version: clearing the cache didn't help. The following code is sufficient for me to get errors in parallel on one machine, even though I get no errors when not in parallel: module RB using Debug using Distributions using FastGaussQuadrature using DataFrames using DBI using PostgreSQL using ImmutableArrays using Gadfly import Gadfly.plot end Here are the errors: julia> nprocs() 4 !julia> @everywhere include("trouble2.jl") WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module DataFrames WARNING: replacing module DataFrames WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\ n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\ taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\ n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\ taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\ rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\ n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\ taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\ n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\ taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\ rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: replacing module DataFrames WARNING: Method definition vcat(Array{Void, 1}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677 overwritten i\ n module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:677. WARNING: Method definition dump(Base.IO, DataArrays.AbstractDataArray{T<:Any, 1}, Int64, Any) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractda\ taframe/abstractdataframe.jl:358 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:358. WARNING: Method definition describe(AbstractArray) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406 overwritte\ n in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:406. WARNING: Method definition describe(Any, AbstractArray{#T<:Number, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractda\ taframe.jl:408 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:408. WARNING: Method definition describe(Any, AbstractArray{#T<:Any, N<:Any}) in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataf\ rame.jl:425 overwritten in module DataFrames at /home/ross/.julia/v0.4/DataFrames/src/abstractdataframe/abstractdataframe.jl:425. WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module DBI WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module PostgreSQL WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module ImmutableArrays WARNING: replacing module Gadfly WARNING: replacing module Gadfly WARNING: replacing module Gadfly WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. WARNING: New definition chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57 is ambiguous with: chain(Gadfly.Data...) at /home/ross/.julia/v0.4/Gadfly/src/data.jl:57. To fix, define chain() before the new definition. History First I deleted everything under ~/.julia/v0.4/ and issued Pkg.add to get the packages back. I did Pkg.update() and load of my original trouble.jl in serial mode and everything was fine. Then I started a parallel version, did @everywhere include, and got errors. Then I realized I had deleted the wrong directory, and deleted everything under ~/.julia/lib/v0.4/. There were no julia's running at the time. I repeated the single user load (except using the stuff trouble2.jl shown above); no error messages. Then I did the parallel load reported above. @tkelman tkelman added parallel and removed needs-more-info labels on Jun 17 @tkelman The Julia Language member tkelman commented on Jun 17 Thanks. I only see warnings there? @RossBoylan RossBoylan commented on Jun 17 Correct. I noticed that and thought perhaps everything was OK. But when I load the real code, which gets some data from a database, I get julia> @everywhere include("trouble.jl") WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Debug WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module Distributions WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature WARNING: replacing module FastGaussQuadrature ERROR: On worker 3: LoadError: LoadError: LoadError: UndefVarError: Postgres not defined in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in require at ./loading.jl:259 in include_string at loading.jl:282 in include_from_node1 at ./loading.jl:323 in eval at ./sysimg.jl:14 in anonymous at multi.jl:1394 in anonymous at multi.jl:923 in run_work_thunk at multi.jl:661 [inlined code] from multi.jl:923 in anonymous at task.jl:63 while loading /home/ross/.julia/v0.4/PostgreSQL/src/dbi_impl.jl, in expression starting on line 3 while loading /home/ross/.julia/v0.4/PostgreSQL/src/PostgreSQL.jl, in expression starting on line 17 while loading /home/ross/PCORI/trouble.jl, in expression starting on line 8 in remotecall_fetch at multi.jl:747 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 ...and 1 other exceptions. in sync_end at ./task.jl:413 in anonymous at multi.jl:1405 The extra code is conn = connect(Postgres, "localhost", "pcori", "xxx", "pcori") stmt = prepare(conn, "SELECT isim::int, id::int, x::int, t::double precision, y::int, regular::int, censor::int, wait::double precision from obs2619 where isim=1") result = execute(stmt) # my previous attempt used fetchdf(stmt) obs = fetchdf(result) stmt = prepare(conn, "SELECT iter::int, negloglike::double precision, beta_0::double precision, beta_g::double precision, beta_t::double precision, beta_i::double pre\ cision, lnsigma_0::double precision, lnsigma_t::double precision, atanhrho::double precision, delta_0::double precision, delta_y ::double precision, lnnu::double preci\ sion from iter where iterset=2") result = execute(stmt) iter = fetchdf(result) finish(stmt) disconnect(conn) although I note the line number in the traceback refers to line 8, using PostgreSQL, which is in the test code I sent. @amitmurthy amitmurthy referenced this issue on Jun 18 Open `using` loads modules on workers but does not put exported bindings in Main #9245 @RossBoylan RossBoylan commented on Jun 18 The using directives and the calls such as connect(Postgres, "localhost", "pcori", "xxx", "pcori") are within the RB module, so I don't see how bindings in Main matter. But perhaps I misunderstand how using works. Do I need to qualify references to the module, PostgreSQL in the case of the last error above, explicitly? Use some other kind of import method? @RossBoylan RossBoylan commented on Jun 18 Regarding the warning messages displayed by my simple test code: does the fact that I don't see them in serial model mean they don't occur, or could this just be a difference in whether warnings are displayed in parallel vs serial mode? @RossBoylan RossBoylan commented on Jun 18 #9245 and related bugs suggested trying an additional include just in the main process. This eliminated all apparent problems: # trouble.jl is the header code given above plus additional substantive code julia> include("/home/ross/PCORI/trouble.jl") RB julia> @everywhere include("trouble.jl") WARNING: replacing module RB # and nothing else I'm delighted to be able to run parallel code. I remain baffled by the behavior. For example, why are the errors and warnings from the child processes affected by what I do in the main process? Outstanding items: protection against multiple updates to the same directory. This affects parallel runs on one machine (my case), multiple independent runs on one machine (me also when I had a couple of julia interactive shells open), and parallel runs on multiple machines sharing a common file system. Note that "updates" includes both explicit calls to Pkg.update() and the implicit updates that occur when reading in code like my sample code, i.e., include a file that has using directives. #9245 and related Other? It's not clear to me if the previous 2 items account for all the misbehavior I've seen. @wildart wildart commented on Jun 19 When you develop a concurrent application you should account for various race conditions in your code. It's a bit naive to expect single process application to work as multiprocess one without any modification. Even though the Julia language provide a friendly framework for multiprocessor computations that eliminates thread-level races, process-level data races could occur when data is modified and accessed simultaneously by different processes. Always check your code and packages so they are correctly work in multiprocessor environments. @RossBoylan RossBoylan commented on Jun 19 julia regenerates the cache somewhat behind the back of user code; it seems to me desirable that it do so in a way that is parallel-safe. @StefanKarpinski The Julia Language member StefanKarpinski commented on Jun 20 We probably should do some file locking here. The main issue is actually a cross-platform implementation and API for file locking: #7176. It still doesn't seem like we'll be able to rely on libuv for this, so someone would need to implement it in Julia. @wildart wildart commented on Jun 20 I think if we move from a in-file storage to a directory storage which would contain manifests of installed packages the parallel installation can be easily solved. @RossBoylan RossBoylan commented on Jun 20 Another scenario: multiple architectures with a shared home directory. Not my original problem, or even an anticipated scenario for me, but maybe something to consider.