Non-deterministic errors in CI - All keys must be unique

[kescobo]

I'm wrapping a NamedDimsArray in Microbiome.jl, and I'm getting some test errors that I can't begin to figure out how to address in BiobakeryUtils.jl, which has Microbiome as a dependency. The errors seem to come from NamedDims, throwing

 Error During Test at /home/runner/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:10
  Got exception outside of a @test
  All keys must be unique
# ...

but it's not consistently on any platform or julia version, and I've run the test suite half a dozen times locally and can't reproduce. I'm totally stumped.

As an example, on this commit, I have

julia version	platform	success
1.6	mac	yes
1.6	ubuntu	no
nightly	mac	yes
nightly	ubuntu	no

for the "push" action, but everything passes on "pull request".

The next commit (only changing docs, no code), I get the opposite

julia version	platform	success
1.6	mac	no
1.6	ubuntu	yes
nightly	mac	no
nightly	ubuntu	yes

for "push", and again everything passing on "pull request".

Then the next commit (again, no code changes)

julia version	platform	success	action
1.6	ubuntu	no	push
nightly	ubuntu	yes	push
1.6	ubuntu	yes	PR
nightly	ubuntu	no	PR

[oxinabox]

Can you post full stack trace?
We should probably enhance that error message to tell you at least the first few duplicate keys.
Which should enhance the understand-ability of the error.
Right now I can't workout where we even though that error.

(Right now the behavior of duplicate keys is mostly undefined)

[mcabbott]

From one of the logs here: https://github.com/BioJulia/BiobakeryUtils.jl/runs/3798723354 it looks like the error is thrown by AxisIndices.jl, although what it's trying to do (and why this isn't deterministic) I have no idea:

161
Metaphlan: Error During Test at /Users/runner/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:10
162
  Got exception outside of a @test
163
  All keys must be unique
164
  Stacktrace:
165
    [1] error(s::String)
166
      @ Base ./error.jl:33
167
    [2] check_unique_keys
168
      @ ~/.julia/packages/AxisIndices/Eb16j/src/errors.jl:16 [inlined]
169
    [3] _compose_axis
170
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:88 [inlined]
171
    [4] compose_axis
172
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:61 [inlined]
173
    [5] compose_axes
174
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:25 [inlined]
175
    [6] compose_axes
176
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:17 [inlined]
177
    [7] AxisArray
178
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:255 [inlined]
179
    [8] NamedDimsArray
180
      @ ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:168 [inlined]
181
    [9] NamedDimsArray
182
      @ ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:176 [inlined]
183
   [10] #NamedAxisArray#134
184
      @ ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:195 [inlined]
185
   [11] CommunityProfile(tab::SparseMatrixCSC{Float64, Int64}, features::Vector{Taxon}, samples::Vector{MicrobiomeSample})
186
      @ Microbiome ~/.julia/packages/Microbiome/M1Xkf/src/profiles.jl:65
187
   [12] metaphlan_profile(path::String, level::Symbol; sample::String)
188
      @ BiobakeryUtils ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:108
189
   [13] metaphlan_profile
190
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:103 [inlined]
191
   [14] metaphlan_profiles(paths::Vector{String}, level::Symbol)
192
      @ BiobakeryUtils ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:255
193
   [15] metaphlan_profiles(paths::Vector{String})
194
      @ BiobakeryUtils ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:253
195
   [16] macro expansion
196
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:32 [inlined]
197
   [17] macro expansion
198
      @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
199
   [18] top-level scope
200
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:11
201
   [19] include(fname::String)
202
      @ Base.MainInclude ./client.jl:444
203
   [20] top-level scope
204
      @ none:6
205
   [21] eval
206
      @ ./boot.jl:360 [inlined]
207
   [22] exec_options(opts::Base.JLOptions)
208
      @ Base ./client.jl:261
209
   [23] _start()
210
      @ Base ./client.jl:485
      ```

[kescobo]

Yeah, thanks @mcabbott, I was thinking the CI log would be more interpretable since it has colors. This is the log I was looking at, which seems to have the NamedDims stuff towards the bottom (oh, they're the same, I wasn't looking at the paths), if I should close and open this in AxisIndices, let me know

Metaphlan: Error During Test at /home/runner/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:10
  Got exception outside of a @test
  All keys must be unique
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:33
    [2] check_unique_keys
      @ ~/.julia/packages/AxisIndices/Eb16j/src/errors.jl:16 [inlined]
    [3] _compose_axis
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:88 [inlined]
    [4] compose_axis
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:61 [inlined]
    [5] compose_axes
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:25 [inlined]
    [6] compose_axes
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:17 [inlined]
    [7] AxisArray
      @ ~/.julia/packages/AxisIndices/Eb16j/src/axis_array.jl:255 [inlined]
    [8] NamedDimsArray
      @ ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:168 [inlined]
    [9] NamedDimsArray
      @ ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:176 [inlined]
   [10] (NamedDims.NamedDimsArray{L, T, N, AxisIndices.AxisArray{T, N, P, AI}} where {L, T, N, P, AI})(x::SparseMatrixCSC{Float64, Int64}; kwargs::Base.Pairs{Symbol, Vector, Tuple{Symbol, Symbol}, NamedTuple{(:features, :samples), Tuple{Vector{Taxon}, Vector{MicrobiomeSample}}}})
      @ AxisIndices ~/.julia/packages/AxisIndices/Eb16j/src/named.jl:195
   [11] CommunityProfile(tab::SparseMatrixCSC{Float64, Int64}, features::Vector{Taxon}, samples::Vector{MicrobiomeSample})
      @ Microbiome ~/.julia/packages/Microbiome/M1Xkf/src/profiles.jl:65
   [12] metaphlan_profile(path::String, level::Symbol; sample::String)
      @ BiobakeryUtils ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:108
   [13] metaphlan_profile (repeats 2 times)
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/src/metaphlan.jl:103 [inlined]
   [14] macro expansion
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:11 [inlined]
   [15] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [16] top-level scope
      @ ~/work/BiobakeryUtils.jl/BiobakeryUtils.jl/test/runtests.jl:11
   [17] include(fname::String)
      @ Base.MainInclude ./client.jl:451
   [18] top-level scope
      @ none:6
   [19] eval
      @ ./boot.jl:368 [inlined]
   [20] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:268
   [21] _start()