Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stateful Iterator omitting last element when used in list comprehension, crashes on 1-element long vectors. #35530

Closed
racinmat opened this issue Apr 20, 2020 · 6 comments · Fixed by #41919

Comments

@racinmat
Copy link

Strage behaviour of Stateful iterator:
Although

julia> for a in Iterators.Stateful([1,2,3])
             print(a)
         end
123

works as it should,
With list comprehension it fails to iterate over last element

julia> [a for a in Iterators.Stateful([1,2,3])]
2-element Array{Int64,1}:
 1
 2
julia> [a for a in Iterators.Stateful([1])]
ERROR: BoundsError: attempt to access 0-element Array{Int64,1} at index [1]

tried on Julia 1.3.1, Julia 1.4.0 and Julia 1.4.1.
More details in https://discourse.julialang.org/t/strange-behavior-with-list-comprehensions-using-iterators-stateful/29715

@StefanKarpinski
Copy link
Member

cc @Keno since IIRC, you implemented the Stateful iterator

@martinholters
Copy link
Member

That looks like a bug in collect for stateful HasLength iterators. It retrieves the first element (to determine its type) by calling iterate once and then invokes _array_for which in the HasLength case calls length to determine the required array size. But the length has at this point decreased by 1 due to the first call to iterate, resulting in an array which is too short by one. I guess this should be restructured to calling length before calling iterate and passing that length on.

@KristofferC
Copy link
Member

@martinholters
Copy link
Member

Ah yes, should have read that first...

@drdozer
Copy link

drdozer commented Aug 18, 2021

I think I just hit this:

begin
	items = ["pigs", "dogs", "cats"]
	project(x) = begin println("Projecting: ", x, "!"); join(["P_", x]) end
	projections = Iterators.Stateful(project(x) for x in items)
	println("Testing isempty")
	isempty(projections)
	println("Testing collect")
	collected = collect(projections)
	println("Done")
	
	(items, collected)
end

This produces

"pigs"
"dogs"
"cats",
"P_pigs"
"P_dogs

@drdozer
Copy link

drdozer commented Aug 18, 2021

I have "fixed" it by adding this:

function Base.collect(si::Iterators.Stateful)
	items = Vector()
 	it = iterate(si)
	while it !== nothing
		i, state = it
		push!(items, i)
		it = iterate(si)
	end
	items
end

It makes my eyes bleed a bit to have to include this in my script though. Although it feels like a plaster over a compound fracture, perhaps it is one approach to fixing this in core?

vtjnash pushed a commit that referenced this issue Aug 24, 2021
Previously this code would drop 1 from the length of some generators.

Fixes #35530
KristofferC pushed a commit that referenced this issue Aug 25, 2021
Previously this code would drop 1 from the length of some generators.

Fixes #35530

(cherry picked from commit 8364a4c)
KristofferC pushed a commit that referenced this issue Aug 25, 2021
Previously this code would drop 1 from the length of some generators.

Fixes #35530

(cherry picked from commit 8364a4c)
KristofferC pushed a commit that referenced this issue Aug 31, 2021
Previously this code would drop 1 from the length of some generators.

Fixes #35530

(cherry picked from commit 8364a4c)
KristofferC pushed a commit that referenced this issue Sep 3, 2021
Previously this code would drop 1 from the length of some generators.

Fixes #35530

(cherry picked from commit 8364a4c)
LilithHafner pushed a commit to LilithHafner/julia that referenced this issue Feb 22, 2022
Previously this code would drop 1 from the length of some generators.

Fixes JuliaLang#35530
LilithHafner pushed a commit to LilithHafner/julia that referenced this issue Mar 8, 2022
Previously this code would drop 1 from the length of some generators.

Fixes JuliaLang#35530
staticfloat pushed a commit that referenced this issue Dec 23, 2022
Previously this code would drop 1 from the length of some generators.

Fixes #35530

(cherry picked from commit 8364a4c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants