Dispatch method selection (RangeIndex... more specific than Integer?)

[timholy]

Not sure if this is a bug or a design issue, but I would argue that in method dispatch insufficient priority is given to the # of arguments. I found the following somewhat surprising:

function f(i::Integer)
    println("f1")
end

function f(i::Union(Int,Integer))
    println("f1union")
end

function f(i1::Integer,i2::Integer)
    println("f2")
end

function f(ind::RangeIndex...)
    println("fn")
end

julia> f(3)
fn

julia> f(uint(3))
f1union

julia> f(convert(Integer,3))
fn

julia> function f(i::Int)
         println("f1Int")
       end

julia> f(3)
f1Int

So even the explicit f(i::Union(Int,Integer)) is not enough to "override" the generic f(ind::RangeIndex...) variant; it requires defining an Int-specific variant. The Int-specific variant can be defined before or after the others, it does not matter which.

[pao]

For reference, here's the function sort orders, most to least specific as reported by Julia. Note there are only three dispatches listed after defining the first four functions, even though the Union(Int, Integer) is accessible as shown in the transcript:

julia> f
Methods for generic function f
f(Union(Range1{Int64},Range{Int64},Int64)...,) at none:2
f(Integer,) at none:2
f(Integer,Integer) at none:2

And with the additional dispatch added in the transcript, still four, not five:

julia> f
Methods for generic function f
f(Int64,) at none:2
f(Union(Range1{Int64},Range{Int64},Int64)...,) at none:2
f(Integer,) at none:2
f(Integer,Integer) at none:2

[nolta]

Doesn't Union(Int,Integer) == Integer?

[pao]

Oh, it's overriding the first method. That's why it's accessible. I didn't realize none of the tests came up with the "f1" response. So it's just a question of why the type RangeIndex... is considered to be tighter than Integer. Added to the issue title.

[Keno]

RangeIndex seems to be Union(Range1{Int64},Range{Int64},Int64) and Int64 is more specific than Integer, not to mention that the dispatch on RangedIndex was defined after that on Integer.

[timholy]

Then Union(Int,Integer) should not resolve to Integer.

But I don't want to lose sight of the other aspect of this: if I provide a general method:

myfun(I::InputsType...)

but then also a specialization for 1 and 2 arguments,

myfun(I::InputsType)
myfun(I1::InputsType,I2::InputsType)

then to me it seems that the number of arguments should trump other considerations, as long as there is a sensible resolution for the one- and two-argument input.

[JeffBezanson]

Union(Int,Integer) is exactly the same type as Integer and cannot be otherwise. It means "Int, or any kind of integer".

RangeIndex is more specific because it contains only concrete types, and Integer is an abstract type.

convert(Integer,3) does nothing. You can't have a value whose concrete type is Integer.

[StefanKarpinski]

convert(Integer,3) does nothing. You can't have a value whose concrete type is Integer.

I've provided a number of conversions to abstract types. I don't see that there's a problem with that. To me, it means, "convert this value to some kind of Integer, but I don't care what kind".

[JeffBezanson]

I didn't say there's a problem with it, just that that particular use does
nothing since 3 is already an integer. This is unlike static type systems
where you might be able to ascribe the type Integer to it to make it
dispatch as Integer rather than Int.
On May 3, 2012 2:19 PM, "Stefan Karpinski" <
[email protected]>
wrote:

convert(Integer,3) does nothing. You can't have a value whose concrete
type is Integer.

I've provided a number of conversions to abstract types. I don't see that
there's a problem with that. To me, it means, "convert this value to some
kind of Integer, but I don't care what kind".

---

Reply to this email directly or view it on GitHub:
#795 (comment)

[timholy]

OK, so what to do about this? I should say this is not purely academic: it seems to be important for my ongoing work on issue 765, where there are specializations of ref and assign for up to 4 dimensions. Now that I'm writing optimized versions that operate on RangeIndexes, there are turning out to be all sorts of nasty surprises in terms of which variants of the functions get called.

Option #1: no change in "core julia". For the 2-input syntax I provide 4 versions:
f(Integer, Integer)
f(Int, Integer)
f(Integer, Int)
f(Int, Int)
in order to prevent the RangeIndex version from taking charge. For 3-inputs, it would be 8, and 4-inputs, 16. It's not as bad as it seems since I should be able to use for loops and @eval to generate the code.

Option #2: we redefine RangeIndex this way:
RangeIndex = Union(Integer, Range{Int}, Range1{Int})
I think this would solve the priority problem. Or I can define my own RangeIndexForRefAndAssignFunctions type.

Option #3: For a function which can take arbitrary # of inputs, in dispatch give higher priority to the # of arguments than to the most concrete type. Not sure whether this would have unintended side effects.

Option #4 (science fiction?): provide a mechanism for providing manual "hints" about the resolving dispatch ambiguity, that allow one to achieve #3 in particular circumstances without affecting policy more globally.

[timholy]

Out of curiosity, why can't Union(Int, Integer) be different from just Integer? I presume Union does not work by "promoting" to the common parent; you can have a Union(Int8, Float64) without needing to bring in all bitstypes. If Int works differently from Integer with regards to method dispatch, it seems inconsistent not to be able to specify that it's a legitimate specialization for a particular argument.

[StefanKarpinski]

Union(Int,Integer) just means the set of all objects x for which isa(x,Int) || isa(x,Integer) are true. Since this is logically equivalent to just isa(x,Integer) the union type is just Integer.

[StefanKarpinski]

Tim, I'm not sure which of your four options is best, but 2 seems like just a work-around (although perhaps a pragmatic one), while 1 is pretty unsatisfying; 4 doesn't feel very Julian to me, which leaves 3. There were reasons to set the priorities the way they are now, but I can't recall what they were. When you get into varargs cases and dipatch, things get pretty hard to reason about.

[timholy]

Thanks for the explanation, that makes sense.

I'm a neophyte when it comes to thinking about type specifications in programming languages. But I have to ask, does Union have a practical use aside from controlling dispatch? If not, shouldn't its behavior mimic the choices made by the dispatch mechanism?

If Union does need to work the way it's working, then is it worth considering having a DispatchUnion which does mimic dispatch's choices? I guess that's basically option 4 in disguise.

[JeffBezanson]

Union is actually originally intended for everything except dispatch, but it happens to work there too. The concept you want here is "Integer, but treat as Int in ambiguous cases", not Union.

I think the best option is one not on the list: only use Int, and not Integer for indexes.

Option 3 is reasonable but there have already been cases where the exact opposite priority behavior was wanted. In a case like Integer vs. Union(Range,Int) it's really quite subjective which is more specific. Both are slightly bigger than Int, in different ways.

[timholy]

Sounds good to me. Given how central arrays are to everything in Julia, we'll quickly learn whether anything breaks.