Avoid method overwriting in `julia1-compat` branch #57

kimikage · 2024-05-07T03:07:05Z

StyledStrings v1.0 currently does method overwriting.

StyledStrings.jl/src/strings/overrides.jl

Lines 1 to 15 in db9f9b5

    
           ## From basic.jl 
        
           function (*)(s1::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...) 
        
               if _isannotated(s1) || any(_isannotated, ss) 
        
                   annotatedstring(s1, ss...) 
        
               else 
        
                   string(s1, ss...) 
        
               end 
        
           end 
        
           # From io.jl 
        
           join(iterator) = _join_preserve_annotations(iterator) 
        
           join(iterator, delim) = _join_preserve_annotations(iterator, delim) 
        
           join(iterator, delim, last) = _join_preserve_annotations(iterator, delim, last)

At least for *, we should be able to avoid overwriting by defining the following (or something like):

function (*)(s1::Union{AnnotatedChar, AnnotatedString}, s2::Union{AbstractChar, AbstractString})
    annotatedstring(s1, s2)
end

function (*)(s1::Union{AbstractChar, AbstractString}, s2::Union{AnnotatedChar, AnnotatedString})
    annotatedstring(s1, s2)
end

function (*)(s1::Union{AnnotatedChar, AnnotatedString}, s2::Union{AnnotatedChar, AnnotatedString})
    annotatedstring(s1, s2)
end

function (*)(s1::Union{AbstractChar, AbstractString}, s2::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...)
    if _isannotated(s1) || _isannotated(s2) || any(_isannotated, ss)
        annotatedstring(s1, s2, ss...)
    else
        string(s1, s2, ss...)
    end
end

Edit:
An alternative is to require that the first 2 or 3 arguments should contain Union{AnnotatedChar, AnnotatedString}.

The text was updated successfully, but these errors were encountered:

tecosaur · 2024-05-07T03:10:24Z

Hmmm, that's an interesting suggestion. At a glance, that looks viable, but I don't think we can do something similar with join unfortunately.

kimikage · 2024-05-07T03:19:51Z

I think limited support would be more beneficial (i.e., less troublesome) than method overwriting.

tecosaur · 2024-05-07T03:22:30Z

Maybe, the goal I went into with this though was to make it behave as close to the 1.11 stdlib as possible, and it's extremely close. As far as I'm aware, the problem with method overwriting here is he latency impact, is there anything beyond that?

kimikage · 2024-05-07T03:35:49Z

Whether or not method overwriting is performed is a greater difference than missing annotations in some cases.
The downstream package can avoid the latter if necessary, but not control the overwriting in StyledStrings.

tecosaur · 2024-05-07T03:51:02Z

Right, but what's the negative impact of the method overwriting? AFAIU it's just the latency.

kimikage · 2024-05-07T05:06:25Z

This is a tautology, but it is a problem because method overwriting is a problem that is worthy of a warning.

Older julia in particular are somewhat fragile with precompilation.
I think other core developers are more knowledgeable about the specific issues. So it might be good to ask the opinions of the experts on Slack, Zulip, or Discourse.

Also, I think we need to clarify the relationship between this package and Compat.jl.

kimikage · 2024-05-07T05:09:37Z

In the first place, the join is buggy.
JuliaLang/julia#51914

any(_isannotated, args) # `any(_isannotated, iterator)` ?

julia> join(("a", styled"{bold:b}"), ", ")|>typeof
String

julia> join(["a", styled"{bold:b}"], ", ")|>typeof
Base.AnnotatedString{String}

julia> versioninfo()
Julia Version 1.12.0-DEV.469
Commit 0d1d4ba068 (2024-05-06 21:04 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-17.0.6 (ORCJIT, tigerlake)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

tecosaur · 2024-05-07T05:09:48Z

Right, but the overwriting doesn't happen during precompilation. Chatting about method overwriting on Triage with Jeff, he said method overwriting is a fine thing to do.

Also, I think we need to clarify the relationship between this package and Compat.jl.

There is no relation, I can't say I see the need?

tecosaur · 2024-05-07T05:12:03Z

In the first place, the join is buggy.

Looks like that's worthy of an issue in JuliaLang/julia, either the join check will need to be tweaked or _isannotated needs another method.

kimikage · 2024-05-07T05:27:20Z

My point here (not in JuliaLang/julia) is that even with limited support for join, no major problems have arisen so far.

The following is just a PoC, but it works minimally.

join(iterator, delim::Union{AbstractString, AbstractChar}) = _join_preserve_annotations(iterator, delim)
join(iterator, delim::Union{AbstractString, AbstractChar}, last) = _join_preserve_annotations(iterator, delim, last)

Edit:
Or more conservatively:

join(iterator, delim::Union{AnnotatedChar, AnnotatedString}) = _join_preserve_annotations(iterator, delim)
join(iterator, delim::Union{AnnotatedChar, AnnotatedString}, last) = _join_preserve_annotations(iterator, delim, last)

kimikage · 2024-05-07T05:36:47Z

Also, I think we need to clarify the relationship between this package and Compat.jl.

There is no relation, I can't say I see the need?

The AnnotatedString is practically for StyledStrings, but is not the property of StyledStrings.
What is being done here is, in principle, piracy.
I do not believe that the piracy itself is the problem. However, I do think some statement should be issued about not doing it on Compat.jl.

tecosaur · 2024-05-07T05:37:30Z

The AnnotatedString is practically for StyledStrings, but is not the property of StyledStrings.

Ah, right I follow now.

tecosaur · 2024-05-07T05:42:04Z

any(_isannotated, args) # `any(_isannotated, iterator)` ?

Not quite, we do want any(_isannotated, args), but it might be worth adding || any(_isannotated, iterator).

kimikage · 2024-05-07T07:39:15Z

xref: https://discourse.julialang.org/t/method-overwriting-in-styledstrings-jl-v1-0-julia1-compat/113935
(for general discussions)

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, _isannotated can erroneously return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete, this check remains able to be done at compile-time. - Should the iterator be stateful and non-concrete, the check can consume some amount of the iterator before join is called. If it weren't for the edge-case of stateful iterators, I'd say this would be a fairly clean/straightforward fix, but since they exist I'm currently tempted to think the best compromise is just to avoid checking any Iterators.Stateful iterators. If somebody can work out a nicer compromise, that would be grand. Reported-by: kimikage <[email protected]>

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, _isannotated can erroneously return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. - Should the iterator be stateful and non-concrete, the check can consume some amount of the iterator before join is called. If it weren't for the edge-case of stateful iterators, I'd say this would be a fairly clean/straightforward fix, but since they exist I'm currently tempted to think the best compromise is just to avoid checking any Iterators.Stateful iterators. If somebody can work out a nicer compromise, that would be grand. Reported-by: kimikage <[email protected]>

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, _isannotated will return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. - Should the iterator be stateful and non-concrete, the check can consume some amount of the iterator before join is called. With this change, join always preserves annotations. The compromise made is that iterators with non-concrete eltypes can result in join inferring a union return type (i.e. type instability with non-concrete iterators), but that doesn't seem too bad to me. Reported-by: kimikage <[email protected]>

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, _isannotated will return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. With this change, join always preserves annotations. The compromise made is that iterators with non-concrete eltypes can result in join inferring a union return type (i.e. type instability with non-concrete iterators), but that doesn't seem too bad to me. Reported-by: kimikage <[email protected]>

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, `_isannotated` will return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we can reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. With this change, join always preserves annotations. The compromise made is that iterators with non-concrete eltypes can result in join inferring a union return type (i.e. type instability with non-concrete iterators), but that doesn't seem too bad to me (I don't see how we could be completely type stable without concrete types here).

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, `_isannotated` will return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we can reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. With this change, join always preserves annotations. The compromise made is that iterators with non-concrete eltypes can result in join inferring a union return type (i.e. type instability with non-concrete iterators), but that doesn't seem too bad to me (I don't see how we could be completely type stable without concrete types here). (cherry picked from commit 462d7f4)

KristofferC · 2024-08-06T09:42:54Z

FWIW, here is some snoop compile output:

julia> using SnoopCompileCore

julia> invalidations = @snoop_invalidations using StyledStrings

julia> using SnoopCompile

julia> trees = invalidation_trees(invalidations)
...
 deleting join(iterator) @ Base strings/io.jl:356 invalidated:
   backedges: 1: superseding join(iterator) @ Base strings/io.jl:356 with MethodInstance for join(::Vector{Char}) (1 children)
             ...
              8: superseding join(iterator) @ Base strings/io.jl:356 with MethodInstance for join(::Tuple{String}) (596 children)
   mt_disable: MethodInstance for join(::Vector{String})
               +5 more

 deleting join(iterator, delim) @ Base strings/io.jl:357 invalidated:
   backedges:  1: superseding join(iterator, delim) @ Base strings/io.jl:357 with MethodInstance for join(::Vector{Symbol}, ::Char) (1 children)
              ...
              11: superseding join(iterator, delim) @ Base strings/io.jl:357 with MethodInstance for join(::Base.Generator{Base.Iterators.Filter{Base.BinaryPlatforms.var"#14#16", Dict{String, String}}, Base.BinaryPlatforms.var"#13#15"}, ::String) (359 children)
              12: superseding join(iterator, delim) @ Base strings/io.jl:357 with MethodInstance for join(::Vector{SubString{String}}, ::String) (620 children)
              13: superseding join(iterator, delim) @ Base strings/io.jl:357 with MethodInstance for join(::Vector{Any}, ::String) (4123 children)
   mt_disable: MethodInstance for join(::Tuple{Symbol}, ::String)

   inserting *(s1::Union{AbstractChar, AbstractString}, s2::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...) @ StyledStrings.AnnotatedStrings ~/.julia/packages/StyledStrings/fsZJu/src/strings/basic.jl:21 invalidated:
   mt_backedges: 1: signature Tuple{typeof(*), Union{Nothing, String}, String} triggered MethodInstance for Pkg.Operations.var"#download_source#44"(::Bool, ::typeof(Pkg.Operations.download_source), ::Pkg.Types.Context) (0 children)
            ....
              13: superseding *(s1::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...) @ Base strings/basic.jl:260 with MethodInstance for *(::String, ::String, ::String) (114 children)
              14: superseding *(s1::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...) @ Base strings/basic.jl:260 with MethodInstance for *(::String, ::String, ::String, ::String, ::String, ::String, ::String) (1160 children)
              15: superseding *(s1::Union{AbstractChar, AbstractString}, ss::Union{AbstractChar, AbstractString}...) @ Base strings/basic.jl:260 with MethodInstance for *(::String, ::String) (6902 children)

As raised in <JuliaLang/StyledStrings.jl#57 (comment)>, when the eltype of an iterator is non-concrete, `_isannotated` will return false. To properly check such cases, we need to see if any of the elements of the iterator are annotated. This is a bit of an interesting case, since: - Most of the time it shouldn't be hit, we can reasonably expect most iterables to infer as producing concrete types - The eltype of the iterator is (generally) known at compile-time, and so in any case other than the ambiguous non-concrete one, this check remains able to be done at compile-time. With this change, join always preserves annotations. The compromise made is that iterators with non-concrete eltypes can result in join inferring a union return type (i.e. type instability with non-concrete iterators), but that doesn't seem too bad to me (I don't see how we could be completely type stable without concrete types here).

tecosaur · 2024-12-18T10:02:57Z

I think this can be closed since these methods were ultimately removed from the compat version.

kimikage mentioned this issue May 27, 2024

Type piracy in package #61

Open

tecosaur mentioned this issue Jun 24, 2024

Fix annotated join with non-concrete eltype iters JuliaLang/julia#54919

Merged

tecosaur mentioned this issue Jul 2, 2024

REPL becomes irresponsive for some seconds if StyledStrings is loaded after OhMyREPL KristofferC/OhMyREPL.jl#355

Closed

MilesCranmer mentioned this issue Oct 30, 2024

Different promotion behavior on 1.10 vs 1.11 #102

Open

tecosaur closed this as completed Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid method overwriting in `julia1-compat` branch #57

Avoid method overwriting in `julia1-compat` branch #57

kimikage commented May 7, 2024 •

edited

Loading

tecosaur commented May 7, 2024

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 •

edited

Loading

tecosaur commented May 7, 2024

kimikage commented May 7, 2024

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 •

edited

Loading

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 •

edited

Loading

KristofferC commented Aug 6, 2024

tecosaur commented Dec 18, 2024

Avoid method overwriting in julia1-compat branch #57

Avoid method overwriting in julia1-compat branch #57

Comments

kimikage commented May 7, 2024 • edited Loading

tecosaur commented May 7, 2024

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 • edited Loading

tecosaur commented May 7, 2024

kimikage commented May 7, 2024

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 • edited Loading

kimikage commented May 7, 2024

tecosaur commented May 7, 2024

tecosaur commented May 7, 2024

kimikage commented May 7, 2024 • edited Loading

KristofferC commented Aug 6, 2024

tecosaur commented Dec 18, 2024

Avoid method overwriting in `julia1-compat` branch #57

Avoid method overwriting in `julia1-compat` branch #57

kimikage commented May 7, 2024 •

edited

Loading

kimikage commented May 7, 2024 •

edited

Loading

kimikage commented May 7, 2024 •

edited

Loading

kimikage commented May 7, 2024 •

edited

Loading