[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT #606

kivanio · 2022-01-27T18:33:13Z

For long time I have a code like this:

STATES = I18n.t('states').with_indifferent_access.freeze

The yaml file is in UTF-8 and it has accents in some words:

pt-BR:
  states:
    Acre: AC
    Amapá: AP
    Ceará: CE
    Piauí: PI
    Paraná: PR

with new release 1.9.0
It starts failing in our CI in a lot of places:

ActionView::Template::Error incompatible character encodings: UTF-8 and ASCII-8BIT
Failure/Error: = f.select(:state, City::STATES,

ActionView::Template::Error:
  incompatible character encodings: UTF-8 and ASCII-8BIT
./app/views/customers/_form.html.slim:86:in `block in

Downgrade to 1.8.11 make everything works again.

With 1.9.0 use rails console it loads like:

rails c
Running via Spring preloader in process 4964
Loading development environment (Rails 6.1.4.4)
[1] pry(main)> I18n.t('states').with_indifferent_access.freeze
=> {"Acre"=>"AC",
 "Alagoas"=>"AL",
 "Amazonas"=>"AM",
 "Amap\xC3\xA1"=>"AP",
 "Bahia"=>"BA",
 "Cear\xC3\xA1"=>"CE",
 "Distrito Federal"=>"DF",
 "Esp\xC3\xADrito Santo"=>"ES",
 "Goi\xC3\xA1s"=>"GO",
 "Maranh\xC3\xA3o"=>"MA",
 "Minas Gerais"=>"MG",
 "Mato Grosso do Sul"=>"MS",
 "Mato Grosso"=>"MT",
 "Par\xC3\xA1"=>"PA",
 "Para\xC3\xADba"=>"PB",
 "Pernambuco"=>"PE",
 "Piau\xC3\xAD"=>"PI",
 "Paran\xC3\xA1"=>"PR",
 "Rio de Janeiro"=>"RJ",
 "Rio Grande do Norte"=>"RN",
 "Rond\xC3\xB4nia"=>"RO",
 "Roraima"=>"RR",
 "Rio Grande do Sul"=>"RS",
 "Santa Catarina"=>"SC",
 "Sergipe"=>"SE",
 "S\xC3\xA3o Paulo"=>"SP",
 "Tocantins"=>"TO"}

With 1.8.11 use rails console it loads like:

rails c
Running via Spring preloader in process 3411
Loading development environment (Rails 6.1.4.4)
[1] pry(main)> I18n.t('states').with_indifferent_access.freeze
=> {"Acre"=>"AC",
 "Alagoas"=>"AL",
 "Amazonas"=>"AM",
 "Amap\xC3\xA1"=>"AP",
 "Bahia"=>"BA",
 "Cear\xC3\xA1"=>"CE",
 "Distrito Federal"=>"DF",
 "Esp\xC3\xADrito Santo"=>"ES",
 "Goi\xC3\xA1s"=>"GO",
 "Maranh\xC3\xA3o"=>"MA",
 "Minas Gerais"=>"MG",
 "Mato Grosso do Sul"=>"MS",
 "Mato Grosso"=>"MT",
 "Par\xC3\xA1"=>"PA",
 "Para\xC3\xADba"=>"PB",
 "Pernambuco"=>"PE",
 "Piau\xC3\xAD"=>"PI",
 "Paran\xC3\xA1"=>"PR",
 "Rio de Janeiro"=>"RJ",
 "Rio Grande do Norte"=>"RN",
 "Rond\xC3\xB4nia"=>"RO",
 "Roraima"=>"RR",
 "Rio Grande do Sul"=>"RS",
 "Santa Catarina"=>"SC",
 "Sergipe"=>"SE",
 "S\xC3\xA3o Paulo"=>"SP",
 "Tocantins"=>"TO"}

The output seems exactly the same.
Is that a bug in new version?

Probably something between new version and load in rails.
This is just a sample
I have others files with accents and all of them are raising same exception.

Versions of i18n, rails, and anything else you think is necessary

ruby: 3.0.3
i18n: 1.9.0
rails: 6.1.4.3
rspec-rails: 5.1.0
rspec: 3.10.0

and # frozen_string_literal: true in ruby files.

The text was updated successfully, but these errors were encountered:

radar · 2022-01-27T22:18:57Z

Hello, thank you for the detailed reproduction steps.

I was unable to reproduce this issue using:

Ruby 3.0.0
i18n 1.9.1
Rails 6.1.4.3

(The RSpec versions are highly likely to be irrelevant to this issue)

I will try with Ruby 3.0.3 now.

radar · 2022-01-27T22:32:16Z

I am unable to reproduce this issue with 3.0.3. Could you please put a Rails app that does reproduce this issue on GitHub so that I can clone it down and investigate?

joergschiller · 2022-01-27T22:56:45Z

We have a similar issue. I've created this repo to demonstrate the difference.

Using i18n 1.8.11 the keys are symbolized with UTF-8 encoding. With 1.9.1 it's ASCII.

gem 'i18n', "< 1.9.0"
I18n.t("foobar").keys => [:ö]
I18n.t("foobar").keys.first.encoding => #<Encoding:UTF-8>

I18n.t("foobar").values.first.encoding => #<Encoding:UTF-8>

gem 'i18n', "> 1.9.0"
I18n.t("foobar").keys => [:"\xC3\xB6"]
I18n.t("foobar").keys.first.encoding => #<Encoding:ASCII-8BIT>

I18n.t("foobar").values.first.encoding => #<Encoding:UTF-8>

https://github.com/joergschiller/i18n_issue_606/blob/293513b938310239e06bf4062be975e0ec772fc0/Gemfile#L6-L12

(With Ruby 3.0.3)

kivanio · 2022-01-28T01:40:47Z

Thank you @joergschiller
I am not alone 🙏🏻
I was going to make a repo and you saved me.
I was believing that It was something desired in new version but now I think we have a bug.

radar · 2022-01-28T03:29:31Z

Thank you @joergschiller. I can reproduce this issue now with your repository. I'll find the commit that broke it.

radar · 2022-01-28T03:32:25Z

Commit that breaks this behaviour is 0fda789, as discovered through a git bisect:

 ~/code/gems/i18n   v1.9.0~7^2 (bisect)  bad
0fda789ea745cd462658a8948ee085201aba5c6f is the first bad commit
commit 0fda789ea745cd462658a8948ee085201aba5c6f
Author: Paarth Madan <[email protected]>
Date:   Wed Nov 3 12:33:12 2021 -0400

    Symbolize and freeze keys when loading from YAML

 lib/i18n/backend/base.rb    | 2 +-
 test/backend/simple_test.rb | 7 ++++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

radar · 2022-01-28T03:32:53Z

@paarthmadan Do you have any time today to investigate this one?

radar · 2022-01-28T04:06:58Z

Looked into where the unsafe_load_file method was coming from bootsnap. cc @casperisfine.

/Users/ryan.bigg/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/gems/bootsnap-1.10.2/lib/bootsnap/compile_cache/yaml.rb:203:in `unsafe_load_file': wrong number of arguments (given 0, expected 1+) (ArgumentError)

If I remove bootsnap from this test application, the issue goes away:

irb(main):001:0> I18n.t(:foobar).keys[0]
=> :ö

@casperisfine: Would you like me to create an issue on the Bootsnap repo page for this one?

casperisfine · 2022-01-28T07:47:51Z

👀

casperisfine · 2022-01-28T07:56:44Z

Ah damn it, I know what the problem is. It's because Bootsnap uses msgpack to accelarate YAML parsing, and MessagePack use an API that doesn't preserve symbols encoding properly. See an issue I opened a while ago msgpack/msgpack-ruby#211

Let me go over my old research see how we could sidestep this in bootsnap. I'll update here ASAP.

casperisfine · 2022-01-28T09:15:10Z

Ok, so the bug is actually in msgpack, I opened a PR here: msgpack/msgpack-ruby#246

You can apply the patch with:

gem 'msgpack', github: 'Shopify/msgpack-ruby', branch: 'symbolize-keys-fix-encoding'

>> I18n.t(:foobar)
=> {:ö=>"ü"}

Alternatively, if you'd rather not run a gem branch, you can disable Bootsnap YAML caching. Sorry for the bug :/

radar · 2022-01-28T22:40:57Z

Thank you very much for the deep investigation here :)

radar · 2022-02-03T23:17:11Z

@casperisfine Is there something we could do to get that msgpack/msgpack-ruby PR merged + released? Is there a bribe of cookies that needs to be made here?

casperisfine · 2022-02-04T07:31:38Z

Not that I know of. He did acknowledge seeing some of my other PRs, he's probably busy.

I don't like to nag maintainers, but I see the same bug was reported again yesterday, so I'll ping him on that on PR just this once.

casperisfine · 2022-02-04T09:10:36Z

He said early next week hopefully.

casperisfine · 2022-02-04T09:34:27Z

If you are interested, I could add some feature checking in I18n, so sidestep the optimization when it's bogus.

radar · 2022-02-04T23:21:51Z

@casperisfine That could be a good workaround in the meantime, I think. Could you please find out what that would take??

casperisfine · 2022-02-08T22:30:31Z

Uh, I just come back to this issue, and I was certain I already answered :/

So first the fix was merged upstream today, but not sure when there will be a release.

For the workaround, we could simply test wether the bug is present with something like:

√: ~

and then from ruby:

if YAML.unsafe_load_file("test.yml", symbolize_names: true).keys.first.encoding == Encoding::UTF_8
  # it works we can use the optimization
  ...

…the bug is present

radar · 2022-02-14T07:42:57Z

Reviewing this again, I think we'll just wait for a new msgpack release to happen, and then advise people who encounter this issue to upgrade to that new version.

I'll be leaving this issue open until that new version is out.

casperisfine · 2022-02-15T13:19:20Z

msgpack 1.4.5 was released a few hours ago and should solve this issue: https://rubygems.org/gems/msgpack/versions/1.4.5

kivanio · 2022-02-15T14:21:56Z

Thank you all for the hard work! 👏🏻

radar added the no-repro label Jan 27, 2022

This was referenced Jan 28, 2022

Add specs for symbols encoding preservation msgpack/msgpack-ruby#211

Closed

Fix symbolize_keys: true to properly create UTF-8 symbols msgpack/msgpack-ruby#246

Merged

vinistock mentioned this issue Jan 31, 2022

Bump rubocop-shopify from 2.1.0 to 2.4.0 Shopify/tapioca#776

Merged

laurafeier mentioned this issue Feb 3, 2022

[BUG] I18n.transliterate broken #614

Closed

radar removed the no-repro label Feb 3, 2022

radar added this to the 1.10 milestone Feb 8, 2022

radar mentioned this issue Feb 8, 2022

LazyLoadable Backend #612

Merged

casperisfine pushed a commit to Shopify/i18n that referenced this issue Feb 8, 2022

Feature test for issue ruby-i18n#606 and disable the optimization if …

ab0b6d9

…the bug is present

casperisfine mentioned this issue Feb 8, 2022

Feature test for issue #606 and disable the optimization if the bug is present #618

Closed

lorenzk mentioned this issue Feb 14, 2022

[BUG] Custom transliterations aren't take into account (ASCII-8BIT encoded keys?) #623

Closed

Laykou mentioned this issue Feb 15, 2022

Assume symbols are either ASCII or UTF-8 msgpack/msgpack-ruby#248

Merged

ahorek mentioned this issue Feb 6, 2023

inherit encoding for interned symbols jruby/jruby#7618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT #606

[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT #606

kivanio commented Jan 27, 2022

radar commented Jan 27, 2022

radar commented Jan 27, 2022

joergschiller commented Jan 27, 2022 •

edited

Loading

kivanio commented Jan 28, 2022

radar commented Jan 28, 2022

radar commented Jan 28, 2022 •

edited

Loading

radar commented Jan 28, 2022

radar commented Jan 28, 2022

casperisfine commented Jan 28, 2022

casperisfine commented Jan 28, 2022

casperisfine commented Jan 28, 2022

radar commented Jan 28, 2022

radar commented Feb 3, 2022

casperisfine commented Feb 4, 2022

casperisfine commented Feb 4, 2022

casperisfine commented Feb 4, 2022

radar commented Feb 4, 2022

casperisfine commented Feb 8, 2022

radar commented Feb 14, 2022

casperisfine commented Feb 15, 2022

kivanio commented Feb 15, 2022

[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT #606

[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT #606

Comments

kivanio commented Jan 27, 2022

Versions of i18n, rails, and anything else you think is necessary

radar commented Jan 27, 2022

radar commented Jan 27, 2022

joergschiller commented Jan 27, 2022 • edited Loading

kivanio commented Jan 28, 2022

radar commented Jan 28, 2022

radar commented Jan 28, 2022 • edited Loading

radar commented Jan 28, 2022

radar commented Jan 28, 2022

casperisfine commented Jan 28, 2022

casperisfine commented Jan 28, 2022

casperisfine commented Jan 28, 2022

radar commented Jan 28, 2022

radar commented Feb 3, 2022

casperisfine commented Feb 4, 2022

casperisfine commented Feb 4, 2022

casperisfine commented Feb 4, 2022

radar commented Feb 4, 2022

casperisfine commented Feb 8, 2022

radar commented Feb 14, 2022

casperisfine commented Feb 15, 2022

kivanio commented Feb 15, 2022

joergschiller commented Jan 27, 2022 •

edited

Loading

radar commented Jan 28, 2022 •

edited

Loading