ice_datagen: Data files bigger than expected?

[naricc]

So, I am still experimenting with using icu4x in .Net, as mentioned in this previous post: #2810.

Right now, I have a rust library where I have put in a stubs for all the APIs we need, and a very tiny amount of actual code. I was trying to see what size of datafile I would get with by analyzing the binary with icu_datagen (I followed directions here: https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/data_management.md). The actual library itself is here: https://github.com/naricc/runtime/tree/naricc/icu4x/src/mono/mono/icu-bridge.

I used this command after building it:

icu4x-datagen --overwrite --cldr-tag latest --icuexport-tag latest --out my-data-blob2 --format blob --keys-for-bin ./target/release/libicu_bridge.a -l fr_CA

fr_CA is just an arbitrary locale I picked out to experiment with. The 658K; isn't that kind of big considering the library is mostly empty? All I have in there so far is a call to Locale::try_from_bytes and TimeZoneFormatter::try_new_unstable. Am I missing something here?

[sffc]

Are you sure that your .a file is being dead-code-eliminated? This seems like the most likely issue. You should point --keys-for-bin at a file that has been DCE'd, not at the library, which hasn't been DCE'd yet.

When you run icu4x-datagen, what keys does the output say it is exporting?

Locale::try_from_bytes has no data dependency, and TimeZoneFormatter::try_new_unstable has only time_zone/formats@1, which is 122 B in fr; you can get postcard size estimates from here: https://github.com/unicode-org/icu4x/blob/main/provider/testdata/data/postcard/fingerprints.csv

[naricc]

@sffc I definately did not make a specific effort to do dead code elimination. Is DCE elimination only possible on a complete executable? Or is there some way to do it conservatively on a library (like, eliminating parts that can never be used by any executable)?

Because for the scenario I have in mind, doing it on a complete executable is going to be quite hard. Basically, I am using this library as a replacement icu library for the Mono runtime (runs .net code). Mono itself is often just a library that is DL opened on some platforms. When a user of mono finally writes some C# code and builds it, we do get a binary executable out at the end, which has the whole runtime and everything linked (possibly dynamically, depending on the platform). I can probably do an experiment where I run something on a final executable for some sample program manually, but its a lot of plumbing to automate. Would it even be possible to do DCE elimination on that thing, which is mostly written in C? How would I go about doing that?

Here is the list of keys it ended up using, I can see that there are many unneeded ones:

INFO  [icu_datagen] Writing key: props/Hyphen@1
INFO  [icu_datagen] Writing key: props/NChar@1
INFO  [icu_datagen] Writing key: props/graph@1
INFO  [icu_datagen] Writing key: plurals/cardinal@1
INFO  [icu_datagen] Writing key: collator/prim@1
INFO  [icu_datagen] Writing key: props/Emoji@1
INFO  [icu_datagen] Writing key: props/Bidi_M@1
INFO  [icu_datagen] Writing key: locid_transform/likelysubtags@1
INFO  [icu_datagen] Writing key: time_zone/generic_short@1
INFO  [icu_datagen] Writing key: props/Gr_Base@1
INFO  [icu_datagen] Writing key: list/and@1
INFO  [icu_datagen] Writing key: locid_transform/aliases@1
INFO  [icu_datagen] Writing key: normalizer/nfkd@1
INFO  [icu_datagen] Writing key: props/blank@1
INFO  [icu_datagen] Writing key: datetime/ethiopic/datelengths@1
INFO  [icu_datagen] Writing key: props/IDC@1
INFO  [icu_datagen] Writing key: normalizer/decomp@1
INFO  [icu_datagen] Writing key: props/Cased@1
INFO  [icu_datagen] Writing key: datetime/buddhist/datelengths@1
INFO  [icu_datagen] Writing key: datetime/gregory/datelengths@1
INFO  [icu_datagen] Writing key: props/CI@1
INFO  [icu_datagen] Writing key: props/GCB@1
INFO  [icu_datagen] Writing key: props/alnum@1
INFO  [icu_datagen] Writing key: props/AHex@1
INFO  [icu_datagen] Writing key: props/bc@1
INFO  [icu_datagen] Writing key: props/lb@1
INFO  [icu_datagen] Writing key: plurals/ordinal@1
INFO  [icu_datagen] Writing key: props/Ideo@1
INFO  [icu_datagen] Writing key: props/Sensitive@1
INFO  [icu_datagen] Writing key: props/IDSB@1
INFO  [icu_datagen] Writing key: props/IDS@1
INFO  [icu_datagen] Writing key: props/CWCF@1
INFO  [icu_datagen] Writing key: props/Alpha@1
INFO  [icu_datagen] Writing key: list/or@1
INFO  [icu_datagen] Writing key: props/IDST@1
INFO  [icu_datagen] Writing key: datetime/indian/datelengths@1
INFO  [icu_datagen] Writing key: normalizer/nfkdex@1
INFO  [icu_datagen] Writing key: props/CWCM@1
INFO  [icu_datagen] Writing key: props/CWL@1
INFO  [icu_datagen] Writing key: props/CWT@1
INFO  [icu_datagen] Writing key: props/Pat_Syn@1
INFO  [icu_datagen] Writing key: props/Pat_WS@1
INFO  [icu_datagen] Writing key: props/PCM@1
INFO  [icu_datagen] Writing key: props/print@1
INFO  [icu_datagen] Writing key: props/CWU@1
INFO  [icu_datagen] Writing key: props/CWKCF@1
INFO  [icu_datagen] Writing key: datetime/coptic/datelengths@1
INFO  [icu_datagen] Writing key: props/QMark@1
INFO  [icu_datagen] Writing key: collator/data@1
INFO  [icu_datagen] Writing key: props/Radical@1
INFO  [icu_datagen] Writing key: datetime/buddhist/datesymbols@1
INFO  [icu_datagen] Writing key: datetime/gregory/datesymbols@1
INFO  [icu_datagen] Writing key: props/RI@1
INFO  [icu_datagen] Writing key: props/sc@1
INFO  [icu_datagen] Writing key: props/LOE@1
INFO  [icu_datagen] Writing key: props/Lower@1
INFO  [icu_datagen] Writing key: datetime/ethiopic/datesymbols@1
INFO  [icu_datagen] Writing key: time_zone/metazone_period@1
INFO  [icu_datagen] Writing key: props/Math@1
INFO  [icu_datagen] Writing key: time_zone/generic_long@1
INFO  [icu_datagen] Writing key: props/Bidi_C@1
INFO  [icu_datagen] Writing key: props/Gr_Ext@1
INFO  [icu_datagen] Writing key: fallback/supplement/co@1
INFO  [icu_datagen] Writing key: collator/jamo@1
INFO  [icu_datagen] Writing key: datetime/indian/datesymbols@1
INFO  [icu_datagen] Writing key: collator/meta@1
INFO  [icu_datagen] Writing key: datetime/coptic/datesymbols@1
INFO  [icu_datagen] Writing key: time_zone/exemplar_cities@1
INFO  [icu_datagen] Writing key: props/Hex@1
INFO  [icu_datagen] Writing key: props/ccc@1
INFO  [icu_datagen] Writing key: props/Gr_Link@1
INFO  [icu_datagen] Writing key: normalizer/nfd@1
INFO  [icu_datagen] Writing key: datetime/japanese/datelengths@1
INFO  [icu_datagen] Writing key: collator/reord@1
INFO  [icu_datagen] Writing key: fallback/likelysubtags@1
INFO  [icu_datagen] Writing key: fallback/parents@1
INFO  [icu_datagen] Writing key: collator/dia@1
INFO  [icu_datagen] Writing key: props/Dash@1
INFO  [icu_datagen] Writing key: decimal/symbols@1
INFO  [icu_datagen] Writing key: props/ExtPict@1
INFO  [icu_datagen] Writing key: normalizer/comp@1
INFO  [icu_datagen] Writing key: props/Ext@1
INFO  [icu_datagen] Writing key: props/Comp_Ex@1
INFO  [icu_datagen] Writing key: props/gc@1
INFO  [icu_datagen] Writing key: datetime/japanese/datesymbols@1
INFO  [icu_datagen] Writing key: props/nfcinert@1
INFO  [icu_datagen] Writing key: time_zone/specific_long@1
INFO  [icu_datagen] Writing key: normalizer/nfdex@1
INFO  [icu_datagen] Writing key: calendar/japanese@1
INFO  [icu_datagen] Writing key: datetime/japanext/datelengths@1
INFO  [icu_datagen] Writing key: props/nfkcinert@1
INFO  [icu_datagen] Writing key: time_zone/specific_short@1
INFO  [icu_datagen] Writing key: props/nfdinert@1
INFO  [icu_datagen] Writing key: calendar/japanext@1
INFO  [icu_datagen] Writing key: props/nfkdinert@1
INFO  [icu_datagen] Writing key: datetime/japanext/datesymbols@1
INFO  [icu_datagen] Writing key: props/Join_C@1
INFO  [icu_datagen] Writing key: props/SD@1
INFO  [icu_datagen] Writing key: props/Term@1
INFO  [icu_datagen] Writing key: props/scx@1
INFO  [icu_datagen] Writing key: props/SB@1
INFO  [icu_datagen] Writing key: datetime/timelengths@1
INFO  [icu_datagen] Writing key: props/segstart@1
INFO  [icu_datagen] Writing key: normalizer/uts46d@1
INFO  [icu_datagen] Writing key: props/STerm@1
INFO  [icu_datagen] Writing key: props/WB@1
INFO  [icu_datagen] Writing key: props/DI@1
INFO  [icu_datagen] Writing key: props/Dep@1
INFO  [icu_datagen] Writing key: props/Dia@1
INFO  [icu_datagen] Writing key: datetime/week_data@1
INFO  [icu_datagen] Writing key: props/WSpace@1
INFO  [icu_datagen] Writing key: props/EComp@1
INFO  [icu_datagen] Writing key: props/ea@1
INFO  [icu_datagen] Writing key: props/EBase@1
INFO  [icu_datagen] Writing key: props/VS@1
INFO  [icu_datagen] Writing key: time_zone/formats@1
INFO  [icu_datagen] Writing key: props/EMod@1
INFO  [icu_datagen] Writing key: props/EPres@1
INFO  [icu_datagen] Writing key: datetime/timesymbols@1
INFO  [icu_datagen] Writing key: list/unit@1
INFO  [icu_datagen] Writing key: props/Upper@1
INFO  [icu_datagen] Writing key: props/UIdeo@1
INFO  [icu_datagen] Writing key: props/XIDC@1
INFO  [icu_datagen] Writing key: props/xdigit@1
INFO  [icu_datagen] Writing key: props/XIDS@1

[naricc]

2. As a more permanent solution, figure out how to customize the link command generating your .a file to only include a subset of symbols. I did something similar in the tinywasm example.

So, I think I see from the example how to do control what symbols are exported. But is there some other step involved to feed that information to a DCE pass? Or is that done automatically after linking?

[naricc]

So I've using these linker args in my .cargo/config.toml to try and strip dead stuff:

[build]
rustflags = ["-C", "link-args=-Wl,-exported_symbols_list /Users/naricc/workspace/runtime/runtime-icu4x/src/mono/mono/icu-bridge/exported_symbols_list", "-C", "link-args=-Wl,-dead_strip", "-C", "link-args=-Wl,-dead_strip_dylibs" ]

But I am still getting the same size data file from the datagen command I used, and it looks like it is exporting the same keys?

Is there something else I need to be doing to strip out dead code?

I don't think I can used path 1. easily, since a major reason I want to use this is that I can't easily figure out what keys are needed.

[naricc]

It looks like maybe it is not limiting itself to the export only the symbols I specified, as I see a lot of junk when I do nm -gU ./target/release/libicu_bridge.a. Some how it is not propagating the linker arguments correctly I guess? Although I do not know why.

[sffc]

Yeah, the way rustc invokes ld is a bit weird. In my tinywasm example, I had to inject a little Python script in-between rustc and ld, so that I can directly overwrite the arguments that are being passed around.

I already filed an upstream issue to fix this: rust-lang/rust#104130

[naricc]

After much fiddling, so I eventually got it to work using RUSTFLAGS="-C link-arg=-Wl,-exported_symbols_list -C link-arg=${BASEDIR}/exported_symbols.list" (use link-arg instead of link-args) and also by making crate-type = ["cdylib"] . Maybe with the right flags it would be possible to make it work with a static library as well, but its not essential to my secenario.