Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Baked data for FFI examples #3630

Merged
merged 1 commit into from
Jul 5, 2023
Merged

Baked data for FFI examples #3630

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions .github/workflows/artifacts-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,7 @@ jobs:
- name: Init node package
run: |
cd ffi/diplomat/js/examples/node
make lib
make icu_capi.wasm
# Manually running datagen with the required keys
cargo run -p icu_datagen -- --key-file ../wasm-demo/required-keys.txt --locales full --format blob --out data.postcard
cd ../wasm-demo
make
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add cd ../wasm-demo after make?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That'll be it, thanks!

npm ci

- name: Run Webpack
Expand Down
1 change: 1 addition & 0 deletions components/locid_transform/src/provider/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ const _: () = {
icu_locid_transform_data::impl_locid_transform_likelysubtags_ext_v1!(Baked);
icu_locid_transform_data::impl_locid_transform_likelysubtags_l_v1!(Baked);
icu_locid_transform_data::impl_locid_transform_likelysubtags_sr_v1!(Baked);
#[cfg(feature = "experimental")]
icu_locid_transform_data::impl_locid_transform_script_dir_v1!(Baked);
};

Expand Down
1 change: 1 addition & 0 deletions ffi/capi_cdylib/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ logging = ["icu_capi/logging"]
simple_logger = ["icu_capi/simple_logger"]
wasm_default = ["icu_capi/wasm_default"]
cpp_default = ["icu_capi/cpp_default"]
default_components = ["icu_capi/default_components"]

[package.metadata.cargo-all-features]
# Tested by icu_capi
Expand Down
1 change: 1 addition & 0 deletions ffi/capi_staticlib/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ logging = ["icu_capi/logging"]
simple_logger = ["icu_capi/simple_logger"]
wasm_default = ["icu_capi/wasm_default"]
cpp_default = ["icu_capi/cpp_default"]
default_components = ["icu_capi/default_components"]

[package.metadata.cargo-all-features]
# Tested by icu_capi
Expand Down
2 changes: 1 addition & 1 deletion ffi/diplomat/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ simple_logger = ["dep:simple_logger"]
cpp_default = ["logging", "simple_logger"]

# meta feature for things we enable by default in wasm
wasm_default = ["buffer_provider", "logging"]
wasm_default = ["logging"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought: It would break people who rely on the buffer provider in WASM if we turn this off, but it's probably okay because I think not too many people are using WASM yet

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably ok. The node package should probably use wasm_default as it's the canonical way to use this, and should also use compiled data.


# Components
default_components = ["icu_calendar", "icu_collator", "icu_datetime", "icu_decimal",
Expand Down
3 changes: 1 addition & 2 deletions ffi/diplomat/c/examples/fixeddecimal_tiny/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,4 @@ optim*
*.elf
*.o
a.out.dSYM
decimal-bn-en.postcard
decimal_bn_en.h
baked_data
31 changes: 14 additions & 17 deletions ffi/diplomat/c/examples/fixeddecimal_tiny/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,54 +15,51 @@ GCC := gcc
CLANG := clang-14
LLD := lld-14

baked_data/macros.rs:
cargo run -p icu_datagen -- --locales en bn --keys all --fallback expand --format mod --use-separate-crates --out baked_data
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observation: Previously you had --keys "decimal/symbols@1" and now you have --keys all. This seems to slow down CI: running this step takes about 1m 18s on CI, whereas previously it took 1s.

You already depend on icu_capi with all features disabled except for icu_decimal. Can you generate the data with only the keys for the decimal component?

However, overall CI speed does not seem to be affected, so this is fine for now if it's not an easy fix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this actually made me notice that we should probably generate empty implementations for all keys, otherwise compiled data will not compile. I'll make that change in datagen and then undo the --keys all

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a separate PR


../../../../../target/debug/libicu_capi_staticlib.a: FORCE
cargo build -p icu_capi_staticlib --no-default-features --features buffer_provider,icu_capi/default_components
cargo build -p icu_capi_staticlib --no-default-features --features compiled_data,default_components

icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a: FORCE
icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a: FORCE baked_data/macros.rs
rustup toolchain install ${ICU4X_NIGHTLY_TOOLCHAIN}
rustup component add rust-src --toolchain ${ICU4X_NIGHTLY_TOOLCHAIN}
cd icu_capi_staticlib_tiny && \
RUSTFLAGS="-Clinker-plugin-lto -Clinker=$(CLANG) -Ccodegen-units=1 -Clink-arg=-flto -Cpanic=abort" cargo +${ICU4X_NIGHTLY_TOOLCHAIN} panic-abort-build --target x86_64-unknown-linux-gnu
RUSTFLAGS="-Clinker-plugin-lto -Clinker=$(CLANG) -Ccodegen-units=1 -Clink-arg=-flto -Cpanic=abort" ICU4X_DATA_DIR=$(shell pwd)/baked_data cargo +${ICU4X_NIGHTLY_TOOLCHAIN} panic-abort-build --target x86_64-unknown-linux-gnu

icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/release-opt-size/libicu_capi_staticlib_tiny.a: FORCE
icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/release-opt-size/libicu_capi_staticlib_tiny.a: FORCE baked_data/macros.rs
rustup toolchain install ${ICU4X_NIGHTLY_TOOLCHAIN}
rustup component add rust-src --toolchain ${ICU4X_NIGHTLY_TOOLCHAIN}
cd icu_capi_staticlib_tiny && \
RUSTFLAGS="-Clinker-plugin-lto -Clinker=$(CLANG) -Ccodegen-units=1 -Clink-arg=-flto -Cpanic=abort" cargo +${ICU4X_NIGHTLY_TOOLCHAIN} panic-abort-build --target x86_64-unknown-linux-gnu --profile=release-opt-size

decimal-bn-en.postcard:
cargo run -p icu_datagen -- --locales en bn --keys "decimal/symbols@1" --format blob --out decimal-bn-en.postcard

decimal_bn_en.h: decimal-bn-en.postcard
xxd -i -C decimal-bn-en.postcard > decimal_bn_en.h
RUSTFLAGS="-Clinker-plugin-lto -Clinker=$(CLANG) -Ccodegen-units=1 -Clink-arg=-flto -Cpanic=abort" ICU4X_DATA_DIR=$(shell pwd)/baked_data cargo +${ICU4X_NIGHTLY_TOOLCHAIN} panic-abort-build --target x86_64-unknown-linux-gnu --profile=release-opt-size

# Naive target: no optimizations, full std
optim0.elf: ../../../../../target/debug/libicu_capi_staticlib.a $(ALL_HEADERS) test.c decimal_bn_en.h
optim0.elf: ../../../../../target/debug/libicu_capi_staticlib.a $(ALL_HEADERS) test.c
$(GCC) test.c ../../../../../target/debug/libicu_capi_staticlib.a -ldl -lpthread -lm -g -o optim0.elf

# optim.elf: gcc with maximum link-time code stripping (gc-sections and strip-all)
optim1.elf: ../../../../../target/debug/libicu_capi_staticlib.a $(ALL_HEADERS) test.c decimal_bn_en.h
optim1.elf: ../../../../../target/debug/libicu_capi_staticlib.a $(ALL_HEADERS) test.c
$(GCC) -fdata-sections -ffunction-sections test.c ../../../../../target/debug/libicu_capi_staticlib.a -ldl -lpthread -lm -g -o optim1.elf -Wl,--gc-sections -Wl,--strip-all

# optim2.elf: clang single-step with gc-sections
optim2.elf: icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a $(ALL_HEADERS) test.c decimal_bn_en.h
optim2.elf: icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a $(ALL_HEADERS) test.c
$(CLANG) -flto -fdata-sections -ffunction-sections test.c icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a -g -o optim2.elf -Wl,--gc-sections

optim3.o: $(ALL_HEADERS) test.c decimal_bn_en.h
optim3.o: $(ALL_HEADERS) test.c
$(CLANG) -c -flto=thin -fdata-sections -ffunction-sections --target=x86_64-unknown-linux-gnu test.c -g -o optim3.o

# optim3.elf: clang two-step with lld, debug mode
optim3.elf: optim3.o icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a
$(CLANG) -flto=thin -fuse-ld=$(LLD) -L . -o optim3.elf optim3.o icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/debug/libicu_capi_staticlib_tiny.a -Wl,--gc-sections

optim4.o: $(ALL_HEADERS) test.c decimal_bn_en.h
optim4.o: $(ALL_HEADERS) test.c
$(CLANG) -c -flto=thin -fdata-sections -ffunction-sections --target=x86_64-unknown-linux-gnu test.c -g -o optim4.o

# optim4.elf: clang two-step with lld, release mode with debug symbols
optim4.elf: optim4.o icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/release-opt-size/libicu_capi_staticlib_tiny.a
$(CLANG) -flto=thin -fuse-ld=$(LLD) -L . -o optim4.elf optim4.o icu_capi_staticlib_tiny/target/x86_64-unknown-linux-gnu/release-opt-size/libicu_capi_staticlib_tiny.a -Wl,--gc-sections

optim5.o: $(ALL_HEADERS) test.c decimal_bn_en.h
optim5.o: $(ALL_HEADERS) test.c
$(CLANG) -c -flto=thin -fdata-sections -ffunction-sections --target=x86_64-unknown-linux-gnu test.c -o optim5.o

# optim5.elf: clang two-step with lld, release mode stripped of debug symbols
Expand Down
Loading