Purge Python #110479

Ezrashaw · 2023-04-18T04:24:49Z

Running the command: fd --extension py -E src/llvm-project gives a list of python files in rust-lang/rust:

compiler/rustc_codegen_gcc/tools/check_intrinsics_duplicates.py
compiler/rustc_codegen_gcc/tools/generate_intrinsics.py
library/core/src/unicode/printable.py
src/bootstrap/bootstrap.py
src/bootstrap/bootstrap_test.py
src/bootstrap/configure.py
src/ci/cpu-usage-over-time.py
src/ci/docker/host-x86_64/test-various/uefi_qemu_test/run.py
src/ci/docker/scripts/android-sdk-manager.py
src/ci/docker/scripts/fuchsia-test-runner.py
src/ci/github-actions/calculate-job-matrix.py
src/ci/scripts/upload-build-metrics.py
src/etc/dec2flt_table.py
src/etc/gdb_load_rust_pretty_printers.py
src/etc/gdb_lookup.py
src/etc/gdb_providers.py
src/etc/generate-deriving-span-tests.py
src/etc/generate-keyword-tests.py
src/etc/htmldocck.py
src/etc/lldb_batchmode.py
src/etc/lldb_lookup.py
src/etc/lldb_providers.py
src/etc/rust_types.py
src/tools/cargo/ci/generate.py
src/tools/cargo/publish.py
src/tools/clippy/util/versions.py
src/tools/miri/ci/scrape-targets.py
src/tools/miri/test-cargo-miri/run-test.py
src/tools/publish_toolstate.py
tests/debuginfo/auxiliary/dependency-with-embedded-visualizers.py
tests/debuginfo/embedded-visualizer-point.py
tests/debuginfo/embedded-visualizer.py
tests/run-make/debugger-visualizer-dep-info/my_gdb_script.py
tests/run-make/libtest-junit/validate_junit.py
x.py

While there will (probably) always be a need for a little bit of Python in bootstrap, we should aim to replace misc scripts that really should be Written In Rust. Especially relevant are files like src/etc/htmldocck.py which is the rustdoc UI test harness(!); we should rewrite this in Rust.

Extra points:

I wonder if CI scripts do actually need to be Python?
Longer term, I wonder if we can require a host installation of Rust (we just download one anyway) and use that for bootstrap?

The text was updated successfully, but these errors were encountered:

Ezrashaw · 2023-04-18T04:25:14Z

cc @jyn514 am I crazy about getting rid of Python from bootstrap?

jyn514 · 2023-04-18T04:34:43Z

I don't see the point in rewriting all this code. The main goal of #94829 is so we don't make people deal with python before they start working on the repo; but half the scripts you mentioned are for CI only. What do we get from rewriting them?

Ezrashaw · 2023-04-18T04:44:06Z

I guess maintainability in a way. I mean, why is compiletest not written in Python? It then makes sense to have the rustdoc part of it written in the same language, possibly src/etc/htmldocck.py could be rewritten and moved into compiletest itself. I'm not saying that this is important, just that it's something that should happen at some point.

jyn514 · 2023-04-18T04:48:09Z

I don't think this is worth spending contributor time on. There are other ways people can get started contributing (https://rustc-dev-guide.rust-lang.org/#what-should-i-work-on).

Ezrashaw · 2023-04-18T04:54:20Z

I agree, I think it's just good to have this here as a long-term goal.

compiler-errors · 2023-04-18T05:45:21Z

It's probably worthwhile to first step back and characterize the existing python that we have. I think only once it's clear what python actually can be rewritten should we actually discuss whether it should be rewritten.

e.g. those gdb and llvm scripts can't be rewritten in rust afaict.

Ezrashaw · 2023-04-18T06:05:06Z

Yeah, perhaps CI scripts as well might not be rewritable?

I do think there is some low-hanging fruit here which could have tangible benefits.

thomcc · 2023-04-18T16:10:13Z

I have a rewrite of library/core/src/unicode/printable.py on another machine, moving it into the rest of the table generation code. I'll try to get it up when I'm more easily able to contribute.

thomcc · 2023-06-04T17:33:00Z

Assigning to myself to do this for the unicode tables because @jyn514 indicated it might be the only one that's worth doing (no strong opinions from me, although I don't like that one being python)

Port the `sysroot-crates-are-unstable` Python script to rmake New version of rust-lang#126231, and a follow-up to rust-lang#129071. One major difference is that the new version no longer tries to report *all* accidentally-stable crates, because the `run_make_support` helpers tend to halt the test as soon as something goes wrong. That's unfortunate, but I think it won't matter much in practice, and preserving the old behaviour doesn't seem worth the extra effort. --- Part of rust-lang#110479 (Python purge), with this being one of the non-trivial Python scripts that actually seems feasible and worthwhile to remove. This is *not* part of rust-lang#121876 (Makefile purge), because the underlying test is already using rmake; this PR just modifies the existing rmake recipe to do all the work itself instead of delegating to Python. So there's no particular urgency here. r? `@jieyouxu` try-job: aarch64-gnu try-job: aarch64-apple try-job: test-various try-job: armhf-gnu try-job: x86_64-msvc try-job: i686-mingw

Port the `sysroot-crates-are-unstable` Python script to rmake New version of rust-lang#126231, and a follow-up to rust-lang#129071. One major difference is that the new version no longer tries to report *all* accidentally-stable crates, because the `run_make_support` helpers tend to halt the test as soon as something goes wrong. That's unfortunate, but I think it won't matter much in practice, and preserving the old behaviour doesn't seem worth the extra effort. --- Part of rust-lang#110479 (Python purge), with this being one of the non-trivial Python scripts that actually seems feasible and worthwhile to remove. This is *not* part of rust-lang#121876 (Makefile purge), because the underlying test is already using rmake; this PR just modifies the existing rmake recipe to do all the work itself instead of delegating to Python. So there's no particular urgency here. r? ``@jieyouxu`` try-job: aarch64-gnu try-job: aarch64-apple try-job: test-various try-job: armhf-gnu try-job: x86_64-msvc try-job: i686-mingw

Port the `sysroot-crates-are-unstable` Python script to rmake New version of rust-lang#126231, and a follow-up to rust-lang#129071. One major difference is that the new version no longer tries to report *all* accidentally-stable crates, because the `run_make_support` helpers tend to halt the test as soon as something goes wrong. That's unfortunate, but I think it won't matter much in practice, and preserving the old behaviour doesn't seem worth the extra effort. --- Part of rust-lang#110479 (Python purge), with this being one of the non-trivial Python scripts that actually seems feasible and worthwhile to remove. This is *not* part of rust-lang#121876 (Makefile purge), because the underlying test is already using rmake; this PR just modifies the existing rmake recipe to do all the work itself instead of delegating to Python. So there's no particular urgency here. r? ```@jieyouxu``` try-job: aarch64-gnu try-job: aarch64-apple try-job: test-various try-job: armhf-gnu try-job: x86_64-msvc try-job: i686-mingw

Port the `sysroot-crates-are-unstable` Python script to rmake New version of rust-lang#126231, and a follow-up to rust-lang#129071. One major difference is that the new version no longer tries to report *all* accidentally-stable crates, because the `run_make_support` helpers tend to halt the test as soon as something goes wrong. That's unfortunate, but I think it won't matter much in practice, and preserving the old behaviour doesn't seem worth the extra effort. --- Part of rust-lang#110479 (Python purge), with this being one of the non-trivial Python scripts that actually seems feasible and worthwhile to remove. This is *not* part of rust-lang#121876 (Makefile purge), because the underlying test is already using rmake; this PR just modifies the existing rmake recipe to do all the work itself instead of delegating to Python. So there's no particular urgency here. r? ````@jieyouxu```` try-job: aarch64-gnu try-job: aarch64-apple try-job: test-various try-job: armhf-gnu try-job: x86_64-msvc try-job: i686-mingw

Rollup merge of rust-lang#129111 - Zalathar:python-sysroot, r=jieyouxu Port the `sysroot-crates-are-unstable` Python script to rmake New version of rust-lang#126231, and a follow-up to rust-lang#129071. One major difference is that the new version no longer tries to report *all* accidentally-stable crates, because the `run_make_support` helpers tend to halt the test as soon as something goes wrong. That's unfortunate, but I think it won't matter much in practice, and preserving the old behaviour doesn't seem worth the extra effort. --- Part of rust-lang#110479 (Python purge), with this being one of the non-trivial Python scripts that actually seems feasible and worthwhile to remove. This is *not* part of rust-lang#121876 (Makefile purge), because the underlying test is already using rmake; this PR just modifies the existing rmake recipe to do all the work itself instead of delegating to Python. So there's no particular urgency here. r? ````@jieyouxu```` try-job: aarch64-gnu try-job: aarch64-apple try-job: test-various try-job: armhf-gnu try-job: x86_64-msvc try-job: i686-mingw

Kobzol · 2024-08-16T07:30:59Z

Once Cargo script becomes stabilized, it should become feasible to turn some of these Python scripts into one-file Rust scripsts (of course only where it would make sense, there's no point in doing that proactively).

Zalathar · 2024-08-22T03:45:58Z

Of the remaining Python scripts in tests, most are gdb visualizer scripts, so trying to remove those doesn't make sense.

The one outlier is run-make/libtest-junit/validate_junit.py, which just takes advantage of Python's built-in XML parser to check for well-formed XML output. As noted in #129387, it's not worth pulling in a Rust XML crate just to get rid of one trivial Python script in one relatively-minor test.

(If we ever reach the point where building/testing the compiler without Python is feasible, we can just add a compiletest directive to skip that test if Python isn't present.)

Zalathar · 2024-08-22T03:54:13Z

Looking at the rest of the list, I mostly see:

More gdb scripts
CI scripts where there is presumably some value in being able to run independently of Rust
- EDIT: And in that case Python is better than Bash or similar
Infrequently-run code generator scripts that in principle should be ported to Rust, but where there's not much reason to actually port them unless people start running into specific maintenance headaches with the existing scripts

ChrisDenton · 2024-08-22T12:05:51Z

In terms of CI scripts, I think the choice is often Python or Bash. And I do prefer we use Python over Bash for anything non-trivial.

Noratrieb added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Apr 18, 2023

thomcc self-assigned this Jun 4, 2023

Zalathar mentioned this issue Jun 10, 2024

Port tests/run-make/sysroot-crates-are-unstable from Python to rmake #126231

Closed

Zalathar mentioned this issue Aug 15, 2024

Port the sysroot-crates-are-unstable Python script to rmake #129111

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Purge Python #110479

Purge Python #110479

Ezrashaw commented Apr 18, 2023 •

edited by Zalathar

Loading

Ezrashaw commented Apr 18, 2023

jyn514 commented Apr 18, 2023

Ezrashaw commented Apr 18, 2023

jyn514 commented Apr 18, 2023

Ezrashaw commented Apr 18, 2023

compiler-errors commented Apr 18, 2023 •

edited

Loading

Ezrashaw commented Apr 18, 2023

thomcc commented Apr 18, 2023

thomcc commented Jun 4, 2023

Kobzol commented Aug 16, 2024

Zalathar commented Aug 22, 2024 •

edited

Loading

Zalathar commented Aug 22, 2024 •

edited

Loading

ChrisDenton commented Aug 22, 2024

Purge Python #110479

Purge Python #110479

Comments

Ezrashaw commented Apr 18, 2023 • edited by Zalathar Loading

Ezrashaw commented Apr 18, 2023

jyn514 commented Apr 18, 2023

Ezrashaw commented Apr 18, 2023

jyn514 commented Apr 18, 2023

Ezrashaw commented Apr 18, 2023

compiler-errors commented Apr 18, 2023 • edited Loading

Ezrashaw commented Apr 18, 2023

thomcc commented Apr 18, 2023

thomcc commented Jun 4, 2023

Kobzol commented Aug 16, 2024

Zalathar commented Aug 22, 2024 • edited Loading

Zalathar commented Aug 22, 2024 • edited Loading

ChrisDenton commented Aug 22, 2024

Ezrashaw commented Apr 18, 2023 •

edited by Zalathar

Loading

compiler-errors commented Apr 18, 2023 •

edited

Loading

Zalathar commented Aug 22, 2024 •

edited

Loading

Zalathar commented Aug 22, 2024 •

edited

Loading