Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rustbuild rebuilds too much on modifications to rustdoc #38318

Closed
ollie27 opened this issue Dec 12, 2016 · 3 comments · Fixed by #43482
Closed

rustbuild rebuilds too much on modifications to rustdoc #38318

ollie27 opened this issue Dec 12, 2016 · 3 comments · Fixed by #43482
Labels
C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@ollie27
Copy link
Member

ollie27 commented Dec 12, 2016

Modifying just rustdoc and rebuilding with the Makefiles by running make doc results in the following:

cfg: version 1.15.0-dev (368e092c2 2016-12-11)
cfg: build triple x86_64-pc-windows-msvc
cfg: host triples x86_64-pc-windows-msvc
cfg: target triples x86_64-pc-windows-msvc
cfg: host for x86_64-pc-windows-msvc is x86_64
cfg: os for x86_64-pc-windows-msvc is pc-windows-msvc
cfg: no good valgrind for x86_64-pc-windows-msvc
cfg: using CC=gcc (CFG_CC)
cfg: disabling valgrind run-pass tests
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/lib/librustdoc
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/rustbook.exe
cp: x86_64-pc-windows-msvc/stage2/bin/librustdoc
cp: x86_64-pc-windows-msvc/stage2/bin/rustbook.exe
rustbook: doc/book/index.html
rustbook: doc/nomicon/index.html
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/error_index_generator.exe
cp: x86_64-pc-windows-msvc/stage2/bin/error_index_generator.exe
error_index_generator: doc/error-index.html
rustc: x86_64-pc-windows-msvc/stage1/lib/rustlib/x86_64-pc-windows-msvc/bin/rustdoc.exe
cp: x86_64-pc-windows-msvc/stage2/bin/rustdoc.exe
rustdoc: doc/not_found.html
rustdoc: doc/index.html
rustdoc: doc/complement-lang-faq.html
rustdoc: doc/complement-design-faq.html
rustdoc: doc/complement-project-faq.html
rustdoc: doc/rustdoc.html
rustdoc: doc/reference.html
rustdoc: doc/grammar.html
rustdoc: doc/guide-crates.html
rustdoc: doc/guide-error-handling.html
rustdoc: doc/guide-ffi.html
rustdoc: doc/guide-macros.html
rustdoc: doc/guide.html
rustdoc: doc/guide-ownership.html
rustdoc: doc/guide-plugins.html
rustdoc: doc/guide-pointers.html
rustdoc: doc/guide-strings.html
rustdoc: doc/guide-tasks.html
rustdoc: doc/guide-testing.html
rustdoc: doc/tutorial.html
rustdoc: doc/intro.html
rustdoc: doc/std/index.html
rustdoc: doc/alloc/index.html
rustdoc: doc/collections/index.html
rustdoc: doc/core/index.html
rustdoc: doc/libc/index.html
rustdoc: doc/rustc_unicode/index.html

Which doesn't take long. However with rustbuild python x.py doc --stage 1 results in:

    Finished debug [unoptimized] target(s) in 0.0 secs
Synchronizing submodule url for 'src/compiler-rt'
Synchronizing submodule url for 'src/jemalloc'
Synchronizing submodule url for 'src/liblibc'
Synchronizing submodule url for 'src/llvm'
Synchronizing submodule url for 'src/rt/hoedown'
Synchronizing submodule url for 'src/rust-installer'
HEAD is now at a8fc4c16 Merge pull request #28 from xen0n/preprocessor-firefighting
HEAD is now at 0ac39c5cc Auto merge of #462 - semarie:openbsd-i686-correct, r=alexcrichton
HEAD is now at d7342a9a957 Merge pull request #59 from pftbest/msp430_backport
HEAD is now at a3736a0 Merge pull request #6 from intelfx/patch-1
HEAD is now at 4f99485 Merge pull request #54 from brson/docdir
Building stage0 std artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Building stage0 test artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Building stage0 compiler artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling rustdoc v0.0.0 (file:///E:/Source/rust/src/librustdoc)
   Compiling rustc-main v0.0.0 (file:///E:/Source/rust/src/rustc)
    Finished release [optimized] target(s) in 95.30 secs
Building stage1 std artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling core v0.0.0 (file:///E:/Source/rust/src/libcore)
   Compiling unwind v0.0.0 (file:///E:/Source/rust/src/libunwind)
   Compiling libc v0.0.0 (file:///E:/Source/rust/src/rustc/libc_shim)
   Compiling compiler_builtins v0.0.0 (file:///E:/Source/rust/src/libcompiler_builtins)
   Compiling std v0.0.0 (file:///E:/Source/rust/src/libstd)
   Compiling alloc v0.0.0 (file:///E:/Source/rust/src/liballoc)
   Compiling rand v0.0.0 (file:///E:/Source/rust/src/librand)
   Compiling rustc_unicode v0.0.0 (file:///E:/Source/rust/src/librustc_unicode)
   Compiling alloc_system v0.0.0 (file:///E:/Source/rust/src/liballoc_system)
   Compiling panic_abort v0.0.0 (file:///E:/Source/rust/src/libpanic_abort)
   Compiling panic_unwind v0.0.0 (file:///E:/Source/rust/src/libpanic_unwind)
   Compiling collections v0.0.0 (file:///E:/Source/rust/src/libcollections)
   Compiling std_shim v0.1.0 (file:///E:/Source/rust/src/rustc/std_shim)
    Finished release [optimized] target(s) in 37.87 secs
Building stage1 test artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling getopts v0.0.0 (file:///E:/Source/rust/src/libgetopts)
   Compiling term v0.0.0 (file:///E:/Source/rust/src/libterm)
   Compiling test v0.0.0 (file:///E:/Source/rust/src/libtest)
   Compiling test_shim v0.1.0 (file:///E:/Source/rust/src/rustc/test_shim)
    Finished release [optimized] target(s) in 13.52 secs
Building stage1 compiler artifacts (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
   Compiling rustc_platform_intrinsics v0.0.0 (file:///E:/Source/rust/src/librustc_platform_intrinsics)
   Compiling rustc_bitflags v0.0.0 (file:///E:/Source/rust/src/librustc_bitflags)
   Compiling log v0.0.0 (file:///E:/Source/rust/src/liblog)
   Compiling arena v0.0.0 (file:///E:/Source/rust/src/libarena)
   Compiling fmt_macros v0.0.0 (file:///E:/Source/rust/src/libfmt_macros)
   Compiling graphviz v0.0.0 (file:///E:/Source/rust/src/libgraphviz)
   Compiling flate v0.0.0 (file:///E:/Source/rust/src/libflate)
   Compiling rustc_llvm v0.0.0 (file:///E:/Source/rust/src/librustc_llvm)
   Compiling rustdoc v0.0.0 (file:///E:/Source/rust/src/librustdoc)
   Compiling serialize v0.0.0 (file:///E:/Source/rust/src/libserialize)
   Compiling rustc_data_structures v0.0.0 (file:///E:/Source/rust/src/librustc_data_structures)
   Compiling syntax_pos v0.0.0 (file:///E:/Source/rust/src/libsyntax_pos)
   Compiling rustc_errors v0.0.0 (file:///E:/Source/rust/src/librustc_errors)
   Compiling syntax v0.0.0 (file:///E:/Source/rust/src/libsyntax)
   Compiling rustc_back v0.0.0 (file:///E:/Source/rust/src/librustc_back)
   Compiling proc_macro v0.0.0 (file:///E:/Source/rust/src/libproc_macro)
   Compiling proc_macro_tokens v0.0.0 (file:///E:/Source/rust/src/libproc_macro_tokens)
   Compiling rustc_const_math v0.0.0 (file:///E:/Source/rust/src/librustc_const_math)
   Compiling syntax_ext v0.0.0 (file:///E:/Source/rust/src/libsyntax_ext)
   Compiling rustc v0.0.0 (file:///E:/Source/rust/src/librustc)
   Compiling rustc_const_eval v0.0.0 (file:///E:/Source/rust/src/librustc_const_eval)
   Compiling rustc_metadata v0.0.0 (file:///E:/Source/rust/src/librustc_metadata)
   Compiling rustc_privacy v0.0.0 (file:///E:/Source/rust/src/librustc_privacy)
   Compiling rustc_incremental v0.0.0 (file:///E:/Source/rust/src/librustc_incremental)
   Compiling rustc_resolve v0.0.0 (file:///E:/Source/rust/src/librustc_resolve)
   Compiling rustc_save_analysis v0.0.0 (file:///E:/Source/rust/src/librustc_save_analysis)
   Compiling rustc_mir v0.0.0 (file:///E:/Source/rust/src/librustc_mir)
   Compiling rustc_passes v0.0.0 (file:///E:/Source/rust/src/librustc_passes)
   Compiling rustc_lint v0.0.0 (file:///E:/Source/rust/src/librustc_lint)
   Compiling rustc_typeck v0.0.0 (file:///E:/Source/rust/src/librustc_typeck)
   Compiling rustc_trans v0.0.0 (file:///E:/Source/rust/src/librustc_trans)
   Compiling rustc_borrowck v0.0.0 (file:///E:/Source/rust/src/librustc_borrowck)
   Compiling rustc_plugin v0.0.0 (file:///E:/Source/rust/src/librustc_plugin)
   Compiling proc_macro_plugin v0.0.0 (file:///E:/Source/rust/src/libproc_macro_plugin)
   Compiling rustc_driver v0.0.0 (file:///E:/Source/rust/src/librustc_driver)
   Compiling rustc-main v0.0.0 (file:///E:/Source/rust/src/rustc)
    Finished release [optimized] target(s) in 676.85 secs
Building stage1 tool rustbook (x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Documenting stage1 standalone (x86_64-pc-windows-msvc)
Documenting stage1 std (x86_64-pc-windows-msvc)
 Documenting core v0.0.0 (file:///E:/Source/rust/src/libcore)
 Documenting libc v0.0.0 (file:///E:/Source/rust/src/rustc/libc_shim)
 Documenting rustc_unicode v0.0.0 (file:///E:/Source/rust/src/librustc_unicode)
 Documenting compiler_builtins v0.0.0 (file:///E:/Source/rust/src/libcompiler_builtins)
 Documenting alloc v0.0.0 (file:///E:/Source/rust/src/liballoc)
 Documenting rand v0.0.0 (file:///E:/Source/rust/src/librand)
 Documenting alloc_system v0.0.0 (file:///E:/Source/rust/src/liballoc_system)
 Documenting panic_abort v0.0.0 (file:///E:/Source/rust/src/libpanic_abort)
 Documenting unwind v0.0.0 (file:///E:/Source/rust/src/libunwind)
 Documenting panic_unwind v0.0.0 (file:///E:/Source/rust/src/libpanic_unwind)
 Documenting collections v0.0.0 (file:///E:/Source/rust/src/libcollections)
 Documenting std v0.0.0 (file:///E:/Source/rust/src/libstd)
 Documenting std_shim v0.1.0 (file:///E:/Source/rust/src/rustc/std_shim)
Building stage1 tool error_index_generator (x86_64-pc-windows-msvc)
    Finished release [optimized] target(s) in 0.0 secs
Documenting stage1 error index (x86_64-pc-windows-msvc)
Documenting stage1 test (x86_64-pc-windows-msvc)
 Documenting getopts v0.0.0 (file:///E:/Source/rust/src/libgetopts)
 Documenting term v0.0.0 (file:///E:/Source/rust/src/libterm)
 Documenting test v0.0.0 (file:///E:/Source/rust/src/libtest)
 Documenting test_shim v0.1.0 (file:///E:/Source/rust/src/rustc/test_shim)
Build completed in 0:14:33

Is there a faster way to rebuild rustdoc and the docs when using rustbuild?
@bluss bluss added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Dec 12, 2016
@alexcrichton
Copy link
Member

Yes rustbuild builds rustdoc in all three stages as opposed to just the end (which the makefiles did).

We may be able to get away with only detecting/cleaning between stages if rustc itself changes, and ignoring changes to rustdoc except for when we think we need to clean docs.

An alternative would be to work with ./x.py doc --stage 0 ...

@Mark-Simulacrum
Copy link
Member

Would #38331 help with this? In theory you could then --keep-stage 1 and then rustbuild would only rebuild post-stage 1, I believe. This might be wrong in practice, though.

@QuietMisdreavus
Copy link
Member

It seems that ./x.py doc doesn't interact with either the --stage or the --keep-stage arguments well at all. Like the default (--stage 2), --stage 1 unilaterally compiles all of stage 1, even if the only change was within librustdoc. --stage 0 seems to only rebuild rustdoc, but then doesn't use it to rebuild the std docs. It just automatically treats it as complete.

--stage 1 --keep-stage 0, when run immediately after --stage 0, proceeds to rebuild the entirety of stage 1. Otherwise, it rebuilds only what's necessary in stage1 and skips actually regenerating the docs, just like --stage 0.

Confusingly, before I decided to systematically dive into this, these combinations would also use old builds of rustdoc for its documentation rendering, or would generate dynamic-link errors by trying to reference .so files from earlier stages but using hashes from later on. I would dive further into this, but my development environment is a tiny server that takes 15 minutes to rebuild a stage, so I'll leave the report at this for now. For the moment it seems like I need to build all of std and rustc, at least once, just to test a rustdoc change.

@QuietMisdreavus QuietMisdreavus mentioned this issue Jun 15, 2017
Closed
@ollie27 ollie27 mentioned this issue Jul 17, 2017
Closed
@Mark-Simulacrum Mark-Simulacrum mentioned this issue Jul 25, 2017
Merged
@Mark-Simulacrum Mark-Simulacrum added the C-bug Category: This is a bug. label Jul 26, 2017
bors added a commit that referenced this issue Jul 27, 2017
@bors
Auto merge of #43482 - Mark-Simulacrum:single-rustdoc, r=alexcrichton
5cc1baa 
Compile rustdoc on-demand

Fixes #43284, fixes #38318, and fixes #39505.

Doesn't directly help with #42686, since we need to rebuild just as much. In fact, this hurts it, since `./x.py doc --stage 0` will now fail. I'm not sure if it did before, but with these changes it runs into the problem where we attempt to use artifacts from bootstrap rustc with a non-bootstrap rustdoc, running into version conflicts. I believe this is solvable, but leaving for a future PR.

This means that rustdoc will no longer be compiled when compiling rustc, by default. However, it is still built from `./x.py build` (for hosts, but not targets, since we don't produce compiler toolchains for them) and will be built for doc tests and crate tests.

After this, the recommended workflow if you want a rustdoc is: `./x.py build --stage 1 src/tools/rustdoc` which will give you a working rustdoc in `build/triple/stage1/bin/rustdoc`. Note that you can add `src/libstd` onto the command to compile libstd as well so that the rustdoc can easily compile crates in the wild. `./x.py doc --stage 1 src/libstd` will document `libstd` with a freshly built rustdoc (if necessary), and will not rebuild rustc on modifications to rustdoc.

r? @alexcrichton
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Compile rustdoc on-demand Mark-Simulacrum/rust
5 participants
@alexcrichton @bluss @Mark-Simulacrum @QuietMisdreavus @ollie27