Auto merge of #47828 - alexcrichton:llvm-6, r=<try>

rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3
rust-lang · Jan 28, 2018 · 38bd381 · 38bd381
2 parents 771873c + d9031ae
commit 38bd381
Show file tree

Hide file tree

Showing 28 changed files with 279 additions and 133 deletions.
diff --git a/.gitmodules b/.gitmodules
@@ -51,3 +51,6 @@
 [submodule "src/doc/rust-by-example"]
 	path = src/doc/rust-by-example
 	url = https://github.com/rust-lang/rust-by-example
+[submodule "src/llvm-emscripten"]
+	path = src/llvm-emscripten
+	url = https://github.com/rust-lang/llvm
diff --git a/.travis.yml b/.travis.yml
@@ -81,7 +81,7 @@ matrix:
     # OSX 10.7 and `xcode7` is the latest Xcode able to compile LLVM for 10.7.
     - env: >
         RUST_CHECK_TARGET=dist
-        RUST_CONFIGURE_ARGS="--build=i686-apple-darwin --enable-extended --enable-profiler"
+        RUST_CONFIGURE_ARGS="--build=i686-apple-darwin --enable-extended --enable-profiler --enable-emscripten"
         SRC=.
         DEPLOY=1
         RUSTC_RETRY_LINKER_ON_SEGFAULT=1
@@ -95,7 +95,7 @@ matrix:
 
     - env: >
         RUST_CHECK_TARGET=dist
-        RUST_CONFIGURE_ARGS="--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler"
+        RUST_CONFIGURE_ARGS="--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler --enable-emscripten"
         SRC=.
         DEPLOY=1
         RUSTC_RETRY_LINKER_ON_SEGFAULT=1

diff --git a/appveyor.yml b/appveyor.yml
@@ -63,25 +63,27 @@ environment:
       --build=x86_64-pc-windows-msvc
       --enable-extended
       --enable-profiler
+      --enable-emscripten
     SCRIPT: python x.py dist
     DEPLOY: 1
   - RUST_CONFIGURE_ARGS: >
       --build=i686-pc-windows-msvc
       --target=i586-pc-windows-msvc
       --enable-extended
       --enable-profiler
+      --enable-emscripten
     SCRIPT: python x.py dist
     DEPLOY: 1
   - MSYS_BITS: 32
-    RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu --enable-extended
+    RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu --enable-extended --enable-emscripten
     SCRIPT: python x.py dist
     MINGW_URL: https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror
     MINGW_ARCHIVE: i686-6.3.0-release-posix-dwarf-rt_v5-rev2.7z
     MINGW_DIR: mingw32
     DEPLOY: 1
   - MSYS_BITS: 64
     SCRIPT: python x.py dist
-    RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu --enable-extended
+    RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu --enable-extended --enable-emscripten
     MINGW_URL: https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror
     MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z
     MINGW_DIR: mingw64

diff --git a/config.toml.example b/config.toml.example
@@ -305,6 +305,13 @@
 # result (broken, compiling, testing) into this JSON file.
 #save-toolstates = "/path/to/toolstates.json"
 
+# This is an array of the codegen backends that will be compiled for the rustc
+# that's being compiled. The default is to only build the LLVM codegen backend,
+# but you can also optionally enable the "emscripten" backend for asm.js or
+# make this an empty array (but that probably won't get too far in the
+# bootstrap)
+#codegen-backends = ["llvm"]
+
 # =============================================================================
 # Options for specific targets
 #

diff --git a/src/bootstrap/bootstrap.py b/src/bootstrap/bootstrap.py
@@ -640,14 +640,23 @@ def update_submodules(self):
              os.path.join(self.rust_root, ".gitmodules"),
              "--get-regexp", "path"]
         ).decode(default_encoding).splitlines()]
-        submodules = [module for module in submodules
-                      if not ((module.endswith("llvm") and
-                               self.get_toml('llvm-config')) or
-                              (module.endswith("jemalloc") and
-                               (self.get_toml('use-jemalloc') == "false" or
-                                self.get_toml('jemalloc'))))]
+        filtered_submodules = []
+        for module in submodules:
+            if module.endswith("llvm"):
+                if self.get_toml('llvm-config'):
+                    continue
+            if module.endswith("llvm-emscripten"):
+                backends = self.get_toml('codegen-backends')
+                if backends is None or not 'emscripten' in backends:
+                    continue
+            if module.endswith("jemalloc"):
+                if self.get_toml('use-jemalloc') == 'false':
+                    continue
+                if self.get_toml('jemalloc'):
+                    continue
+            filtered_submodules.append(module)
         run(["git", "submodule", "update",
-             "--init", "--recursive"] + submodules,
+             "--init", "--recursive"] + filtered_submodules,
             cwd=self.rust_root, verbose=self.verbose)
         run(["git", "submodule", "-q", "foreach", "git",
              "reset", "-q", "--hard"],

diff --git a/src/bootstrap/compile.rs b/src/bootstrap/compile.rs
@@ -581,24 +581,30 @@ impl Step for RustcLink {
 }
 
 #[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
-pub struct RustcTrans {
+pub struct CodegenBackend {
     pub compiler: Compiler,
     pub target: Interned<String>,
+    pub backend: Interned<String>,
 }
 
-impl Step for RustcTrans {
+impl Step for CodegenBackend {
     type Output = ();
     const ONLY_HOSTS: bool = true;
     const DEFAULT: bool = true;
 
     fn should_run(run: ShouldRun) -> ShouldRun {
-        run.path("src/librustc_trans").krate("rustc_trans")
+        run.path("src/librustc_trans")
     }
 
     fn make_run(run: RunConfig) {
-        run.builder.ensure(RustcTrans {
+        let backend = run.builder.config.rust_codegen_backends.get(0);
+        let backend = backend.cloned().unwrap_or_else(|| {
+            INTERNER.intern_str("llvm")
+        });
+        run.builder.ensure(CodegenBackend {
             compiler: run.builder.compiler(run.builder.top_stage, run.host),
             target: run.target,
+            backend
         });
     }
 
@@ -609,58 +615,92 @@ impl Step for RustcTrans {
 
         builder.ensure(Rustc { compiler, target });
 
-        // Build LLVM for our target. This will implicitly build the host LLVM
-        // if necessary.
-        builder.ensure(native::Llvm { target });
-
         if build.force_use_stage1(compiler, target) {
-            builder.ensure(RustcTrans {
+            builder.ensure(CodegenBackend {
                 compiler: builder.compiler(1, build.build),
                 target,
+                backend: self.backend,
             });
             return;
         }
 
-        let _folder = build.fold_output(|| format!("stage{}-rustc_trans", compiler.stage));
-        println!("Building stage{} trans artifacts ({} -> {})",
-                 compiler.stage, &compiler.host, target);
-
         let mut cargo = builder.cargo(compiler, Mode::Librustc, target, "build");
+        let mut features = build.rustc_features().to_string();
         cargo.arg("--manifest-path")
-            .arg(build.src.join("src/librustc_trans/Cargo.toml"))
-            .arg("--features").arg(build.rustc_features());
+            .arg(build.src.join("src/librustc_trans/Cargo.toml"));
         rustc_cargo_env(build, &mut cargo);
 
-        // Pass down configuration from the LLVM build into the build of
-        // librustc_llvm and librustc_trans.
-        if build.is_rust_llvm(target) {
-            cargo.env("LLVM_RUSTLLVM", "1");
-        }
-        cargo.env("LLVM_CONFIG", build.llvm_config(target));
-        let target_config = build.config.target_config.get(&target);
-        if let Some(s) = target_config.and_then(|c| c.llvm_config.as_ref()) {
-            cargo.env("CFG_LLVM_ROOT", s);
-        }
-        // Building with a static libstdc++ is only supported on linux right now,
-        // not for MSVC or macOS
-        if build.config.llvm_static_stdcpp &&
-           !target.contains("freebsd") &&
-           !target.contains("windows") &&
-           !target.contains("apple") {
-            let file = compiler_file(build,
-                                     build.cxx(target).unwrap(),
-                                     target,
-                                     "libstdc++.a");
-            cargo.env("LLVM_STATIC_STDCPP", file);
-        }
-        if build.config.llvm_link_shared {
-            cargo.env("LLVM_LINK_SHARED", "1");
+        match &*self.backend {
+            "llvm" | "emscripten" => {
+                // Build LLVM for our target. This will implicitly build the
+                // host LLVM if necessary.
+                let llvm_config = builder.ensure(native::Llvm {
+                    target,
+                    emscripten: self.backend == "emscripten",
+                });
+
+                if self.backend == "emscripten" {
+                    features.push_str(" emscripten");
+                }
+
+                let _folder = build.fold_output(|| format!("stage{}-rustc_trans", compiler.stage));
+                println!("Building stage{} codegen artifacts ({} -> {}, {})",
+                         compiler.stage, &compiler.host, target, self.backend);
+
+                // Pass down configuration from the LLVM build into the build of
+                // librustc_llvm and librustc_trans.
+                if build.is_rust_llvm(target) {
+                    cargo.env("LLVM_RUSTLLVM", "1");
+                }
+                cargo.env("LLVM_CONFIG", &llvm_config);
+                if self.backend != "emscripten" {
+                    let target_config = build.config.target_config.get(&target);
+                    if let Some(s) = target_config.and_then(|c| c.llvm_config.as_ref()) {
+                        cargo.env("CFG_LLVM_ROOT", s);
+                    }
+                }
+                // Building with a static libstdc++ is only supported on linux right now,
+                // not for MSVC or macOS
+                if build.config.llvm_static_stdcpp &&
+                   !target.contains("freebsd") &&
+                   !target.contains("windows") &&
+                   !target.contains("apple") {
+                    let file = compiler_file(build,
+                                             build.cxx(target).unwrap(),
+                                             target,
+                                             "libstdc++.a");
+                    cargo.env("LLVM_STATIC_STDCPP", file);
+                }
+                if build.config.llvm_link_shared {
+                    cargo.env("LLVM_LINK_SHARED", "1");
+                }
+            }
+            _ => panic!("unknown backend: {}", self.backend),
         }
 
-        run_cargo(build,
-                  &mut cargo,
-                  &librustc_trans_stamp(build, compiler, target),
-                  false);
+        let tmp_stamp = build.cargo_out(compiler, Mode::Librustc, target)
+            .join(".tmp.stamp");
+        let files = run_cargo(build,
+                              cargo.arg("--features").arg(features),
+                              &tmp_stamp,
+                              false);
+        let mut files = files.into_iter()
+            .filter(|f| {
+                let filename = f.file_name().unwrap().to_str().unwrap();
+                is_dylib(filename) && filename.contains("rustc_trans-")
+            });
+        let codegen_backend = match files.next() {
+            Some(f) => f,
+            None => panic!("no dylibs built for codegen backend?"),
+        };
+        if let Some(f) = files.next() {
+            panic!("codegen backend built two dylibs:\n{}\n{}",
+                   codegen_backend.display(),
+                   f.display());
+        }
+        let stamp = codegen_backend_stamp(build, compiler, target, self.backend);
+        let codegen_backend = codegen_backend.to_str().unwrap();
+        t!(t!(File::create(&stamp)).write_all(codegen_backend.as_bytes()));
     }
 }
 
@@ -682,33 +722,29 @@ fn copy_codegen_backends_to_sysroot(builder: &Builder,
     // not linked into the main compiler by default but is rather dynamically
     // selected at runtime for inclusion.
     //
-    // Here we're looking for the output dylib of the `RustcTrans` step and
+    // Here we're looking for the output dylib of the `CodegenBackend` step and
     // we're copying that into the `codegen-backends` folder.
     let libdir = builder.sysroot_libdir(target_compiler, target);
     let dst = libdir.join("codegen-backends");
     t!(fs::create_dir_all(&dst));
-    let stamp = librustc_trans_stamp(build, compiler, target);
 
-    let mut copied = None;
-    for file in read_stamp_file(&stamp) {
-        let filename = match file.file_name().and_then(|s| s.to_str()) {
-            Some(s) => s,
-            None => continue,
+    for backend in builder.config.rust_codegen_backends.iter() {
+        let stamp = codegen_backend_stamp(build, compiler, target, *backend);
+        let mut dylib = String::new();
+        t!(t!(File::open(&stamp)).read_to_string(&mut dylib));
+        let file = Path::new(&dylib);
+        let filename = file.file_name().unwrap().to_str().unwrap();
+        // change `librustc_trans-xxxxxx.so` to `librustc_trans-llvm.so`
+        let target_filename = {
+            let dash = filename.find("-").unwrap();
+            let dot = filename.find(".").unwrap();
+            format!("{}-{}{}",
+                    &filename[..dash],
+                    backend,
+                    &filename[dot..])
         };
-        if !is_dylib(filename) || !filename.contains("rustc_trans-") {
-            continue
-        }
-        match copied {
-            None => copied = Some(file.clone()),
-            Some(ref s) => {
-                panic!("copied two codegen backends:\n{}\n{}",
-                       s.display(),
-                       file.display());
-            }
-        }
-        copy(&file, &dst.join(filename));
+        copy(&file, &dst.join(target_filename));
     }
-    assert!(copied.is_some(), "failed to find a codegen backend to copy");
 }
 
 /// Cargo's output path for the standard library in a given stage, compiled
@@ -729,10 +765,12 @@ pub fn librustc_stamp(build: &Build, compiler: Compiler, target: Interned<String
     build.cargo_out(compiler, Mode::Librustc, target).join(".librustc.stamp")
 }
 
-pub fn librustc_trans_stamp(build: &Build,
-                            compiler: Compiler,
-                            target: Interned<String>) -> PathBuf {
-    build.cargo_out(compiler, Mode::Librustc, target).join(".librustc_trans.stamp")
+fn codegen_backend_stamp(build: &Build,
+                         compiler: Compiler,
+                         target: Interned<String>,
+                         backend: Interned<String>) -> PathBuf {
+    build.cargo_out(compiler, Mode::Librustc, target)
+        .join(format!(".librustc_trans-{}.stamp", backend))
 }
 
 fn compiler_file(build: &Build,
@@ -849,10 +887,13 @@ impl Step for Assemble {
                 compiler: build_compiler,
                 target: target_compiler.host,
             });
-            builder.ensure(RustcTrans {
-                compiler: build_compiler,
-                target: target_compiler.host,
-            });
+            for &backend in build.config.rust_codegen_backends.iter() {
+                builder.ensure(CodegenBackend {
+                    compiler: build_compiler,
+                    target: target_compiler.host,
+                    backend,
+                });
+            }
         }
 
         let stage = target_compiler.stage;
@@ -922,7 +963,9 @@ fn stderr_isatty() -> bool {
     }
 }
 
-pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: bool) {
+pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: bool)
+    -> Vec<PathBuf>
+{
     // Instruct Cargo to give us json messages on stdout, critically leaving
     // stderr as piped so we can get those pretty colors.
     cargo.arg("--message-format").arg("json")
@@ -1066,8 +1109,8 @@ pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: boo
     let mut new_contents = Vec::new();
     let mut max = None;
     let mut max_path = None;
-    for dep in deps {
-        let mtime = mtime(&dep);
+    for dep in deps.iter() {
+        let mtime = mtime(dep);
         if Some(mtime) > max {
             max = Some(mtime);
             max_path = Some(dep.clone());
@@ -1080,12 +1123,13 @@ pub fn run_cargo(build: &Build, cargo: &mut Command, stamp: &Path, is_check: boo
     if stamp_contents == new_contents && max <= stamp_mtime {
         build.verbose(&format!("not updating {:?}; contents equal and {} <= {}",
                 stamp, max, stamp_mtime));
-        return
+        return deps
     }
     if max > stamp_mtime {
         build.verbose(&format!("updating {:?} as {:?} changed", stamp, max_path));
     } else {
         build.verbose(&format!("updating {:?} as deps changed", stamp));
     }
     t!(t!(File::create(stamp)).write_all(&new_contents));
+    deps
 }