BUG: make build_script_run write to BUILD_OUT_DIR instead of OUT_DIR

[qknight]

Problem

build.rs files pose a special problem for nix based systems during build:

• build.rs in the build_script build stage writes to OUT_DIR
• the parent crate, with the same name, also writes to OUT_DIR

Proposed solution

• build.rs must write to BUILD_OUT_DIR instead of OUT_DIR
• in legacy cargo we would initialize BUILD_OUT_DIR=$OUT_DIR
• Documentation must be updated

This feature request would then be in alignment with the unstable feature multiple-build-scripts (already in nightly).

If there is agreement that this would be ok from the cargo team, I'd create a PR for this.

Example: coreutils patch

https://github.com/uutils/coreutils/blob/5fc9f8e7e0e47c9d7a6e29a63f5c6bbd3e5cf558/build.rs#L50

In nix, the OUT_DIR of the last stage, coreutils.rs, which wants to use uutils_map.rs will not find it in OUT_DIR but rather in the current directory.

diff --git a/src/bin/coreutils.rs b/src/bin/coreutils.rs
index 6a9141936..59b9a37e5 100644
--- a/src/bin/coreutils.rs
+++ b/src/bin/coreutils.rs
@@ -12,7 +12,7 @@ use std::process;
 
 const VERSION: &str = env!("CARGO_PKG_VERSION");
 
-include!(concat!(env!("OUT_DIR"), "/uutils_map.rs"));
+include!(concat!(env!("BUILD_OUT_DIR"), "/uutils_map.rs"));

/home/nixos/tests/coreutils/target/debug/nix/derivations/coreutils-0.5.0-a1a4167b4152c2f4.nix
+    BUILD_OUT_DIR="${coreutils-0_5_0-script_build_run-7d8760345f435e2a}";

/home/nixos/tests/coreutils/target/debug/nix/derivations/coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix
+    BUILD_OUT_DIR="${coreutils-0_5_0-script_build_run-7d8760345f435e2a}";

Explanation on what happens

Coreutils crate example:

The main cause is that cargo creates 4 units to compile:

1. coreutils-0.5.0-script_build-f1108c19c7d0dd7a.nix
2. coreutils-0.5.0-script_build_run-7d8760345f435e2a.nix
3. coreutils-0.5.0-a1a4167b4152c2f4.nix
4. coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix

In the legacy backend all 4 build steps share the same working directory. In contrast the nix-backend: we have 4 different OUT_DIR folders, one per build, so (2.) coreutils-0.5.0-script_build_run-7d8760345f435e2a.nix writes to /nix/store/path2 (OUT_DIR) and (3.) coreutils-0.5.0-a1a4167b4152c2f4.nix assumes files in OUT_DIR which is now /nix/store/path3.
Step (2.), in this coreutils example, the generated uutils_map.rs is used from (4.) coreutils-0.5.0-bin-f0d6fd778fb8e17e.nix which has OUT_DIR set to /nix/store/path4.

This does not even cover the experimental feature of multiple-build-scripts but it comes with a nice feature:

Accessing Output Directories: Output directory of each build script can be accessed by using _OUT_DIR where the is the file-stem of the build script, exactly as-is. For example, bar_OUT_DIR for script at foo/bar.rs. (Only set during compilation, can be accessed via env! macro)

https://doc.rust-lang.org/cargo/reference/unstable.html#multiple-build-scripts

alternative nix implementations to cargo

for details visit: https://github.com/qknight/build_rs_example

cargo2nix, crane, naersk:

the outsource this problem by calling cargo build

crate2nix

only crate2nix builds build.rs into build_script_build but runs it inside the same mkDerivation (in the configurePhase) and later uses it in the buildPhase

• I don't quite understand how the generated files, like the foobar.rs in my example, ends up at the right place.
• I didn't check either if there is some cleanup going on, if it all ends up in OUT_DIR

cargo (nix-backend)

in this implementation i currently, exactly like cargo does, create 3 seperate calls to rustc:

1. build.rs is compiled into build_script_build
2. build_script_build is executed
3. crate is built but using the results from build_script_build run

each is its own mkDerivation which is in nix term for a derivation with its own store path and running in a sandbox.

[weihanglo]

build.rs files pose a special problem for nix based systems during build:

• build.rs in the build_script build stage writes to OUT_DIR
• the parent crate, with the same name, also writes to OUT_DIR

Is it a bug or a feature request? If it is a bug, could you provide a minimal reproducible example in plain Cargo? If it is a bug how Nix uses Cargo, could you also provide a minimal reproducible in nix?

[qknight]

@weihanglo i'll provide a nix example with cargo2nix in a short while, thanks for your help!

[qknight]

@weihanglo as you requested, here are the results for cargo2nix, crane, naersk, crate2nix:

for details visit: https://github.com/qknight/build_rs_example?tab=readme-ov-file#results

i've updated the ticket text but i'm not sure if it is a BUG or a FEATURE as it could be both.