Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files #355053

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files #355053

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1b8744e
yarn-berry_3: init at 3.8.6
gador Nov 10, 2024
6eac70d
yarn-config-hook: use $TMP instead of /tmp
gador Nov 20, 2024
aef1535
fetch-yarn-deps: add yarn-berry support
gador Nov 10, 2024
5e2366d
pgadmin4: use new yarnBerry3ConfigHook
gador Nov 10, 2024
2ff6888
yarn-berry: add gador as maintainer
gador Nov 12, 2024
e9079ea
yarn-berry[_3|_4]: move to pkgs/by-name
gador Nov 14, 2024
bad8ab6
doc/javascript: document yarn-berry for fetchYarnDeps
gador Nov 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions doc/languages-frameworks/javascript.section.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -594,6 +594,24 @@ To install the package `yarnInstallHook` uses both `npm` and `yarn` to cleanup p

- `yarnKeepDevDeps`: Disables the removal of devDependencies from `node_modules` before installation.

#### Use with `yarn-berry` and newer lockfiles {#javascript-yarnberry}

The above works well with `yarn` version 1. However, `yarn-berry` introduced a new version of lockfiles, which will not work with classic `yarn`. To still be able to use `fetchYarnDeps` with `yarn-berry`, you will need to specify the `yarnVersion` like so:

```nix
yarnOfflineCache = fetchYarnDeps {
yarnLock = finalAttrs.src + "/yarn.lock";
yarnVersion = 3;
hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
};
```

`yarnVersion = 3` is for `yarn-berry_3` lockfiles and `yarnVersion = 4` will use `yarn-berry_4` to generate `node_modules`.

The hook to be used with `yarn-berry` is called `yarnBerry3ConfigHook` for version 3 lockfiles and `yarnBerry4ConfigHook` for version 4 lockfiles.

There is currently neither a `yarnBuildHook` nor a `yarnInstallHook` for `yarn-berry`, so you will need to implement the buildPhase and/or installPhase yourself. You cannot mix a `yarn-berry` cache with the v1 `yarnBuildHook` or `yarnInstallHook`, since the version of `yarn` as well as the lockfiles differ.

### yarn2nix {#javascript-yarn2nix}

WARNING: The `yarn2nix` functions have been deprecated in favor of the new `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook`. Documentation for them still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
Expand Down
3 changes: 3 additions & 0 deletions doc/redirects.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3034,6 +3034,9 @@
"javascript-yarninstallhook": [
"index.html#javascript-yarninstallhook"
],
"javascript-yarnberry": [
"index.html#javascript-yarnberry"
],
"javascript-yarn2nix": [
"index.html#javascript-yarn2nix"
],
Expand Down
4 changes: 4 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Please be _very_ careful when changing the content of `yarn-berry-supported-archs.json`.
This file is used to instruct `yarn-berry` which packages to fetch and install into our
offline cache. If this file is changed, _ALL_ cached offline contents for `yarn-berry`
are invalidated and will need their hashes to be updated!
89 changes: 83 additions & 6 deletions pkgs/build-support/node/fetch-yarn-deps/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
prefetch-yarn-deps,
fixup-yarn-lock,
yarn,
yarn-berry_3,
yarn-berry_4,
makeSetupHook,
cacert,
callPackage,
Expand Down Expand Up @@ -96,6 +98,9 @@ in
};
};

# TODO: Move from `yarnLock` environment variable to the actual `src` argument for
# `yarn` and `yarn-berry`. This should work for both versions in place, because both
# versions will look for a yarn.lock file in the current working directory (`src`)
fetchYarnDeps =
let
f =
Expand All @@ -104,6 +109,7 @@ in
src ? null,
hash ? "",
sha256 ? "",
yarnVersion ? 1,
...
}@args:
let
Expand Down Expand Up @@ -132,22 +138,71 @@ in
dontInstall = true;

nativeBuildInputs = [
prefetch-yarn-deps
(
{
"1" = prefetch-yarn-deps;
"3" = yarn-berry_3;
"4" = yarn-berry_4;
gador marked this conversation as resolved.
Show resolved Hide resolved
}
.${builtins.toString yarnVersion}
)
cacert
];
GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt";
NODE_EXTRA_CA_CERTS = "${cacert}/etc/ssl/certs/ca-bundle.crt";

buildPhase = ''
runHook preBuild
configurePhase = lib.optionalString (yarnVersion > 1) ''
runHook preConfigure

export HOME="$NIX_BUILD_TOP"
export YARN_ENABLE_TELEMETRY=0
yarnLock=''${yarnLock:=$PWD/yarn.lock}
mkdir -p $out
(cd $out; prefetch-yarn-deps --verbose --builder $yarnLock)
DIR=$(dirname $yarnLock)
if [ $DIR = "${builtins.storeDir}" ]; then
echo "Only a yarn.lock file has been passed (e.g. yarnLock = ./yarn.lock;)"
echo "Please change this so a directory can be deduced (e.g. yarnLock = finalAttrs.src + "/yarn.lock";)"
exit 1
fi
if [ ! -f $DIR/package.json ]; then
echo "No package.json has been found in the same directory as the yarn.lock file"
exit 1
fi
if ! grep -q "__metadata:" $DIR/yarn.lock; then
echo "This yarn.lock file appears to be a version 1 yarn.lock file"
echo "Please use fetchYarnDeps with yarnVersion = 1"
exit 1
fi
cp $DIR/yarn.lock $DIR/package.json $HOME/
chmod +w $HOME/yarn.lock $HOME/package.json
yarn config set enableGlobalCache false
yarn config set cacheFolder $out
yarn config set supportedArchitectures --json "$(cat ${./yarn-berry-supported-archs.json})"

runHook postBuild
runHook postConfigure
'';

buildPhase =
''
runHook preBuild

mkdir -p $out
''
+ lib.optionalString (yarnVersion > 1) ''
doronbehar marked this conversation as resolved.
Show resolved Hide resolved
yarn install --immutable --mode skip-build
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will break the FODs almost instantly if Yarn changes anything in how they pull deps, and also is probably not reproducible between platforms when platform-specific dependencies are used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point. This is partly why there are two yarn versions.

Also this could be said for all external package managers, not just yarn-berry.

I'm uncertain about your point about platform specific dependencies. I'll have to look into it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Platform specific dependencies should be made reproducible by explicitly setting supportedArchitectures. Your point about Yarn breaking how they fetch deps may be valid depending on the upstream stability guarantees. Explicitly picking the version of yarn somewhat mitigates that issue, but it's no guarantee yes. This is the same path taken in pnpm.fetchDeps #290715

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winterqt what is your reply on the arguments laid out above? Your "request for changes" is blocking the merge of the PR, and this is the only review comment left really open.

''
+ lib.optionalString (yarnVersion == 1) ''
yarnLock=''${yarnLock:=$PWD/yarn.lock}
if grep -q "__metadata:" $yarnLock; then
echo "This yarn.lock file does not appear to be a version 1 yarn.lock file"
echo "Please use fetchYarnDeps with eiher yarnVersion = 3 or yarnVersion = 4"
exit 1
fi
(cd $out; prefetch-yarn-deps --verbose --builder $yarnLock)
''
+ ''
runHook postBuild
'';

outputHashMode = "recursive";
}
// hash_
Expand All @@ -174,6 +229,28 @@ in
};
} ./yarn-config-hook.sh;

yarnBerry3ConfigHook = makeSetupHook {
Copy link
Member

@szlend szlend Nov 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I like how in pnpm they put all their hooks under the package itself (e.g. pnpm_9.configHook).

This way you can just:

let
  yarn = yarn-berry_3;
in
{
  yarnOfflineCache = yarn.fetchDeps { ... };
  nativeBuildInputs = [ yarn yarn.configHook ];
}

Just an observation, not necessarily a blocker.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea. However this would mean a major refactoring of how the yarn system works. Right now, you can just use fetchYarnDeps. With your suggestion, it will mean a refactor of all packages using yarn to now use yarn.fetchYarnDeps.
This can of course be done, but IMHO would be too much for this PR. Lets keep it in mind and maybe do this refactor in a separate PR or tracking issue. This is what @doronbehar suggested for the long-term ecosystem for the src argument (#355053 (comment))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With your suggestion, it will mean a refactor of all packages using yarn to now use yarn.fetchYarnDeps.

Small correction: yarn.fetchDeps :).

This can of course be done, but IMHO would be too much for this PR. Lets keep it in mind and maybe do this refactor in a separate PR or tracking issue. This is what @doronbehar suggested for the long-term ecosystem for the src argument (#355053 (comment))

Furthermore, this suggestion can ease the refactoring: We could make yarn.fetchDeps get the new signature with src, and simply make fetchYarnDeps throw a message like fetchYarnDeps is deprecated in favor of yarn.fetchDeps which accepts an src argument instead of yarnLock argument. This would allow us to not make the effort of making the same function compatible with both signatures.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean we could easily just make these as aliases. But agree that this couls be done in another PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, lets target that for another PR

name = "yarn-config-hook";
propagatedBuildInputs = [ yarn-berry_3 ];
substitutions = {
yarn = lib.getExe yarn-berry_3;
};
meta = {
description = "Install nodejs dependencies from an offline yarn berry cache (version 3)";
};
} ./yarn-berry-config-hook.sh;

yarnBerry4ConfigHook = makeSetupHook {
name = "yarn-config-hook";
propagatedBuildInputs = [ yarn-berry_4 ];
substitutions = {
yarn = lib.getExe yarn-berry_4;
};
meta = {
description = "Install nodejs dependencies from an offline yarn berry cache (version 4)";
};
} ./yarn-berry-config-hook.sh;

yarnBuildHook = makeSetupHook {
name = "yarn-build-hook";
meta = {
Expand Down
7 changes: 7 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/tests/berry_3/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"name": "yarn-testing",
"packageManager": "[email protected]",
"dependencies": {
"lit-html": "^3.2.1"
}
}
30 changes: 30 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/tests/berry_3/yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# This file is generated by running "yarn install" inside your project.
# Manual changes might be lost - proceed with caution!

__metadata:
version: 6
cacheKey: 10c0

"@types/trusted-types@npm:^2.0.2":
version: 2.0.7
resolution: "@types/trusted-types@npm:2.0.7"
checksum: 4c4855f10de7c6c135e0d32ce462419d8abbbc33713b31d294596c0cc34ae1fa6112a2f9da729c8f7a20707782b0d69da3b1f8df6645b0366d08825ca1522e0c
languageName: node
linkType: hard

"lit-html@npm:^3.2.1":
version: 3.2.1
resolution: "lit-html@npm:3.2.1"
dependencies:
"@types/trusted-types": "npm:^2.0.2"
checksum: 31c02df2148bf9a73545570cbe57aae01c4de1d9b44060b6ff13641837d38e39e6b1abcf92e13882cc84f5fee37605ed79602b91ad479728549014462808118e
languageName: node
linkType: hard

"yarn-testing@workspace:.":
version: 0.0.0-use.local
resolution: "yarn-testing@workspace:."
dependencies:
lit-html: ^3.2.1
languageName: unknown
linkType: soft
7 changes: 7 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/tests/berry_4/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"name": "yarn-testing",
"packageManager": "[email protected]",
"dependencies": {
"lit-html": "^3.2.1"
}
}
30 changes: 30 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/tests/berry_4/yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# This file is generated by running "yarn install" inside your project.
# Manual changes might be lost - proceed with caution!

__metadata:
version: 8
cacheKey: 10c0

"@types/trusted-types@npm:^2.0.2":
version: 2.0.7
resolution: "@types/trusted-types@npm:2.0.7"
checksum: 10c0/4c4855f10de7c6c135e0d32ce462419d8abbbc33713b31d294596c0cc34ae1fa6112a2f9da729c8f7a20707782b0d69da3b1f8df6645b0366d08825ca1522e0c
languageName: node
linkType: hard

"lit-html@npm:^3.2.1":
version: 3.2.1
resolution: "lit-html@npm:3.2.1"
dependencies:
"@types/trusted-types": "npm:^2.0.2"
checksum: 10c0/31c02df2148bf9a73545570cbe57aae01c4de1d9b44060b6ff13641837d38e39e6b1abcf92e13882cc84f5fee37605ed79602b91ad479728549014462808118e
languageName: node
linkType: hard

"yarn-testing@workspace:.":
version: 0.0.0-use.local
resolution: "yarn-testing@workspace:."
dependencies:
lit-html: "npm:^3.2.1"
languageName: unknown
linkType: soft
10 changes: 10 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/tests/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,14 @@
yarnLock = ./giturl.lock;
sha256 = "sha256-VPnyqN6lePQZGXwR7VhbFnP7/0/LB621RZwT1F+KzVQ=";
};
berry_3 = testers.invalidateFetcherByDrvHash fetchYarnDeps {
yarnLock = "${./berry_3}/yarn.lock";
yarnVersion = 3;
sha256 = "sha256-eiVOpRHuAS7zb6JpCzS3TIHbqQU+wCrMItzZGrE6Fbo=";
};
berry_4 = testers.invalidateFetcherByDrvHash fetchYarnDeps {
yarnLock = "${./berry_4}/yarn.lock";
yarnVersion = 4;
sha256 = "sha256-Eq37qKn5P7rkufJBhME0OG/3BsWlCsFYVnWHP5uVhAQ=";
};
}
31 changes: 31 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/yarn-berry-config-hook.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
yarnBerryConfigHook(){
echo "Executing yarnBerryConfigHook"

# Use a constant HOME directory, don't use $NIX_BUILD_TOP because of
# https://github.com/NixOS/nixpkgs/issues/189691
mkdir -p "$TMP/home"
export HOME="$TMP/home"
if [[ -n "$yarnOfflineCache" ]]; then
offlineCache="$yarnOfflineCache"
fi
if [[ -z "$offlineCache" ]]; then
echo yarnConfigHook: No yarnOfflineCache or offlineCache were defined\! >&2
exit 2
fi

export YARN_ENABLE_TELEMETRY=0
yarn config set enableGlobalCache false
yarn config set enableScripts false
yarn config set cacheFolder "$offlineCache"

yarn install --immutable --immutable-cache

# TODO: Check if this is really needed
patchShebangs node_modules

echo "finished yarnConfigHook"
}

if [[ -z "${dontYarnInstallDeps-}" ]]; then
postConfigureHooks+=(yarnBerryConfigHook)
fi
25 changes: 25 additions & 0 deletions pkgs/build-support/node/fetch-yarn-deps/yarn-berry-supported-archs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a comment somewhere so that people avoid touching this file/are very careful when doing it. Ideally we'd be able to use JSON with comments...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file means that we can never expand our platform support without mass FOD breaks across the tree? That seems very bad. There are already platforms you can boot NixOS on that would be ruled out by this file. At the very least we should try to list every value already used in npm here so that it’s relatively safe to expand in future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since JSON and comments don't agree (and we also read the file directly), I added a Readme.md to the folder

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file means that we can never expand our platform support without mass FOD breaks across the tree?

yes

That seems very bad. There are already platforms you can boot NixOS on that would be ruled out by this file. At the very least we should try to list every value already used in npm here so that it’s relatively safe to expand in future?

We did add all the relevant platforms, cpu's and libc's (see https://yarnpkg.com/configuration/yarnrc#supportedArchitectures) that are available. The only one missing is win32 and I don't think we need to add this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it’s everything Yarn supports then I guess it’s not so bad – assuming we will be able to add additional platforms if Yarn does support more in future without invalidating every FOD?

I do think we should include Windows. You can use Nixpkgs to cross‐build a lot of stuff for Windows today, and people are working on native bring‐up. What’s the harm?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.
On a sidenode: I did neither have to change the test hashes, nor the hash for the (seriously complex and long) yarn pgadmin offlineCache. (and yes, I obviously invalidated them, re-downloaded the packages and compared the hashes)
So maybe this won't be such a big deal after all

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the idea is that packages can’t include any platforms other than the ones Yarn supports, so no existing FOD should change as long as we keep up with Yarn?

Yes, I think your right. My guess is that adding another platform or compiler will a huge undertaking anyway, so maybe they will publish yet another big version upgrade, which would make it easier for us (e.g. yarn-berry_5)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I think I was correct here:

Or can packages on npm reference other platforms and Yarn just can’t support fetching them?

The npm docs imply that the fields are quite freeform; e.g. the cpu example uses mips as an example. That means that this is just a limitation of Yarn’s configuration format rather than a constraint on the actual packages in existence, and there are packages that could change in the future if we add new platforms like RISC‐V.

The correct solution here would be to fetch everything independent of platform – theoretically Yarn could add all values here to opt in to fetching everything, but writing our own deterministic fetcher would make it easy and also obviate the other reproducibility concerns. If neither of those is an option, perhaps we can find a way to check that nothing in the dependency tree references any platform we haven’t included in this file, and otherwise bail out? If we can’t do even that, then it’d be good if someone could write a script to scan the npm repository and collect every value that is used in practice so that we can include them all from the start and reduce the scope of potential changes.

Because we’re adding fundamental packaging infrastructure here that is part of our public interface and will surely be relied upon by out‐of‐tree packages, and changing FOD hashes are very difficult to spot and debug, I don’t think we’ll have the option of just breaking it in the future, unless we can show that the backwards incompatibility would be entirely theoretical in the existing public corpus.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, I think I was correct here:

Or can packages on npm reference other platforms and Yarn just can’t support fetching them?

The npm docs imply that the fields are quite freeform; e.g. the cpu example uses mips as an example. That means that this is just a limitation of Yarn’s configuration format rather than a constraint on the actual packages in existence, and there are packages that could change in the future if we add new platforms like RISC‐V.

Actually, it is not a limitation of the configuration, as it can be adapted to include everything node supports. https://yarnpkg.com/configuration/yarnrc#supportedArchitectures says:

List of CPU architectures to cover.
See https://nodejs.org/docs/latest/api/process.html#processarch for the architectures supported by Node.js

which states:

The operating system CPU architecture for which the Node.js binary was compiled. Possible values are: 'arm', 'arm64', 'ia32', 'loong64', 'mips', 'mipsel', 'ppc', 'ppc64', 'riscv64', 's390', 's390x', and 'x64'.

So we can include all of those for our supported-archs. Adding this, btw, didn't change the cache for pgadmin, either.
I looked into how yarn-berry does its comparisons (since these values are for the local .yarnrc.toml config file) and it compares it to upstreams package.json file. This file has the same fields (https://yarnpkg.com/configuration/manifest#cpu) and, more importantly, only the libc and os fields we already have.

The correct solution here would be to fetch everything independent of platform – theoretically Yarn could add all values here to opt in to fetching everything, but writing our own deterministic fetcher would make it easy and also obviate the other reproducibility concerns.

Yes, writing our own fetcher could be a solution. This will be quite a task to parse the yarn.lock file and generate the cache. I really feel uncomftable reinventing the wheel, though. We do have a package manager, which already parses this file and also controlls the specs about our architectures. I don't want to write yet another solution of yarn2nix

If neither of those is an option, perhaps we can find a way to check that nothing in the dependency tree references any platform we haven’t included in this file, and otherwise bail out?

There seems to be an option (yarn info --all --manifest --json) which prints all the manifests, but not one had a cpu, libc or os field. There is another option (yarn npm info _packageName_ --json) which prints the upstream manifest and from the quick peek I took, didn't include any of those, either.

If we can’t do even that, then it’d be good if someone could write a script to scan the npm repository and collect every value that is used in practice so that we can include them all from the start and reduce the scope of potential changes.
Because we’re adding fundamental packaging infrastructure here that is part of our public interface and will surely be relied upon by out‐of‐tree packages, and changing FOD hashes are very difficult to spot and debug, I don’t think we’ll have the option of just breaking it in the future, unless we can show that the backwards incompatibility would be entirely theoretical in the existing public corpus.

Totally agree. This is why we added versions here. Not only because upstream has two supported versions (3 and 4), but also because their yarn.lock files differ.
If there is a situation where either the file format changes, or the supported architectures, I would suggest adding another version (e.g. yarn-berry_5 or yarn-berry_39). This will keep all the hashes intakt, be reproducible and still be able to adapt to future changes.

Copy link
Member

@szlend szlend Dec 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The npm docs imply that the fields are quite freeform; e.g. the cpu example uses mips as an example. That means that this is just a limitation of Yarn’s configuration format rather than a constraint on the actual packages in existence, and there are packages that could change in the future if we add new platforms like RISC‐V.

The documentation says these values are based on Node process.{arch,cpu} which has a known documented set of values - namely platforms supported by nodejs. I wouldn't exactly call it freeform.

I don't really have a better idea on how to handle adding new platforms other than invalidating the FOD (by changing name) and fixing up all the failed builds, or introducing a new builder version like @gador suggested above.

I wonder how pnpm managed to avoid this issue. I don't see it handling supportedArchitectures at all.

"os": [
"darwin",
"linux",
"win32"
],
"cpu": [
"arm",
"arm64",
"ia32",
"x64",
"loong64",
"mips",
"mipsel",
"ppc",
"ppc64",
"riscv64",
"s390",
"s390x"
],
"libc": [
"glibc",
"musl"
]
}
4 changes: 2 additions & 2 deletions pkgs/build-support/node/fetch-yarn-deps/yarn-config-hook.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ yarnConfigHook(){
echo "Executing yarnConfigHook"

# Use a constant HOME directory
mkdir -p /tmp/home
export HOME=/tmp/home
mkdir -p "$TMP/home"
export HOME="$TMP/home"
if [[ -n "$yarnOfflineCache" ]]; then
offlineCache="$yarnOfflineCache"
fi
Expand Down
18 changes: 16 additions & 2 deletions .../development/tools/yarn-berry/default.nix → pkgs/by-name/ya/yarn-berry/package.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,30 @@
stdenv,
testers,
yarn,
berryVersion ? 4,
}:

let
# Please be _very_ careful, when updating these versions,
# as they are used to generate FOD using fetchYarnDeps.
# At the very least run the tests prefetch-yarn-deps.tests
# and make sure the hashes do not change (and if they do, we'll
# need to add another version here for stability across nixpkgs)
version_4 = "4.5.3";
version_3 = "3.8.6";
hash_4 = "sha256-ywg+SYjFlWUMQftw1eZE5UY3nfxn6xy1NIawgmH/4vY=";
hash_3 = "sha256-DVmIY0cNmLiPi8B6VrHrXseDw9F6ApHxVn/KZxnWfpA=";
in

stdenv.mkDerivation (finalAttrs: {
pname = "yarn-berry";
version = "4.5.3";
version = if berryVersion == 4 then version_4 else version_3;

src = fetchFromGitHub {
owner = "yarnpkg";
repo = "berry";
rev = "@yarnpkg/cli/${finalAttrs.version}";
hash = "sha256-ywg+SYjFlWUMQftw1eZE5UY3nfxn6xy1NIawgmH/4vY=";
hash = if berryVersion == 4 then hash_4 else hash_3;
};

buildInputs = [
Expand Down Expand Up @@ -56,6 +69,7 @@ stdenv.mkDerivation (finalAttrs: {
ryota-ka
pyrox0
DimitarNestorov
gador
];
platforms = platforms.unix;
mainProgram = "yarn";
Expand Down
4 changes: 3 additions & 1 deletion pkgs/development/tools/yarn-berry/update.sh → pkgs/by-name/ya/yarn-berry/update.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,7 @@ EOF
)

version=$(jq -r "[.data.repository.tag.nodes[].name | select(contains(\"-\")|not)] | max_by(split(\".\") | map(tonumber))" <<< "$payload")
version3=$(jq -r "[.data.repository.tag.nodes[].name | select(contains(\"-\")|not)] | map(select(. < \"4.0.0\")) | sort | last" <<< "$payload")

update-source-version yarn-berry "$version"
update-source-version yarn-berry_4 "$version" --version-key="version_4"
update-source-version yarn-berry_3 "$version3" --version-key="version_3"
7 changes: 7 additions & 0 deletions pkgs/by-name/ya/yarn-berry_3/package.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
yarn-berry,
}:

yarn-berry.override {
berryVersion = 3;
}
Loading
Loading