build: make .tar.xz creation opt-out, fail if no xz #24551

rvagg · 2018-11-21T12:59:30Z

During a recent release, the 8.x before last I think, we discovered belatedly that the macos release machine didn't have xz installed on it. This change was discussed (somewhere, but I know @MylesBorins was involved) to help resolve such oversights in future.

Change XZ to HAS_XZ and make it 0=no and 1=yes
Introduce SKIP_XZ as a possible user-supplied value, defaulting to 0=no
Check for either HAS_XZ=1 or SKIP_XZ=1 on any release build that produces a tarball of some kind and fail if the system doesn't have xz and SKIP_XZ=1 hasn't been supplied.
Only produce a .tar.xz if both HAS_XZ=1 and SKIP_XZ=0 (i.e. we have xz and haven't explicitly opted to skip)

This would get backported to active lines and we'd have to put a SKIP_XZ=1 for AIX builds (only).

I've tested the main permutations of this and it seems to work fine. I'd appreciate some experienced Makefile eyes though, @nodejs/build @bnoordhuis @joyeecheung?

nodejs-github-bot · 2018-11-21T12:59:32Z

@rvagg build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1664/pipeline

refack

LGTM % quastions

Makefile

refack · 2018-11-21T13:28:44Z

/CC @nodejs/build-files

rvagg · 2018-11-22T09:54:01Z

OK, second commit on this PR, 94a2c9c8e5, implements the above behaviour suggested by @refack and the checks around the execution of xz is now simple again, although inverted from previous: ifeq ($(XZ), 1). That should make it easier to understand than the original logical-and within a single ifeq which is pretty opaque.

refack · 2018-11-22T17:32:09Z

Lite-CI (since it's not covered in CI anyway): https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/1678/

Reviewers, please 👍 if you are in favor of fast-tracking this PR.

richardlau · 2018-11-22T17:42:19Z

Lite-CI (since it's not covered in CI anyway): https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/1678/

Reviewers, please 👍 if you are in favor of fast-tracking this PR.

Is it worth asking @nodejs/releasers for a test release build?

refack · 2018-11-22T17:56:15Z

Is it worth asking @nodejs/releasers for a test release build?

~~https://ci-release.nodejs.org/job/iojs+release/3954/~~
https://ci-release.nodejs.org/job/iojs+release/3956 ✔️

refack · 2018-11-22T23:19:12Z

The ci-release job finished successfully, the output is at https://nodejs.org/download/nightly/v12.0.0-nightly20181122.2.94a2c9c8e5/ (no .xz for AIX)

rvagg · 2018-11-23T02:13:40Z

@refack you made a nightly with this with a weird date string: https://ci-release.nodejs.org/job/iojs+release/3956/ and now we have screwed up nightly version strings which won't parse with tools that expect a predictable format on this directory:

Please don't do that. Nightlies are strictly not for testing. I'm going to have to figure out if I can clean this one out now.

There's a bunch of good reasons we don't hand out ci-release access liberally, and the ability to make this kind of mess is one of them.

refack · 2018-11-23T02:22:03Z

Please don't do that. Nightlies are strictly not for testing. I'm going to have to figure out if I can clean this one out now.

My bad. What I meant to do was run a test build.

P.S. As a postmortem, we should add some validation code to assert our and our tools format assumptions (nodejs/build#1592)

rvagg · 2018-11-23T02:24:27Z

OK, so this should have failed on AIX but it didn't.

The config doesn't have SKIP_XZ=1 for AIX, and the output https://ci-release.nodejs.org/job/iojs+release/nodes=aix61-ppc64/3956/consoleFull doesn't mention anything about SKIP_XZ. It either hasn't invoked check-xz on there (not sure why it wouldn't) or HAS_XZ=1, which doesn't make sense because it also doesn't invoke xz..

refack · 2018-11-23T02:26:01Z

P.P.S. we also have this one - https://nodejs.org/download/nightly/v11.0.0-nightly201806050300f7c68cffa5de7f0099e3afa95093b0bbbe3c/

rvagg · 2018-11-23T02:36:45Z

Ah, so we have this in iojs+release:

if [[ "X${disttype}" != "Xrelease" ]]; then
  perl -pi -e "s/: release-only/:/g" Makefile
fi

That traces back to the addition of the REPLACEME stuff for API docs which is too strict. I tried to fix that back in #12958 but it got stalled with too much discussion, so the perl replacement has stuck and we're without the release-only checks entirely for release builds.. yay.

refack · 2018-11-23T02:40:02Z

Makefile

 XZ_COMPRESSION ?= 9e
 PKG=$(TARNAME).pkg
 MACOSOUTDIR=out/macos

+ifeq ($(SKIP_XZ), 1)
+check-xz:


Maybe it's because check-xz should be .PHONY?

Ref: nodejs#24551 Ref: nodejs#12958 Ref: nodejs#12957 Ref: nodejs#8325

rvagg · 2018-11-23T02:51:55Z

Proposed fix in #24575 so we can remove the perl release-only stripping.

rvagg · 2018-12-05T00:07:08Z

Discovered some problems, not sure why these weren't picked up with test builds in ci-release for the PR. #24841

rvagg · 2018-12-05T00:13:28Z

@nodejs/build FYI this is now in iojs+release for the ${ENV, var="OSTYPE"} = AIX61 build script:

It's building successfully now:

Results for this test going to https://nodejs.org/download/test/v12.0.0-test201812040ba0e0fd89/

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: nodejs#24551 PR-URL: nodejs#24841 Refs: nodejs#24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24575 Refs: #24551 Refs: #12958 Refs: #12957 Refs: #8325 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: nodejs#24575 Refs: nodejs#24551 Refs: nodejs#12958 Refs: nodejs#12957 Refs: nodejs#8325 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: nodejs#24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: nodejs#24551 PR-URL: nodejs#24841 Refs: nodejs#24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

5e80a9a introduced check-xz, using `[[ .. ]]` syntax, but this is a bash builtin and some platforms default to `sh` when doing `$(shell ...)` in Makefiles. Fix is to make it sh friendly. Ref: #24551 PR-URL: #24841 Refs: #24551 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Richard Lau <[email protected]>

PR-URL: #24575 Refs: #24551 Refs: #12958 Refs: #12957 Refs: #8325 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]> Reviewed-By: Richard Lau <[email protected]>

rvagg added the build Issues and PRs related to build files or the CI. label Nov 21, 2018

rvagg force-pushed the rvagg/xz-opt-out branch from d2e0367 to ba2647a Compare November 21, 2018 13:00

refack approved these changes Nov 21, 2018

View reviewed changes

Makefile Show resolved Hide resolved

Makefile Show resolved Hide resolved

refack added the aix Issues and PRs related to the AIX platform. label Nov 21, 2018

richardlau approved these changes Nov 22, 2018

View reviewed changes

richardlau mentioned this pull request Nov 22, 2018

Proposal: remove .gz downloads for some platforms nodejs/build#1584

Closed

rvagg force-pushed the rvagg/xz-opt-out branch from 14eedc0 to 94a2c9c Compare November 22, 2018 09:51

refack approved these changes Nov 22, 2018

View reviewed changes

richardlau approved these changes Nov 22, 2018

View reviewed changes

refack added fast-track PRs that do not need to wait for 48 hours to land. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Nov 22, 2018

refack mentioned this pull request Nov 23, 2018

FR: add some validation to ci-release iojs+release nodejs/build#1592

Closed

refack reviewed Nov 23, 2018

View reviewed changes

rvagg added a commit to rvagg/io.js that referenced this pull request Nov 23, 2018

build: only check REPLACEME & DEP...X for releases

6759a2e

Ref: nodejs#24551 Ref: nodejs#12958 Ref: nodejs#12957 Ref: nodejs#8325

rvagg mentioned this pull request Nov 23, 2018

build: only check REPLACEME & DEP...X for releases #24575

Closed

Trott removed author ready PRs that have at least one approval, no pending requests for changes, and a CI started. fast-track PRs that do not need to wait for 48 hours to land. labels Nov 25, 2018

BridgeAR pushed a commit that referenced this pull request Dec 5, 2018

build: make tar.xz creation opt-out, fail if no xz

4647706

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

BridgeAR mentioned this pull request Dec 5, 2018

v11.4.0 proposal #24854

Merged

4 tasks

BridgeAR pushed a commit that referenced this pull request Dec 7, 2018

build: make tar.xz creation opt-out, fail if no xz

3a24c91

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

BridgeAR pushed a commit that referenced this pull request Dec 7, 2018

build: make tar.xz creation opt-out, fail if no xz

ecb669a

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

refack pushed a commit to refack/node that referenced this pull request Jan 14, 2019

build: make tar.xz creation opt-out, fail if no xz

8a371cf

PR-URL: nodejs#24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

BethGriggs pushed a commit that referenced this pull request Feb 12, 2019

build: make tar.xz creation opt-out, fail if no xz

920cab7

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

BethGriggs mentioned this pull request Feb 12, 2019

v10.15.3 proposal #26063

Merged

BethGriggs pushed a commit that referenced this pull request Feb 20, 2019

build: make tar.xz creation opt-out, fail if no xz

560010f

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

rvagg added a commit that referenced this pull request Feb 28, 2019

build: make tar.xz creation opt-out, fail if no xz

5b4a826

PR-URL: #24551 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Richard Lau <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: make .tar.xz creation opt-out, fail if no xz #24551

build: make .tar.xz creation opt-out, fail if no xz #24551

rvagg commented Nov 21, 2018

nodejs-github-bot commented Nov 21, 2018

refack left a comment

refack commented Nov 21, 2018

rvagg commented Nov 22, 2018

refack commented Nov 22, 2018 •

edited

Loading

richardlau commented Nov 22, 2018

refack commented Nov 22, 2018 •

edited

Loading

refack commented Nov 22, 2018 •

edited

Loading

rvagg commented Nov 23, 2018 •

edited

Loading

refack commented Nov 23, 2018 •

edited

Loading

rvagg commented Nov 23, 2018 •

edited

Loading

refack commented Nov 23, 2018

rvagg commented Nov 23, 2018

refack Nov 23, 2018

rvagg commented Nov 23, 2018

rvagg commented Dec 5, 2018

rvagg commented Dec 5, 2018

build: make .tar.xz creation opt-out, fail if no xz #24551

build: make .tar.xz creation opt-out, fail if no xz #24551

Conversation

rvagg commented Nov 21, 2018

nodejs-github-bot commented Nov 21, 2018

refack left a comment

Choose a reason for hiding this comment

refack commented Nov 21, 2018

rvagg commented Nov 22, 2018

refack commented Nov 22, 2018 • edited Loading

richardlau commented Nov 22, 2018

refack commented Nov 22, 2018 • edited Loading

refack commented Nov 22, 2018 • edited Loading

rvagg commented Nov 23, 2018 • edited Loading

refack commented Nov 23, 2018 • edited Loading

rvagg commented Nov 23, 2018 • edited Loading

refack commented Nov 23, 2018

rvagg commented Nov 23, 2018

refack Nov 23, 2018

Choose a reason for hiding this comment

rvagg commented Nov 23, 2018

rvagg commented Dec 5, 2018

rvagg commented Dec 5, 2018

refack commented Nov 22, 2018 •

edited

Loading

refack commented Nov 22, 2018 •

edited

Loading

refack commented Nov 22, 2018 •

edited

Loading

rvagg commented Nov 23, 2018 •

edited

Loading

refack commented Nov 23, 2018 •

edited

Loading

rvagg commented Nov 23, 2018 •

edited

Loading