No nightly builds since 2023-10-09

[richardlau]

Reported in Slack.

https://nodejs.org/download/nightly/index.json
There wasn't a nightly build uploaded since 2023-10-09, any idea why that is?

[richardlau]

Manually ran the promotion script and it is erroring as there's a build more than 7 days old that cannot be promoted -- we change the permissions on the directories in dist after seven days to prevent the builds being changed/overwritten.

build/ansible/www-standalone/resources/scripts/dist-perms

Lines 1 to 10 in a8ab18b

	#!/bin/sh
	find /home/dist/nodejs/ \
	-mindepth 2 \
	\( -type f -o -type d \) \
	-regextype posix-extended \
	! -regex '/home/dist/nodejs/[a-z0-9-]*/index.(tab|json)' \
	-user dist \
	-mtime +7 \
	-exec chown root.root '{}' \;

$ /home/staging/tools/promote/promote_nightly.sh nodejs
Promoting v21.0.0-nightly2023093051f4ff2450/node-v21.0.0-nightly2023093051f4ff2450-linux-armv7l.tar.gz...
cp: cannot create regular file '/home/dist/nodejs/nightly/v21.0.0-nightly2023093051f4ff2450/node-v21.0.0-nightly2023093051f4ff2450-linux-armv7l.tar.gz': Permission denied
$

$ ls -alrt /home/dist/nodejs/nightly/
...
drwxr-xr-x   6 root root   4096 Sep 29 07:30 v21.0.0-nightly202309295570c29780
drwxr-xr-x   6 root root   4096 Sep 30 07:00 v21.0.0-nightly2023093051f4ff2450
drwxr-xr-x   6 root root   4096 Oct  1 13:51 v21.0.0-nightly20231001092fb9f541
drwxr-xr-x   6 root root   4096 Oct  2 07:00 v21.0.0-nightly2023100285c09f178c
drwxr-xr-x   5 root root   4096 Oct  3 08:00 v21.0.0-nightly202310039c683204db
drwxr-xr-x   5 root root   4096 Oct  5 07:00 v21.0.0-nightly202310051d220b55ac
drwxr-xr-x   4 root root   4096 Oct  6 07:30 v21.0.0-nightly20231006f73650ea52
drwxr-xr-x   6 root root   4096 Oct  6 11:00 v21.0.0-nightly202310041a839f388e
drwxr-xr-x   6 root root   4096 Oct  7 07:00 v21.0.0-nightly202310076b599a3f4e
drwxr-xr-x   6 dist dist   4096 Oct  8 08:00 v21.0.0-nightly20231008fce8fbadcd
drwxr-xr-x 330 dist dist 180224 Oct  9 06:30 .
drwxr-xr-x   5 dist dist   4096 Oct  9 07:30 v21.0.0-nightly20231009387e2929fe
-rw-rw-r--   1 dist dist 103057 Oct  9 07:30 index.tab
-rw-rw-r--   1 dist dist 152727 Oct  9 07:30 index.json
$

[richardlau]

First failing directory was 20230930 which is 17 days ago. This command shows the directories since then (-mtime -17) owned by root:

root@infra-digitalocean-ubuntu1604-x64-1:/home/dist# find /home/dist/nodejs/nightly/ -maxdepth 1 -mindepth 1 -type d -mtime -17 -user root | sort -n
/home/dist/nodejs/nightly/v21.0.0-nightly2023093051f4ff2450
/home/dist/nodejs/nightly/v21.0.0-nightly20231001092fb9f541
/home/dist/nodejs/nightly/v21.0.0-nightly2023100285c09f178c
/home/dist/nodejs/nightly/v21.0.0-nightly202310039c683204db
/home/dist/nodejs/nightly/v21.0.0-nightly202310041a839f388e
/home/dist/nodejs/nightly/v21.0.0-nightly202310051d220b55ac
/home/dist/nodejs/nightly/v21.0.0-nightly20231006f73650ea52
/home/dist/nodejs/nightly/v21.0.0-nightly202310076b599a3f4e
root@infra-digitalocean-ubuntu1604-x64-1:/home/dist#

Change ownership to allow dist to write to the directories (note removal of -maxdepth to change the subdirectories as well):

find /home/dist/nodejs/nightly/ -mindepth 1 -type d -mtime -17 -user root -exec chown -R dist.dist '{}' \;

The same command as earlier now shows no directories in that time period owned by root:

root@infra-digitalocean-ubuntu1604-x64-1:/home/dist# find /home/dist/nodejs/nightly/ -maxdepth 1 -mindepth 1 -type d -mtime -17 -user root | sort -n
root@infra-digitalocean-ubuntu1604-x64-1:/home/dist#

This allows the promotion to continue:

dist@infra-digitalocean-ubuntu1604-x64-1:/home/staging$ /home/staging/tools/promote/promote_nightly.sh nodejs
Promoting v21.0.0-nightly2023093051f4ff2450/node-v21.0.0-nightly2023093051f4ff2450-linux-armv7l.tar.gz...
Promoting v21.0.0-nightly2023093051f4ff2450/node-v21.0.0-nightly2023093051f4ff2450-linux-armv7l.tar.xz...
Promoting v21.0.0-nightly20231001092fb9f541/node-v21.0.0-nightly20231001092fb9f541-linux-armv7l.tar.gz...
Promoting v21.0.0-nightly20231001092fb9f541/node-v21.0.0-nightly20231001092fb9f541-linux-armv7l.tar.xz...
/home/dist/nodejs/nightly/v21.0.0-nightly20231001092fb9f541/SHASUMS256.txt
...

(promotion is still going)

[MoLow]

@richardlau thanks for handling this. do you know if the original issue is the same as #3508 (comment)?

[targos]

@MoLow No, the original issue is a job that blocked the queue for 10 days: https://ci-release.nodejs.org/job/iojs+release/9667/
After I aborted it, the next (pending) jobs went on and the promotion script couldn't promote them because the target directories were already made read-only.

[richardlau]

This is now fixed. We have builds up to v21.0.0-nightly20231016f09a50c39d.