Proposal: New Release Model

[jasnell]

/cc @nodejs/releasers @nodejs/tsc

Our current Node.js release model is... complex.

• Rather than a single Semantic Version stream, we currently have four, one for the HEAD, and one each for the current Long Term Support branches. This makes multiple things fairly complicated. For instance, understanding when a new feature was actually introduced -- in the documentation, we end up with metadata saying that a feature was introduced in 15.x, 14.x, 13.x, 12.x, etc ... with the versions reflecting whatever downstream version is happens to have been backported to. This makes it extremely difficult to reason about what Node.js versions has a given feature.

• Cutting a new release is... complicated. In addition to the main branch, we have Release and Staging branches for each downlevel LTS stream (so, currently, there are 9 separate branches for HEAD, 15.x-staging, 15.x, 14.x-staging, 14.x, 12.x-staging, 12.x, 10.x-staging, 10.x, not to mention the private github organization and branches we use for security triage. Backporting changes from one to the other is complex, with significant and inconsistent lag times in when various features make their way back to older Node.js versions. Sometimes useful features never get backported, other times the simplest oddest things easily jump back multiple Node.js version levels. Backporting is made even more complex by dependencies of semver-patch and semver-minor commits on prior semver-major commits, and we've had quite a few cases over the years where documentation bugs have crept up because some doc change was backported when it shouldn't have been because the underlying feature hadn't actually been backported yet or was based on a semver-major change.

• Cutting a new release is... time consuming. Folks preparing a new release have to curate a set of commits back first to staging, then to a working proposed release branch, then to the actual release branch, then reconcile that back to both the staging branch and the HEAD branch, updating release metadata in the process.

I could keep listing issues but anyone who has gone through the process of cutting a new Node.js release already knows that the process is difficult and time consuming.

I'd like to propose a change detailed here: https://docs.google.com/presentation/d/13JkM4SLm176XE92pIiDSNtx-MMY2v7eLs9b_qzSc8DA/edit?usp=sharing

I'm sure there are going to be lots of questions and things to discuss. What I ask right at the start is that you please ask questions to clarify before commenting or offering opinions.

The short version is this:

We switch to a Canary / Current / Stable model.

Canary releases are automatic and weekly. Once per week, the current HEAD is automatically rolled into an updated Canary release. All semantic versioning happens only in the Canary stream, with semver-major and semver-minor bumps happening as necessary as things land. The Canary releases are tagged as {major}.{minor}.{patch}-{datestamp}-canary.

Current releases involve nothing more than taking an existing candidate canary tag and promoting it (with a few additional metadata bits) as a new current tag. There's no selection of commits into a release. We take the existing canary tag, update the metadata with a commit, and tag it with a new current tag {major}.{minor}.{patch}-{datestamp}-current, where the {major}.{minor}.{patch} does not change. A new current can be promoted at any time but would happen at least once per month.

Similarly, Stable releases involve nothing more than taking an existing candidate current tag and promoting it (also with additional metadata bits) as a new stable tag. There's no selection of commits into a release. We take the existing current tag, update the metadata with a commit, and tag it with a new stable tag {major}.{minor}.{patch}-{datestamp}-stable, where the {major}.{minor}.{patch} does not change. New stables can be promoted at any time but would happen at least twice per year.

Current release tags are never updated. If new features, bug fixes, security vulnerabilities occur, we land those first in the HEAD, cut a new canary tag, and promote a new current tag. There is no backport flow to current, ever.

Stable release tags only ever receive targeted critical security and bug fixes applied as patch sets that only update the build metadata (the {datestamp}). There is no backport flow for semver-minor or semver-patch changes that are not considered critical bug fixes.

One Stable release tag per year would be selected for 3 years of Long Term Support effective from the day the tag is promoted. This would be maintenance only subject to the backport rule just described. Specifically, LTS stables would never receive anything more than critical security and bug fixes.

This approach offers a number of important benefits:

• Eliminates Active LTS. The Active LTS cycle has been difficult and complex due to the backporting flow. It also increases the risk of taking on breaking changes and increases the complexity of our branch and version management issues. Eliminating Active LTS is, by itself, a net positive.

• Eliminates persistent staging and down-level release branches. Patch sets to update maintenance for critical bug and security fixes done in transient working branches with PR back to tag when ready (similar to the way GitHub's new security advisory workflow works... in fact, we'd be better positioned to use that mechanism with this new approach). Importantly, we could eliminate the multi-branch staging-release structure we have now.

• Simplifies current release process by eliminating the "backport and cherry-pick a bunch of commits into a staging branch" part of the flow.

• Eliminates the multi-stream semver model we use now. Maintenance updates will bump build metadata rather than version. All semver happens on canary stream only. Make version numbers less important for maintenance.

• Eliminates backporting of new features. This does mean that users may have to take on breaking updates to get new features but the overall maintenance burden is reduced.

• Decouples the release process from any reliance on the specific semver number (semver numbers become far less important overall).

• Allows taking on new V8 versions more rapidly, gaining new features and fixes at a faster pace. These would flow naturally through canary and into current releases, promoting into stable approximately twice per year.

• It preserves existing Long Term Support timelines. Just like now, there would be one guaranteed stable release per year that would receive three years of guaranteed support.

• It simplifies changelog management and reasoning about when features are added. Since features are never backported, they are only ever added to a single specific Node.js version.

• It does mean that semver-major bumps could happen as often as every week but this approach eliminates the focus on specific Node.js version numbers and instead focuses on the Stream and the datestamp. Enterprises that want stable Node.js versions to build on would select a Stable tag covered by LTS. Developers that want the newest features frequently would select the latest Current. Developers who want to live on the cutting edge would select the latest Canary. The specific semantic version associated with any specific tag just becomes additional build metadata.

• It does mean that users will not receive new features without also taking on potentially breaking changes. This is a key trade off. New features roll out faster in Canary and Current, but never into existing Stables.

The reference google slides (https://docs.google.com/presentation/d/13JkM4SLm176XE92pIiDSNtx-MMY2v7eLs9b_qzSc8DA/edit#slide=id.gbaf0f0fe08_8_20) include a number of other questions and discussion points that have come up from folks. Before weighing in on this proposal, I ask that you please take a look through the slide deck and formulate any questions that you may have about the proposal.

I'd like to keep the discussion focused and moderated as I'm sure there are many Strongly Held Opinions. Let's start first with clarifying questions before we move on to debate.

[wesleytodd]

How would this new release process work for a package author in this scenario?

1. They have committed to supporting active_lts in the current model and would like to give their users a similar commitment going forward
2. The new release process is introduced
3. They have a compelling reason to adopt a new feature introduced in a node version under the new release plan

I see two possible outcomes:

1. They major version rev their package, drop support for existing versions and require users to update (the most common way package authors work with major lines today).
2. They are now responsible for supporting many more major version lines (one per week at most) of their package to support their users.

The problem with number 1 is those older lines are never going to get security and bug patches. In order for users to update they also are required to take on breaking changes in node. Imagine a slow to update native dep (hard to imagine right :P) along side a package which updates for new node features then has a CVE on all previous versions. In this case a user cannot take on new security patch of one package because the are unable to update node versions because of the broken native dep.

The problem with number 2 is that we have greatly increased the workload on package maintainers, with all the rather obvious issues related work offered freely.

I don't think the current release process is without these type of issues, but also the edges are more well known in the community and node breaking changes are slow enough to keep up. This new way sounds like it has more risk of these types of issues appearing.

[Trott]

Is it possible to devise an incremental approach to going from where we are now to something like this proposal? I think the answer is "no, not really", but if I'm wrong about that, I'd be interested to know what the first incremental step or steps might be.

[targos]

I feel like this should be converted to a discussion (to allow parallel threads)

[bnb]

After some discussion with James's thread on Twitter with the intent to respect his request to fully parse and ask questions before sharing opinions, I... want to share opinions.

My perception so far is that this proposal... changes the mental burden on users without meaningfully improving it.

Specifically, we're extending a ecosystem standard paradigm with our own special context that they'll need to learn just for us. I... am concerned about putting the burden on our users - who have consistently voiced that the current system is too confusing - rather than finding an avenue to manage this from our side of the release process.

I do think that the proposed system could be beneficial in reducing the burden on releasers and help increase the project's velocity. I am just... extremely cautious about the potential putting additional burden on the ecosystem and our users with little tangible benefit to them.

[BethGriggs]

I have read the proposal through several times and asked a few questions, but as someone who has shouldered a reasonable portion of the release work recently, I’d like to share some $thoughts.

While I agree that the release process can be time-consuming/demanding, I don’t believe the biggest pain points are solved by the new proposal, while it has significant impact on end-users.

$thoughts:

• Canary - Semver-major bumps could happen as often as every week
  • Is this what our users desire? I’d have thought if users want to be this close to HEAD, then I’d have thought we would see more requests/downloads/uptake of our nightly builds.
  • For Canary and Current, we can expect more regular breaking changes. When we introduce breakage in the Current release line, our users report it and we revert or fix it. There’s an existing expectation for Current that it does not have major breaking changes. Changing this expectation would be a challenge - and offering users the option of dealing with regular breaking changes or no new feature additions doesn’t seem fair (IMHO).
• Eliminating Active LTS
  • Previously, we have sought to backport as much of what we can to make backporting easier in the future by reducing the divergence (with security/critical fixes in mind).
  • This would eliminate the backporting of new features. But for many new features, backporting to LTS has been crucial for adoption or easing migration (ESM, for example).
• Backporting security fixes to LTS release lines
  • This is mostly a challenge because of the delta/divergence between the release lines over time which means a bespoke security patch is needed for the earlier LTS versions. I don’t believe this proposal will make that aspect easier, as the delta/divergence between the Canary, Current, and Stable will still exist (and I believe is likely to be slightly larger as it may include more breaking changes). We could still have almost ~3-years worth of divergence between Canary/Current and Stable.
• Branch management
  • I have not seen our branch management as a burden during the release process. In the new approach, I believe we’d have to create a temporary/working {major}.{minor}.{patch}-{datestamp}-stable proposal branch for the next release. Does that mean when contributors are trying to target a fix to a Stable line, they have to go find the latest datestamp of that version and open a PR against that? That on the surface appears to be more complex than having a branch that is always vN.x-staging that contributors can target.

The key benefits of the proposal read to me as being mostly to the maintainers/Node.js core developers. It allows Node.js core developers to get their breaking changes out in releases quicker and aims to reduce the burden on releasers. (Although, I don’t believe the proposed model would significantly solve the key challenges I see as a releaser.). I agree with @bnb's thoughts that this will add a burden on the ecosystem and our users with little tangible benefit to them.

Is it possible to devise an incremental approach to going from where we are now to something like this proposal?

As an incremental step, we could preserve most of the current model and start by dropping the ‘Active LTS’ phase for Node.js 16. That would give some time to confirm some of the proposed benefits - such as whether it makes maintenance easier, reduces the burden on releasers, etc. But would also allow us and end-users time to determine the benefits/consequences of not receiving new features in LTS/Stable version.

[jasnell]

Converting to a discussion as suggested...