[RRFC] Add support to plugin dependencies.

[mshima]

Motivation ("The Why")

Add support to plugin dependencies.
More specific title would be: add support to optional (not installed by default) global-style dependencies.
This would allow a running application installed at current (node_modules) to install an optional dependency and load it.

• Plugin dependencies should be installed at root level.
• They need to be installed with global style, so the node_modules tree won't mutate.
• Can or should be managed by the application that installed it.
• It would be great to have it inside package-lock.json for reproducibility.
  • Can or should be ignored by npm ci, and forced by npm ci --with-plugins.

Example

Using a real yeoman project generation:

mkdir new-project
cd new-project
npm install yo --save-dev
npx yo node --experimental

• npx yo node is using a generator called node that is provided by generator-node package.
• --experimental flag uses an auto install feature that installs generator-node inside a hidden global-style node_modules.
  • Once installed, yo loads it and executes.
  • yo is running so if we mutate current node_modules, it will break.
  • node generator can install another generator, so if we mutate the hidden node_modules will break node generator.

This hidden global-style node_modules is a workaround, I want to get rid of it.

A implementation would be to add dependenciesMeta like peerDependenciesMeta:

{
  devDependencies: {
    'generator-node': '2.8.0'
  },
  dependenciesMeta: { // Unified meta for *Dependencies
    'generator-node': {
      globalStyle: true,
      optional: true,
      autoInstall: false
    }
  }
}

How

Current Behaviour

• No support this feature.

Desired Behaviour

• Add support to global_style dependency.

References

• [FEATURE] Add support to plugin dependencies. arborist#121

[ljharb]

Global things are not and should not be requireable.

If you need something in a project, why wouldn't you want it installed locally, in that project?

[mshima]

@ljharb not a global dependency, a global-style dependency.
Npm currently supports global-style repositories.
I asking to support some global-style dependency on a local repository.

npm/arborist#46:

• globalStyle - boolean, install as if it's a top-level global package (ie, don't resolve any deps above the top level, except peer deps)

[ruyadorno]

Hi @mshima I do wonder what is the main motivation behind the proposal, I'm trying to follow along with the example and can't really figure out the concept:

• Is it a way to make it easier for users to run yeoman generators from npx?
• Is it something to enable sub generators from within generated projects?
• Something else? Maybe elaborate a bit more on what you're trying to achieve 😊

[mshima]

Hi @ruyadorno.

Probably yo isn’t the best example since it’s supposed to run globally.
Let’s do another example with jhipster

• npm install -g [email protected]
• mkdir bug-tracker; cd bug-tracker
• jhipster jdl bug-tracker.jh
  • creates a full application with java and angular
  • at the final of the project generation, jhipster 6.10.1 is installed locally for reproducibility.
  • next execution locally installed jhipster will be used.
• now I want to try vue instead of angular and others blueprints:
• rm -rf src/main/webapp src/test/javascript
  • remove angular related files
• jhipster --with-entities --blueprints vuejs,liquibase
  • it will run jhipster (locally installed now)
  • install generator-jhipster-vuejs, generator-jhipster-liquibase and use them.
• ls .yo-repository/lib/node_modules/
  • generator-jhipster-liquibase generator-jhipster-vuejs

The main reason for this feature request is to allow to install an optional dependency (blueprints, which are yeoman generators) locally in current node_modules without breaking the running application (jhipster).
Currently we are installing in an hidden node_modules (to don't break jhipster) with global-style (global-style to don't break already installed blueprints).

[ruyadorno]

ok, so the actual problem is that jhipster is a hanging/ongoing process and you want to avoid swapping the files that it consumes from within the current node_modules folder?

[mshima]

@ruyadorno yes, that's the main reason.

[mshima]

A little more background about Yeoman.

Lookup (resolver)

It looks for generators everywhere (npm/yarn paths, local/global, and sub-packages probably to workaround missing peer dependency).
And then register into the store.
https://github.com/yeoman/environment/blob/17bf3c30b0217b81171ca42d566e8c333ca283de/lib/resolver.js

Caching a generator (store)

It implements it's own generator cache.
https://github.com/yeoman/environment/blob/17bf3c30b0217b81171ca42d566e8c333ca283de/lib/store.js#L65-L78

Running a generator

Yo command runs the root generator yo foo.

Composing with another generator

• Without auto-install and using npm 6 the generator is forced to add the composed generator as a hard dependency.
  It looks like this:
  this.composeWith(require.resolve('generator-license/app'));
  https://github.com/yeoman/generator-node/blob/25d1ca4ea2122ce08b88398848e6df41f54cbe1a/generators/app/index.js#L362-L368
• With npm 7 (peer dependency auto install), this will work:
  this.composeWith('license/app');

Running a missing generator:

$ yo license
Error license

You don't seem to have a generator with the name “license” installed.
But help is on the way:

You can see available generators via npm search yeoman-generator or via http://yeoman.io/generators/.
Install them with npm install generator-license.

To see all your installed generators run yo without any arguments. Adding the --help option will also show subgenerators.

If yo cannot find the generator, run yo doctor to troubleshoot your system.

Running a missing generator with plugin feature:

Plugin feature is experimental because it does too much that npm 7 makes unnecessary.

$ yo license --experimental

> [email protected] postinstall /Users/mshima/aplicacoes/yo/.yo-repository/lib/node_modules/generator-license/node_modules/ejs
> node ./postinstall.js

+ [email protected]
added 448 packages from 349 contributors in 22.505s
✔ generator-license installed.
? What's your name: (Marcelo)

$ find .yo-repository/
.yo-repository/
.yo-repository//lib
.yo-repository//lib/node_modules
.yo-repository//lib/node_modules/generator-license
.yo-repository//lib/node_modules/generator-license/app
.yo-repository//lib/node_modules/generator-license/app/index.js

NPX

npx installs yo and yo install generator-license, you can see both install process when executing this command:

npx yo license --experimental

Plugin using NPX implementation

It should be a lib (preferably) that installs modules like npx.

• it should allow versioned installs.
• it should return the install destination so yeoman can load it.
• we should consider to install peer dependencies and return them too.

I don't know specifics of how it's installed and how yeoman would load it.

[ruyadorno]

Action item from the OpenRFC call:

• Let's create a follow up RFC for refactoring libnpmexec into its own lib so that projects such as Yeoman can easily consume it through a programmatic API

[mshima]

@ruyadorno looks great thanks.
I didn't had time to watch live.
I will watch it soon.