Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cacheDirectory babel-loader #5436

Conversation

MichaelrMentele
Copy link
Contributor

@MichaelrMentele MichaelrMentele commented May 5, 2022

Noticed when investigating #5231 that we aren't caching transpilation when using the babel-loader plugin that glues together webpack and babel.

The impact of this is that we are potentially retranspiling ALL of our files when often some tiny fraction of them actually need to be transpiled (when we call bebel with webpack through babel-loader)

Seems to have a net O(10%) speed up for net build time for the redwood web side. Course babel-loader itself is only one part of that process -- according to the docs this should give an O(2x) speed boost when cached so working backwards sounds like our overall transpilation time is about ~6 s or roughly 20% of overall build time -- for a relatively small redwood project on my machine.

Redwood Build Time Before:

➜  web git:(main) ✗ python -m timeit "__import__('os').system('yarn workspace web rw build web')
"
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
1 loop, best of 5: 42.6 sec per loop

Redwood Build Time After:

➜  web git:(main) ✗ python -m timeit "__import__('os').system('yarn workspace web rw build web')
"
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
1 loop, best of 5: 39.2 sec per loop

@netlify
Copy link

netlify bot commented May 5, 2022

Deploy Preview for redwoodjs-docs canceled.

Name Link
🔨 Latest commit aa90c49
🔍 Latest deploy log https://app.netlify.com/sites/redwoodjs-docs/deploys/62742ed710cbd100095af8f5

@MichaelrMentele MichaelrMentele marked this pull request as ready for review May 5, 2022 02:24
@MichaelrMentele
Copy link
Contributor Author

Some reference plugins I used for profiling -- note the built in profiler does NOT work -- throws when dropped into chrome debugger. @dac09 mentioned it was neever updated to work with webpack 5.

const webpack = require('webpack')
const WebpackBar = require('webpackbar')
const path = require('node:path')
const SpeedMeasurePlugin = require('speed-measure-webpack-plugin')

/** @returns {import('webpack').Configuration} Webpack Configuration */
module.exports = (config, { mode }) => {
  if (mode === 'development') {
    // Add dev plugin
  }

  // Add custom rules for your project
  // config.module.rules.push(YOUR_RULE)
  // config.plugins.push(new webpack.debug.ProfilingPlugin())
  // config.plugins.push(
  //   new webpack.debug.ProfilingPlugin({
  //     outputPath: path.join(__dirname, 'profiling/profileEvents.json'),
  //   })
  // )

  const smp = new SpeedMeasurePlugin({
    granularLoaderData: true,
    outputFormat: 'human',
    // outputFormat: 'humanVerbose',
    loaderTopFiles: 20,
  })
  config.plugins.push(smp)

  // Add custom plugins for your project
  // config.plugins.push(YOUR_PLUGIN)
  // config.devtool = false
  // config.cache = true

  config.devtool = 'source-map'
  config.cache = true
  // config.watch = true
  // config.plugins.push(new WebpackBar({ profile: true, color: 'red' }))

  // disable source map saves about 1-2 seconds
  // enabling caching saves

  // config.watch = true

  return config
}

@MichaelrMentele
Copy link
Contributor Author

Running SMB -- where do the other 10 s go?

➜  leftlane_app git:(main) ✗ yarn rw build web --verbose
[12:29:56] Cleaning Web... [started]
[12:29:56] Cleaning Web... [completed]
[12:29:56] Building Web... [started]
(node:50101) [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK] DeprecationWarning: Compilation.hooks.normalModuleLoader was moved to NormalModule.getCompilationHooks(compilation).loader
(Use `node --trace-deprecation ...` to show where the warning was created)


 SMP  ⏱
General output time took 25.81 secs

 SMP  ⏱  Loaders
modules with no loaders took 8.02 secs
  module count                                                                                                     = 12542
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/material/Popover/Popover.js                    = 0.204 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/material/Paper/paperClasses.js                 = 0.201 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/icons-material/esm/index.js                    = 0.2 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_unicodeToArray.js                           = 0.193 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseToString.js                             = 0.193 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_asciiToArray.js                             = 0.189 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseSlice.js                                = 0.188 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_defineProperty.js                           = 0.186 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseFor.js                                  = 0.186 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/noop.js                                      = 0.185 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_trimmedEndIndex.js                          = 0.185 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/core-js-pure/modules/es.date.now.js                 = 0.184 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_Set.js                                      = 0.184 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@popperjs/core/lib/dom-utils/getWindowScrollBarX.js = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_setCacheHas.js                              = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseIndexOf.js                              = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_MapCache.js                                 = 0.182 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_setCacheAdd.js                              = 0.182 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseProperty.js                             = 0.181 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_basePropertyDeep.js                         = 0.181 secs
babel-loader took 1.71 secs
  module count                                                                                                                   = 70
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Stage/NewStage/NewStage.tsx                                 = 0.263 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/VehicleCard/VehicleCard.tsx                                 = 0.256 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/StageCard/StageCard.tsx                                     = 0.255 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/NewVehicleForm/VinLookupCell/VinLookupCell.tsx              = 0.254 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/NewVehicleForm/VinForm/VinForm.tsx                          = 0.253 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/EditPipelineCell/EditPipelineCell.tsx              = 0.252 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/PipelineCell/PipelineCell.tsx                      = 0.251 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/NewPipeline/NewPipeline.tsx                        = 0.25 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/PipelinesCell/PipelinesCell.tsx                    = 0.249 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/App.tsx                                                                = 0.248 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/api/src/lib/logger.ts                                                          = 0.17 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/SettingsPage/SettingsPage.tsx                                    = 0.154 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/UploadVehicleImageComponent/UploadVehicleImageComponent.tsx = 0.153 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/SignupPage/SignupPage.tsx                                        = 0.138 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/VehicleDetailPage/VehicleDetailPage.tsx                          = 0.135 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/RemoteImageCell/RemoteImageCell.tsx                         = 0.135 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/VehiclePipelinePage/VehiclePipelinePage.tsx                      = 0.132 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/layouts/DefaultLayout/DefaultLayout.tsx                                = 0.125 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/Pipeline/EditPipelinePage/EditPipelinePage.tsx                   = 0.124 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/Pipeline/PipelinePage/PipelinePage.tsx                           = 0.121 secs
mini-css-extract-plugin, and
css-loader took 0.214 secs
  module count                                                         = 2
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css = 0.212 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css    = 0.203 secs
css-loader took 0.058 secs
  module count                                                                                                                                                                                                                                                                                                               = 2
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css.webpack[javascript/auto]!=!/Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css       = 0.057 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css.webpack[javascript/auto]!=!/Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css = 0.016 secs
null-loader took 0.025 secs
  module count                                                                                       = 1
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/router/dist/splash-page.js = 0.025 secs
html-webpack-plugin took 0.01 secs
  module count                                                                                                                                                      = 1
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/html-webpack-plugin/lib/loader.js!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.html = 0.01 secs



assets by path static/js/*.js 1.76 MiB
  assets by chunk 490 KiB (id hint: vendors)
    asset static/js/709.e1832718.chunk.js 378 KiB [emitted] [immutable] [minimized] [big] (id hint: vendors) 2 related assets
    asset static/js/846.3512a8f1.chunk.js 57.6 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/857.da72520d.chunk.js 32.4 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/961.c36d2134.chunk.js 16.1 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/248.65d25381.chunk.js 6.54 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
  + 27 assets
asset static/css/app.1fe0c5b7.css 5.83 KiB [emitted] [immutable] [minimized] (name: app)
asset build-manifest.json 4.71 KiB [emitted]
asset mockServiceWorker.js 4.18 KiB [emitted] [from: public/mockServiceWorker.js] [copied] [minimized]
asset README.md 1.9 KiB [emitted] [from: public/README.md] [copied]
asset favicon.png 1.7 KiB [emitted] [from: public/favicon.png] [copied]
asset index.html 936 bytes [emitted]
asset robots.txt 24 bytes [emitted] [from: public/robots.txt] [copied]
Entrypoint app [big] 659 KiB (2.53 MiB) = static/js/runtime-app.62d0b37a.js 4.44 KiB static/css/app.1fe0c5b7.css 5.83 KiB static/js/app.60e81984.js 649 KiB 2 auxiliary assets
orphan modules 8.94 MiB (javascript) 2.4 KiB (runtime) [orphan] 11589 modules
runtime modules 13.2 KiB 21 modules
cacheable modules 4.4 MiB (javascript) 6.37 KiB (css/mini-extract)
  modules by path ../node_modules/ 3.21 MiB 1001 modules
  modules by path ./src/ 1.19 MiB (javascript) 6.37 KiB (css/mini-extract)
    modules by path ./src/pages/ 1.1 MiB 24 modules
    modules by path ./src/components/ 21.4 KiB
      ./src/components/UI/LoadingSpinner/LoadingSpinner.tsx 253 bytes [built] [code generated]
      ./src/components/VehicleSummaryCell/VehicleSummaryCell.tsx 9.6 KiB [built] [code generated]
      ./src/components/RemoteImageCell/RemoteImageCell.tsx 2.72 KiB [built] [code generated]
      + 3 modules
    modules by path ./src/*.css 6.37 KiB
      css ../node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!./src/scaffold.css 6.37 KiB [built] [code generated]
      css ../node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!./src/index.css 0 bytes [built] [code generated]
    ./src/App.tsx + 23 modules 72.4 KiB [built] [code generated]

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets:
  static/js/app.60e81984.js (649 KiB)
  static/js/180.163d65f8.chunk.js (372 KiB)
  static/js/709.e1832718.chunk.js (378 KiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (659 KiB)
      static/js/runtime-app.62d0b37a.js
      static/css/app.1fe0c5b7.css
      static/js/app.60e81984.js


webpack 5.70.0 compiled with 2 warnings in 25814 ms
Creating 200.html...
[12:30:27] Building Web... [completed]
[12:30:27] Prerendering Web... [started]
[12:30:27] Prerendering Web... [completed]
➜  leftlane_app git:(main) ✗

@dac09
Copy link
Contributor

dac09 commented May 10, 2022

Hey @MichaelrMentele - appreciate all the effort into these PRs - but something I wanted to point out - the jest tests don't actually use webpack - and therefore doesn't use babel loader. Yes rw build performance may improve with this cache directory setting, but you can see that its quite minimal!

I would say the two files that are most likely the culprits for the jest initialisation time are these:

  1. jest-preset.js
  2. jest.setup.js

In particular, I have a feeling it may be the loading of the project mocks in the beforeEach in jest setup.

@dac09 dac09 added the release:chore This PR is a chore (means nothing for users) label May 10, 2022
@jtoar jtoar linked an issue May 20, 2022 that may be closed by this pull request
@jtoar
Copy link
Contributor

jtoar commented Jul 7, 2022

@dac09 Michael mentioned in the OP that this was for build, not for test. And it seems that it does improve build time, even if it’s a little. We’ve certainly merged smaller performance gains before. So thoughts on getting this one in? It's not critical, but it'd nice to bring this one to a close one way or another.

@jtoar jtoar assigned dac09 and jtoar Jul 7, 2022
@jtoar
Copy link
Contributor

jtoar commented Jul 28, 2022

@MichaelrMentele I discussed with @dac09 the other day. We're going to close this one cause we're not sure that the possible performance gain is worth the change webpack config, which just comes with some unknowns for us. If we had more tests around our webpack config, we'd be more confident in this. Sorry that this was open for so long—thanks for taking the time to do the research you did and contribute!

@jtoar jtoar closed this Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release:chore This PR is a chore (means nothing for users)
Projects
Status: Archived
Development

Successfully merging this pull request may close these issues.

Jest Web Test Run Initialization is Slow O(10x) Slower than it Should Be
3 participants