[docs] Docs restructure

[siriwatknp]

From the decision in the Notion, the documentation will be restructured into each product.

Goal

Incrementally and smoothly transition to the new structure by preserving the same DX for the community and maintainers (tests should be added along the way).

The new repo structure:

Improvements are not the focus of this initiative. However, they can be listed in this issue and prioritized later.

Plan

To achieve a smooth transition, the work is split into 3 phases.

1. Preparation

Prepare E2E tests, migration tool, update scripts and add UI (each PR should be able to merge without affecting the production code, either set as feature flag or exclude from production build).

• E2E tests for the website via playwright (playwright is already existed in the project). [test] Add E2E website tests #30128
• Create product pages config and update _app.js to switch between product. [docs] Add script to clone pages #30107
• Migration script to duplicate pages to the new location (under /material/*). [docs] Add script to clone pages #30107
• Update these scripts to compat with the new structure (it must still work with the current structure)
  • docs:api:build [core] Update buildApi script to support new structure #30245
  • docs:i18n turns out this one needs no change because the translations directory remains the same 😊
  • docs:typescript:formatted [core] Update docs:typescript:formatted to support new structure #30248
• Update search to redirect to /material/* for material content. [docs] Redirect old urls to /material/* when search #30286
• Refine scripts to generate pages with the decided structure (components & APIs) [docs] Prepare scripts for migrating to new structure #30386
• Create similar migration workflow for mui-x repo [docs] Prepare scripts + E2E tests for migration mui-x#3515
• UI: Product identifier + versioning. [docs] Add products identifier and drawer #30283
• UI: Product drawer. (Product drawer is open when clicking on the app icon) [docs] Add products identifier and drawer #30283
• Deploy to a migration.mui.com to let ahrefs crawl to make sure that all of the links are working as expected. (need to turn enable_redirects on)

2.1 Migrate content

At this phase, the new URLs have to be deployed to production so that algolia can crawl and index. However, before algolia finish the indexing I think we should preserve the old URLs (meaning, people still browsing old URLs and won't realize that we are migrating the content).

• Run the migration scripts for Core yarn docs:migrate:pages && yarn docs:api. [docs] Migrate content to the new location #30757
• Run the migration scripts for MUI-X yarn docs:migrate:pages && yarn docs:api. [docs] Migrate content to the new location mui-x#3730

What'd happen after running the scripts

• data (demos, markdowns) will be moved from docs/src/pages/* to docs/data/*. (Both the old & new URLs are using the same demos, markdowns from the new location so we are always using the single source of truth)
• feature toggle enable_product_scope: true is turned on. This enables several things such as E2E tests for the new URLs, buildApi script, and query markdowns from the new location.
• product identifier & navigation are rendered on new locations but not on old URLs
• All of the links for existing pages remain the same, which means users won’t realize that the duplicated contents exist. (But they can accidentally access via direct URL).
• yarn docs:api will generate API pages for both previous & new URLs

3. New product spaces

Est. mid of Mar.

• Rename material to /material-ui/* [docs] Use material-ui for product name #31200
• [docs] Refining the product identifier & drawer #31201
• Set up a new Algolia crawler for indexing new URLs (old URLs are neglected). This new crawler has a different ID than the existing one. (old ID for existing URLs, new ID for new URLs, this way the redirect can be done at any time. [docs] Use new Algolia app for new structure #31178
• Turn on feature toggle enable_mui_base and initialize MUI Base installation page. [docs] Add Base installation page #30969
• Refine the search experience for multiple product spaces [RFC] Search experience for new product spaces #31253
• Move MUI Base content from material to its own space [docs] Move unstyled components' docs to the Base space #31414
• Write a short announcement on the blog. [blog] Share what's changed about the new docs structure #32044
• Date pickers docs [DatePicker] Initial commit mui-x#3451
• Remove date pickers from Material-UI [docs] Update mui-x on material-ui navigation #31810

4. Go live

Est. 4th April 2022.

• Merge these PRs and deploy.
  • Core: [docs] Redirect to new urls #32048
  • X: [docs] Go live with the new urls! mui-x#4313

5. Clean up

• remove material-redirects.spec e2e after redirects period
• delete old pages inside docs/pages/...`
• `remove material-old e2e test
• remove replaceHtmlLinks, replaceUrlLinks
• remove migration script
• remove old config in buildApi.ts
• use new logic in parseMarkdown.js for the API section

cc @mui-org/core @mui-org/x

[oliviertassinari]

The current plan looks great. A few more thoughts:

1. I think that we should move with as small steps as possible. As far as I understand it, we only need two docs to justify doing the restructuring, and the blog post. So we could do two, but anything extra (e.g. 4) would be even better. Now, my point is that I would challenge that delaying the release beyond Q1 2022 for having more docs has an overall positive ROI. Why? Because we wouldn't go to prod as fast as we can, and people tend to forget, and not notice things (we would be eventually in a great state with all the docs created).
2. I feel that https://mui.com/material-ui/react-modal/ would resonate better than https://mui.com/material/react-modal/. Am I the only one? My main concern is that https://mui.com/x/ works because the full product name is MUI X. But https://mui.com/material/, the product name is not MUI Material (decided to not have developers re-learn the existing brand name). I don't have a strong point of view on this but wanted to raise this potential opportunity in case it resonates with other team members. Related to [docs] Add Base installation page #30969 (comment)

[mnajdova]

I feel that https://mui.com/material-ui/react-modal/ would resonate better than https://mui.com/material/react-modal/. Am I the only one?

+1 on this one

[siriwatknp]

I feel that https://mui.com/material-ui/react-modal/ would resonate better than https://mui.com/material/react-modal/. Am I the only one?

🥲 Another migration, but it looks like it should be mui.com/material-ui/*.

[siriwatknp]

@oliviertassinari Another thought, is it

https://mui.com/material-ui/react-modal/ or
https://mui.com/materialui/react-modal/

? given that we decided to go with "Material UI" without a hyphen.

I had this thought after seeing https://headlessui.dev

[michaldudak]

Hyphens are commonly placed in URLs in place of spaces, so even though the name of the library is Material UI, to me it's perfectly fine to have material-ui in the URL.