Token categories rely on global state

[bd82]

Specifically this:

• https://github.com/SAP/chevrotain/blob/5e77fc21c1ab39281acf874472a28b2698b2e334/packages/chevrotain/src/scan/tokens.ts#L34

This can cause issues when there are multiple Chevrotain packages in existence.
E.g when the Lexer and Parser are in separate npm packages.

This could also cause issues in splitting up Chevrotain into multiple smaller packages.

[Sciumo]

A good use case of Chevrotain is to execute parse tasks in web workers, possibly with many different parsers handling parse events.

[bd82]

I am not sure if web workers are relevant to this issue. Optimization for web-workers may be around
reducing initialization time, e.g via serialization, particularly if a new web-worker is created for each parsing task.

I have started some guides/feature related to initialization performance here:

• https://sap.github.io/chevrotain/docs/guide/initialization_performance.html

[Sciumo]

Web workers do not share state with the UI thread or other workers. All state must be serialized and transferred by a message; no shared global state. There is a use case for serializing state between lexing and parsing workers. Admittedly probably a corner case from your POV.

[bd82]

It seems less likely that separate web workers would be used for the lexer and the parser which would be needed to re-produce this issue.

But yeah its the same issue 😄

[bd82]

Fixing this (e.g via state on lexer/parser) is likely to cause too many API changes for too little benefit
as I've only encountered an edge case with this state once...

Closing unless this edge case starts causing "real" problems.