Expand and improve workers documentation and example configuration

[ShadowJonathan]

A general call;

• Add more documentation surrounding workers (just in general)
• Add example configuration file (a-la sample_config.yaml) for each worker type, or for all workers in one
• Add docker-compose file that includes ready-to-work example for synapse workers

Imo: Currently i think workers are very hard to set up or figure out on its own, because knowledge of it is mostly limited to large setups, and thus that knowledge is never reflected in documentation, or workers seen as an asset to be documented with the same priority.

[clokep]

Add more documentation surrounding workers (just in general)

Is there specific information that you think is missing? the workers docs were recently expanded to include a bit more information.

Add example configuration file (a-la sample_config.yaml) for each worker type, or for all workers in one

I think this is pretty much #8557.

Add docker-compose file that includes ready-to-work example for synapse workers

The current docker-compose stuff is under contrib, so I unfortunately this will probably need to be provided via a contribution. 😄

[ShadowJonathan]

@clokep why was info-needed added here, i thought i was pretty clear; improve workers documentation and add samples, for the first one there's already an issue (good, cuz then i can track that via here), and for the second one i havent seen a thing (with issue or PR), or should that issue be added to another repo?

[clokep]

@ShadowJonathan The bit that is unclear to me is:

Add more documentation surrounding workers (just in general)

Is there specific information that you think is missing? the workers docs were recently expanded to include a bit more information.

What information do you feel is missing?

[ShadowJonathan]

Oh, that part is a general call for a look at the documentation, but i'm not holding it for a requirement (thus the absence of checkmark), its generally just "hey, please give a look at the documentation for workers and judge if everything you know about setting up and configuring workers is in there" request, should i remove it to avoid confusion?

[richvdh]

we certainly need better documentation for how to run workers under docker

[TheDiscordian]

I was asked to drop my opinion in here while we were discussing this issue in chat:

There's a lot of info on types of workers, and little on how to redirect queries from a reverse proxy to the workers effectively. I know it sounds simple to say "just run a worker per CPU", but I didn't find my experience to feel quite so clear-cut.

Put simply, an example showing how to run a server with 4+ cores, using Nginx as a reverse proxy, with an appropriate worker setup / Nginx config would go a long way I think.

[ShadowJonathan]

Extra thoughts;

• docker-compose sample
• full setup sample config

[geier1993]

The Loadbalancing Section is a little bit confusing.
Is there loadbalancing by default? I read that it is customizable to redirect same users to same nodes, or specific rooms to same nodes.
However is this already working by default, how can I configure it?

At least it should be statet that i need to bring my own load balancer.

[geier1993]

the main issue I found with the worker docs is they do describe what to do, but not quite how to get there, ie; the keys are noted but not really the surrounding config for context

... asked by @jwh:covert.tel from the discourse on #synapse:matrix.org

[geier1993]

Maybe reference existing helm charts for Kubernetes?

Thes deprecated repo (https://github.com/ananace/matrix-synapse) references

https://gitlab.com/ananace/charts (espacially https://gitlab.com/ananace/charts/-/tree/master/charts/matrix-synapse up to date!)
and
https://github.com/dacruz21/matrix-chart/

At least concerning "large deployments", concrete kubernetes images are very helpful - at least to mention their existence

EDIT: Within the INSTALL.md some docker images and ansible... but these seem to focus on single process instances - multiple workers/processes are not even mentioned here yet

[richvdh]

Related issues about workers docs:

• Worker docs don't mention how to route /typing #9477
• Document that to_device streams and encryption endpoints can be on workers  #9046
• Document that presence can be on a worker #10010
• Document which Admin APIs can be routed to workers #9629
• Create a machine-readable document for routing requests to workers #12139
• Better documention for worker synapse "resource" requirements #12221

[dklimpel]

I have found some more issues with worker docs:

1. Is the TCP replication up to date? Is this part of the redis replication or the old TCP replication? -> Remove old-style (direct) TCP replication support #11728
2. There are more streams than documented. Or are they not all usable in worker?

• presence_federation
• pushers
• presence
• caches
• to_device
• receipts
• groups
• push_rules
• user_signature
• account_data
• tag_account_data
• typing
• device_lists

3. The example uses a deprecated worker type federation_reader -> systemd-with-workers talks about federation_reader, which is undocumented #8740
   

   synapse/docs/systemd-with-workers/workers/federation_reader.yaml

   Line 1 in 3201863

   worker_app: synapse.app.federation_reader

4. The diagram does not show a connection from Event Persister to main process. But the stream writer for events is a generic_worker and starts only with a connection (worker_replication_host) to main proccess

5. frontend_proxy worker claims it forwards requests to the main process, but it doesn't #3717

[Thumbscrew]

I've raised PR #12737 with some fairly detailed instructions for using Docker Compose to create workers. Please feel free to provide feedback and let me know if this is helpful.