Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Run complement with Synapse workers manually. #10039

Merged
merged 5 commits into from
May 24, 2021
Merged

Run complement with Synapse workers manually. #10039

merged 5 commits into from
May 24, 2021

Conversation

clokep
Copy link
Member

@clokep clokep commented May 21, 2021 •
edited
Loading

This attempts to be able to run complement with Synapse worker mode via the complement.sh script. Adding a WORKERS=1 will cause them to be used.

It also fixes a couple of minor things necessary to get this to work.

Corresponding complement PR: matrix-org/complement#116.

@clokep clokep mentioned this pull request May 21, 2021
Merged
@clokep
Copy link
Member Author

clokep commented May 21, 2021

Right now this is seems to boot OK, but gives an error:

    msc3083_test.go:139: Deploy: Deploy returned error Deploy: Failed to deploy image {Containers:-1 Created:1621608279 ID:sha256:ef45373e3bf36da9921bafd8f0ca774d4da2fb0f7edd7493b9194e75f3a0b31c Labels:map[access_token_@alice:hs1:syt_YWxpY2U_TtQsTDoStvJgGUvtdCAk_3eUy8k access_token_@bob:hs1:syt_Ym9i_ZMEmNUKPALcDQnsQKAdp_0LPGjt complement_blueprint:one_to_one_room complement_context:one_to_one_room.hs1 complement_hs_name:hs1 desktop.docker.io/mounts/0/Source:/Users/clokep/matrix/complement/tests/ca desktop.docker.io/mounts/0/SourceKind:hostFile desktop.docker.io/mounts/0/Target:/ca org.opencontainers.image.documentation:https://github.com/matrix-org/synapse/blob/master/docker/README.md org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/matrix-org/synapse.git org.opencontainers.image.url:https://matrix.org/docs/projects/server/synapse] ParentID:sha256:a0e9cf735b5dd856164b87476dc9e4a59f4625cc8fe5afde328f1fe3347978e8 RepoDigests:[] RepoTags:[localhost/complement:one_to_one_room.hs1] SharedSize:-1 Size:741996284 VirtualSize:741996284} : one_to_one_room.hs1: failed to check server is up. GET http://localhost:56020/_matrix/client/versions => error: Get "http://localhost:56020/_matrix/client/versions": dial tcp: lookup localhost: no such host

(I was running an individual test-case just to reduce noise.)

@clokep clokep marked this pull request as ready for review May 24, 2021 15:55
@clokep
Copy link
Member Author

clokep commented May 24, 2021

I think this is all improvements although it isn't quite workable yet, so requesting review.

@clokep clokep requested a review from anoadragon453 May 24, 2021 15:55
clokep
clokep commented May 24, 2021
View reviewed changes
scripts-dev/complement.sh Outdated
Comment on lines 37 to 38
DOCKERTAG_SUFFIX=-workers
DOCKERFILE_SUFFIX=Workers
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anoadragon453 asked for these to just be separate variables instead of suffixes.

@clokep clokep removed the request for review from anoadragon453 May 24, 2021 15:56
@clokep clokep requested a review from anoadragon453 May 24, 2021 17:35
anoadragon453
anoadragon453 approved these changes May 24, 2021
View reviewed changes
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some documentation to the top of complement.sh that mentions that WORKERS can be set to test Synapse in worker mode? Otherwise this lgtm, thanks!

@clokep clokep merged commit 7d90d6c into develop May 24, 2021
@clokep clokep deleted the clokep/complement-workers branch May 24, 2021 19:32
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jun 5, 2021
@Midar
Update chat/matrix-synapses to 1.35.1
7f7c8d5 
Synapse 1.35.1 (2021-06-03)
===========================

Bugfixes
--------

- Fix a bug introduced in v1.35.0 where invite-only rooms would be shown to all users in a space, regardless of if the user had access to it. ([\#10109](matrix-org/synapse#10109))


Synapse 1.35.0 (2021-06-01)
===========================

Note that [the tag](https://github.com/matrix-org/synapse/releases/tag/v1.35.0rc3) and [docker images](https://hub.docker.com/layers/matrixdotorg/synapse/v1.35.0rc3/images/sha256-34ccc87bd99a17e2cbc0902e678b5937d16bdc1991ead097eee6096481ecf2c4?context=explore) for `v1.35.0rc3` were incorrectly built. If you are experiencing issues with either, it is recommended to upgrade to the equivalent tag or docker image for the `v1.35.0` release.

Deprecations and Removals
-------------------------

- The core Synapse development team plan to drop support for the [unstable API of MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option, in August 2021. Client authors should ensure that their clients are updated to use the stable API (which has been supported since Synapse 1.30) well before that time, to give their users time to upgrade. ([\#10101](matrix-org/synapse#10101))

Bugfixes
--------

- Fixed a bug causing replication requests to fail when receiving a lot of events via federation. Introduced in v1.33.0. ([\#10082](matrix-org/synapse#10082))
- Fix HTTP response size limit to allow joining very large rooms over federation. Introduced in v1.33.0. ([\#10093](matrix-org/synapse#10093))


Internal Changes
----------------

- Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091))


Synapse 1.35.0rc2 (2021-05-27)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](matrix-org/synapse#10079))


Synapse 1.35.0rc1 (2021-05-25)
==============================

Features
--------

- Add experimental support to allow a user who could join a restricted room to view it in the spaces summary. ([\#9922](matrix-org/synapse#9922), [\#10007](matrix-org/synapse#10007), [\#10038](matrix-org/synapse#10038))
- Reduce memory usage when joining very large rooms over federation. ([\#9958](matrix-org/synapse#9958))
- Add a configuration option which allows enabling opentracing by user id. ([\#9978](matrix-org/synapse#9978))
- Enable experimental support for [MSC2946](matrix-org/matrix-spec-proposals#2946) (spaces summary API) and [MSC3083](matrix-org/matrix-spec-proposals#3083) (restricted join rules) by default. ([\#10011](matrix-org/synapse#10011))


Bugfixes
--------

- Fix a bug introduced in v1.26.0 which meant that `synapse_port_db` would not correctly initialise some postgres sequences, requiring manual updates afterwards. ([\#9991](matrix-org/synapse#9991))
- Fix `synctl`'s `--no-daemonize` parameter to work correctly with worker processes. ([\#9995](matrix-org/synapse#9995))
- Fix a validation bug introduced in v1.34.0 in the ordering of spaces in the space summary API. ([\#10002](matrix-org/synapse#10002))
- Fixed deletion of new presence stream states from database. ([\#10014](matrix-org/synapse#10014), [\#10033](matrix-org/synapse#10033))
- Fixed a bug with very high resolution image uploads throwing internal server errors. ([\#10029](matrix-org/synapse#10029))


Updates to the Docker image
---------------------------

- Fix bug introduced in Synapse 1.33.0 which caused a `Permission denied: '/homeserver.log'` error when starting Synapse with the generated log configuration. Contributed by Sergio Miguéns Iglesias. ([\#10045](matrix-org/synapse#10045))


Improved Documentation
----------------------

- Add hardened systemd files as proposed in [#9760](matrix-org/synapse#9760) and added them to `contrib/`. Change the docs to reflect the presence of these files. ([\#9803](matrix-org/synapse#9803))
- Clarify documentation around SSO mapping providers generating unique IDs and localparts. ([\#9980](matrix-org/synapse#9980))
- Updates to the PostgreSQL documentation (`postgres.md`). ([\#9988](matrix-org/synapse#9988), [\#9989](matrix-org/synapse#9989))
- Fix broken link in user directory documentation. Contributed by @junquera. ([\#10016](matrix-org/synapse#10016))
- Add missing room state entry to the table of contents of room admin API. ([\#10043](matrix-org/synapse#10043))


Deprecations and Removals
-------------------------

- Removed support for the deprecated `tls_fingerprints` configuration setting. Contributed by Jerin J Titus. ([\#9280](matrix-org/synapse#9280))


Internal Changes
----------------

- Allow sending full presence to users via workers other than the one that called `ModuleApi.send_local_online_presence_to`. ([\#9823](matrix-org/synapse#9823))
- Update comments in the space summary handler. ([\#9974](matrix-org/synapse#9974))
- Minor enhancements to the `@cachedList` descriptor. ([\#9975](matrix-org/synapse#9975))
- Split multipart email sending into a dedicated handler. ([\#9977](matrix-org/synapse#9977))
- Run `black` on files in the `scripts` directory. ([\#9981](matrix-org/synapse#9981))
- Add missing type hints to `synapse.util` module. ([\#9982](matrix-org/synapse#9982))
- Simplify a few helper functions. ([\#9984](matrix-org/synapse#9984), [\#9985](matrix-org/synapse#9985), [\#9986](matrix-org/synapse#9986))
- Remove unnecessary property from SQLBaseStore. ([\#9987](matrix-org/synapse#9987))
- Remove `keylen` param on `LruCache`. ([\#9993](matrix-org/synapse#9993))
- Update the Grafana dashboard in `contrib/`. ([\#10001](matrix-org/synapse#10001))
- Add a batching queue implementation. ([\#10017](matrix-org/synapse#10017))
- Reduce memory usage when verifying signatures on large numbers of events at once. ([\#10018](matrix-org/synapse#10018))
- Properly invalidate caches for destination retry timings every (instead of expiring entries every 5 minutes). ([\#10036](matrix-org/synapse#10036))
- Fix running complement tests with Synapse workers. ([\#10039](matrix-org/synapse#10039))
- Fix typo in `get_state_ids_for_event` docstring where the return type was incorrect. ([\#10050](matrix-org/synapse#10050))
@clokep clokep mentioned this pull request Aug 24, 2021
Merged
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Sep 1, 2021
@babolivier
Merge tag 'v1.35.0' into babolivier/dinsic_1.41.0
200ee12 
Synapse 1.35.0 (2021-06-01)
===========================

Note that [the tag](https://github.com/matrix-org/synapse/releases/tag/v1.35.0rc3) and [docker images](https://hub.docker.com/layers/matrixdotorg/synapse/v1.35.0rc3/images/sha256-34ccc87bd99a17e2cbc0902e678b5937d16bdc1991ead097eee6096481ecf2c4?context=explore) for `v1.35.0rc3` were incorrectly built. If you are experiencing issues with either, it is recommended to upgrade to the equivalent tag or docker image for the `v1.35.0` release.

Deprecations and Removals
-------------------------

- The core Synapse development team plan to drop support for the [unstable API of MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option, in August 2021. Client authors should ensure that their clients are updated to use the stable API (which has been supported since Synapse 1.30) well before that time, to give their users time to upgrade. ([\#10101](matrix-org/synapse#10101))

Bugfixes
--------

- Fixed a bug causing replication requests to fail when receiving a lot of events via federation. Introduced in v1.33.0. ([\#10082](matrix-org/synapse#10082))
- Fix HTTP response size limit to allow joining very large rooms over federation. Introduced in v1.33.0. ([\#10093](matrix-org/synapse#10093))

Internal Changes
----------------

- Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091))

Synapse 1.35.0rc2 (2021-05-27)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](matrix-org/synapse#10079))

Synapse 1.35.0rc1 (2021-05-25)
==============================

Features
--------

- Add experimental support to allow a user who could join a restricted room to view it in the spaces summary. ([\#9922](matrix-org/synapse#9922), [\#10007](matrix-org/synapse#10007), [\#10038](matrix-org/synapse#10038))
- Reduce memory usage when joining very large rooms over federation. ([\#9958](matrix-org/synapse#9958))
- Add a configuration option which allows enabling opentracing by user id. ([\#9978](matrix-org/synapse#9978))
- Enable experimental support for [MSC2946](matrix-org/matrix-spec-proposals#2946) (spaces summary API) and [MSC3083](matrix-org/matrix-spec-proposals#3083) (restricted join rules) by default. ([\#10011](matrix-org/synapse#10011))

Bugfixes
--------

- Fix a bug introduced in v1.26.0 which meant that `synapse_port_db` would not correctly initialise some postgres sequences, requiring manual updates afterwards. ([\#9991](matrix-org/synapse#9991))
- Fix `synctl`'s `--no-daemonize` parameter to work correctly with worker processes. ([\#9995](matrix-org/synapse#9995))
- Fix a validation bug introduced in v1.34.0 in the ordering of spaces in the space summary API. ([\#10002](matrix-org/synapse#10002))
- Fixed deletion of new presence stream states from database. ([\#10014](matrix-org/synapse#10014), [\#10033](matrix-org/synapse#10033))
- Fixed a bug with very high resolution image uploads throwing internal server errors. ([\#10029](matrix-org/synapse#10029))

Updates to the Docker image
---------------------------

- Fix bug introduced in Synapse 1.33.0 which caused a `Permission denied: '/homeserver.log'` error when starting Synapse with the generated log configuration. Contributed by Sergio Miguéns Iglesias. ([\#10045](matrix-org/synapse#10045))

Improved Documentation
----------------------

- Add hardened systemd files as proposed in [#9760](matrix-org/synapse#9760) and added them to `contrib/`. Change the docs to reflect the presence of these files. ([\#9803](matrix-org/synapse#9803))
- Clarify documentation around SSO mapping providers generating unique IDs and localparts. ([\#9980](matrix-org/synapse#9980))
- Updates to the PostgreSQL documentation (`postgres.md`). ([\#9988](matrix-org/synapse#9988), [\#9989](matrix-org/synapse#9989))
- Fix broken link in user directory documentation. Contributed by @junquera. ([\#10016](matrix-org/synapse#10016))
- Add missing room state entry to the table of contents of room admin API. ([\#10043](matrix-org/synapse#10043))

Deprecations and Removals
-------------------------

- Removed support for the deprecated `tls_fingerprints` configuration setting. Contributed by Jerin J Titus. ([\#9280](matrix-org/synapse#9280))

Internal Changes
----------------

- Allow sending full presence to users via workers other than the one that called `ModuleApi.send_local_online_presence_to`. ([\#9823](matrix-org/synapse#9823))
- Update comments in the space summary handler. ([\#9974](matrix-org/synapse#9974))
- Minor enhancements to the `@cachedList` descriptor. ([\#9975](matrix-org/synapse#9975))
- Split multipart email sending into a dedicated handler. ([\#9977](matrix-org/synapse#9977))
- Run `black` on files in the `scripts` directory. ([\#9981](matrix-org/synapse#9981))
- Add missing type hints to `synapse.util` module. ([\#9982](matrix-org/synapse#9982))
- Simplify a few helper functions. ([\#9984](matrix-org/synapse#9984), [\#9985](matrix-org/synapse#9985), [\#9986](matrix-org/synapse#9986))
- Remove unnecessary property from SQLBaseStore. ([\#9987](matrix-org/synapse#9987))
- Remove `keylen` param on `LruCache`. ([\#9993](matrix-org/synapse#9993))
- Update the Grafana dashboard in `contrib/`. ([\#10001](matrix-org/synapse#10001))
- Add a batching queue implementation. ([\#10017](matrix-org/synapse#10017))
- Reduce memory usage when verifying signatures on large numbers of events at once. ([\#10018](matrix-org/synapse#10018))
- Properly invalidate caches for destination retry timings every (instead of expiring entries every 5 minutes). ([\#10036](matrix-org/synapse#10036))
- Fix running complement tests with Synapse workers. ([\#10039](matrix-org/synapse#10039))
- Fix typo in `get_state_ids_for_event` docstring where the return type was incorrect. ([\#10050](matrix-org/synapse#10050))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@anoadragon453 anoadragon453 anoadragon453 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@clokep @anoadragon453