Proof of concept for GitHub Actions #9661

callahad · 2021-03-22T08:51:20Z

Compare to https://github.com/matrix-org/pipelines/blob/master/synapse/pipeline.yml

Per discussion in https://github.com/matrix-org/synapse/discussions/9622

(I think this is complete, aside from double-checking logs to and log uploading from the Buildkite pipeline)

Signed-off-by: Dan Callahan [email protected]

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
Pull request includes a sign off
Code style is correct (run the linters)

Signed-off-by: Dan Callahan <[email protected]>

.github/workflows/tests.yml

ShadowJonathan · 2021-03-22T09:12:08Z

.github/workflows/tests.yml

+
+  trial-pypy:
+    # Very slow; only run if the branch name includes 'pypy'
+    if: ${{ contains(github.ref, 'pypy') && !failure() }}


Could it be made that it also runs if the PR/commit name has "pypy"? (preferably PR name)

It could, but I think the expression would be quite a bit longer?

I think it'd be something like github.event == 'pull_request' && (contains(github.event.pull_request.title, 'pypy') || contains(github.ref, 'pypy') ) && !failure())?

Looks good, and I don't think the expression being longer is a problem, most people wouldn't see this unless they were editing this specific part of the pipeline.

Adding some comments to deconstruct what is happening could help, tho.

.github/workflows/tests.yml

ShadowJonathan · 2021-03-22T09:24:14Z

For some reason, this test flaked, but it didn't mark the job as failed.

callahad · 2021-03-22T09:29:15Z

Oh weird. I thought that was failing properly before I converted that whole job to run inside the sytest container... will poke at it shortly.

callahad · 2021-03-22T09:30:31Z

(Or if you want to try to fix it yourself, turn on Actions in your fork of this repo, push up a branch to your fork, and you should be able to run and iterate on the tests to your heart's desire!)

ShadowJonathan · 2021-03-22T09:50:16Z

Thanks, I'll try to see if i have time this week, i just left behind a few comments on some things i noticed while glancing :P

callahad · 2021-03-22T09:56:15Z

Ooh, I misunderstood step.continue-on-failure. What I really wanted was strategy.fail-fast

Signed-off-by: Dan Callahan <[email protected]>

callahad · 2021-04-01T15:08:13Z

I believe this is ready to review.

Deviations from our buildkite pipeline:

I'm not excluding dinsic branches from tox and mypy tests... if we still want that we can add it back in after merging the broader PR, but I wanted to keep the logic as clear as possible initially.
I only check for a newsfile for PRs into develop, which I think is semantically correct. Happy to broaden it back up to checking for all pull requests, though. (Or I guess we also need to check for PRs going into release branches, right? Any other targets we might need?)
I'm not uploading _trial_temp/*/*.log from Trial tests -- do we need these, beyond what's already in stdout?
I'm not uploading [ "logs/**/*.log", "logs/**/*.log.*", "logs/results.tap" ] from SyTest -- do we need these, beyond what's already in stdout?

babolivier · 2021-04-01T16:15:00Z

I'm not excluding dinsic branches from tox and mypy tests... if we still want that we can add it back in after merging the broader PR, but I wanted to keep the logic as clear as possible initially.

This shouldn't be an issue since this repo's dinsic branch should be considered dead now given it's been replaced by the fork over at https://github.com/matrix-org/synapse-dinsic - this was done after our move to buildkite hence why it still has an exception for it.

clokep · 2021-04-01T16:48:42Z

I only check for a newsfile for PRs into develop, which I think is semantically correct. Happy to broaden it back up to checking for all pull requests, though. (Or I guess we also need to check for PRs going into release branches, right? Any other targets we might need?)

I suspect checking for the release branches also makes sense.

I'm not uploading _trial_temp/*/*.log from Trial tests -- do we need these, beyond what's already in stdout?

These are the server logs of why a test fails. I usually run the test locally, but sometimes they're useful if something only fails in CI (e.g. a Python or platform difference).

I'm not uploading [ "logs/**/*.log", "logs/**/*.log.*", "logs/results.tap" ] from SyTest -- do we need these, beyond what's already in stdout?

What's on standard out for sytest isn't very useful. You really need the results.tap file. If something is failing, the log files are useful to see what the homeserver was doing.

.github/workflows/tests.yml

Signed-off-by: Dan Callahan <[email protected]>

richvdh · 2021-04-06T14:35:30Z

this looks like it could do with a merge from develop to fix buildkite CI, incidentally.

(on which note: isn't the fact it is failing in buildkite but not GHA a red flag?)

callahad · 2021-04-06T14:48:33Z

this looks like it could do with a merge from develop to fix buildkite CI, incidentally.

(on which note: isn't the fact it is failing in buildkite but not GHA a red flag?)

That does strike me as a red flag, but with Buildkite not GitHub Actions: This pull request targets develop already. If manually merging develop into this branch would cause the Buildkite tests to pass, then it indicates that Buildkite isn't testing the result of the proposed merge, right?

Dumping them to stdout slowed GitHub Actions' log viewer to a crawl Signed-off-by: Dan Callahan <[email protected]>

ShadowJonathan · 2021-04-06T15:08:06Z

@richvdh GHA, with merges, acts as if the merge has taken place before running the CI pipeline, this way you get an accurate representation of how the changes can affect develop.

(This is also why bors was mentioned by @callahad, as that tests continually while the target branch is updated, so a CI result from a week ago isn't "outdated" against the target branch (though correct me if I'm wrong on that))

.github/workflows/tests.yml

richvdh · 2021-04-06T15:15:29Z

That does strike me as a red flag, but with Buildkite not GitHub Actions: This pull request targets develop already. If manually merging develop into this branch would cause the Buildkite tests to pass, then it indicates that Buildkite isn't testing the result of the proposed merge, right?

Ah fair enough. The BK pipeline doesn't do the merge_base_branch for the lint steps.

richvdh · 2021-04-06T15:15:51Z

generally this looks sensible to me.

Signed-off-by: Dan Callahan <[email protected]>

callahad · 2021-04-06T15:20:20Z

I may be off my rocker re: branches / merges / etc. GHA in my fork is now failing along the same lines as BuildKite. Will look more closely tonight.

Signed-off-by: Dan Callahan <[email protected]>

callahad · 2021-04-06T16:45:47Z

Alright, I think review concerns have been addressed.

Open question: given how GitHub Actions treats artifacts (one big namespace per pipeline run), should we only upload SyTest logs when it fails? It'd make it easier to pick out the ones you want from this list:

Alternatively: I could name the archives to include the result status ("SyTest Logs PASS ..." or "SyTest Logs FAIL ...").

Unfortunately, I don't think there's a good way to make these viewable directly in the browser: they're just too big to stream into GHA's normal logging, and uploaded artifacts are implicitly wrapped in a zip container.

ShadowJonathan · 2021-04-06T18:57:58Z

One vague suggestion i might have is to take those logs files as input for another pipeline step (if they failed) and extract some easily-readable information from there in console-readable format, or a shorter file that gives an analysis of what went wrong, something similar to what we're currently doing at the "top" of buildkite's build pages.

Signed-off-by: Dan Callahan <[email protected]>

callahad · 2021-04-07T21:52:01Z

One vague suggestion i might have is to take those logs files as input for another pipeline step (if they failed) and extract some easily-readable information from there in console-readable format, or a shorter file that gives an analysis of what went wrong, something similar to what we're currently doing at the "top" of buildkite's build pages.

So that's handled by the format_tap.pl script in SyTest which gets run on results.tap upon failure. The result is passed to our annotate-buildkite-plugin... we could probably do similar things easily enough in GHA, but probably best to not block on it.

callahad · 2021-04-07T22:14:30Z

Reviewers: I think this is good to go

Signed-off-by: Dan Callahan <[email protected]>

callahad · 2021-04-08T15:01:53Z

trial-olddeps is failing pending updates to our test_old_deps.sh and tox.ini in #9766

Otherwise this should be good to go

callahad · 2021-04-08T15:16:56Z

Yep, everything else is green

richvdh

lgtm, assuming that CI starts passing once we get our other ducks (#9769?) in a row.

@dklimpel

Synapse 1.32.2 (2021-04-22) =========================== This release includes a fix for a regression introduced in 1.32.0. Bugfixes -------- - Fix a regression in Synapse 1.32.0 and 1.32.1 which caused `LoggingContext` errors in plugins. ([\#9857](matrix-org/synapse#9857)) Synapse 1.32.1 (2021-04-21) =========================== This release fixes [a regression](matrix-org/synapse#9853) in Synapse 1.32.0 that caused connected Prometheus instances to become unstable. However, as this release is still subject to the `LoggingContext` change in 1.32.0, it is recommended to remain on or downgrade to 1.31.0. Bugfixes -------- - Fix a regression in Synapse 1.32.0 which caused Synapse to report large numbers of Prometheus time series, potentially overwhelming Prometheus instances. ([\#9854](matrix-org/synapse#9854)) Synapse 1.32.0 (2021-04-20) =========================== **Note:** This release introduces [a regression](matrix-org/synapse#9853) that can overwhelm connected Prometheus instances. This issue was not present in 1.32.0rc1. If affected, it is recommended to downgrade to 1.31.0 in the meantime, and follow [these instructions](matrix-org/synapse#9854 (comment)) to clean up any excess writeahead logs. **Note:** This release also mistakenly included a change that may affected Synapse modules that import `synapse.logging.context.LoggingContext`, such as [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider). This will be fixed in a later Synapse version. **Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+. This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities. This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date. If you are using the `packages.matrix.org` Debian repository for Synapse packages, note that we have recently updated the expiry date on the gpg signing key. If you see an error similar to `The following signatures were invalid: EXPKEYSIG F473DD4473365DE1`, you will need to get a fresh copy of the keys. You can do so with: ```sh sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg ``` Bugfixes -------- - Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](matrix-org/synapse#9829)) Synapse 1.32.0rc1 (2021-04-13) ============================== Features -------- - Add a Synapse module for routing presence updates between users. ([\#9491](matrix-org/synapse#9491)) - Add an admin API to manage ratelimit for a specific user. ([\#9648](matrix-org/synapse#9648)) - Include request information in structured logging output. ([\#9654](matrix-org/synapse#9654)) - Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](matrix-org/synapse#9691)) - Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](matrix-org/synapse#9700)) - Add experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9717](matrix-org/synapse#9717), [\#9735](matrix-org/synapse#9735)) - Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](matrix-org/synapse#9710)) - Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](matrix-org/synapse#9766)) Bugfixes -------- - Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](matrix-org/synapse#8926)) - Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](matrix-org/synapse#9711)) - Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](matrix-org/synapse#9725)) - Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](matrix-org/synapse#9770)) - Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](matrix-org/synapse#9780)) Updates to the Docker image --------------------------- - Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](matrix-org/synapse#9765)) Improved Documentation ---------------------- - Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](matrix-org/synapse#9719)) Deprecations and Removals ------------------------- - Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](matrix-org/synapse#9401)) - Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](matrix-org/synapse#9548)) Internal Changes ---------------- - Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](matrix-org/synapse#9718)) - Experiment with GitHub Actions for CI. ([\#9661](matrix-org/synapse#9661)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](matrix-org/synapse#9682)) - Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](matrix-org/synapse#9685)) - Improve Jaeger tracing for `to_device` messages. ([\#9686](matrix-org/synapse#9686)) - Add release helper script for automating part of the Synapse release process. ([\#9713](matrix-org/synapse#9713)) - Add type hints to expiring cache. ([\#9730](matrix-org/synapse#9730)) - Convert various testcases to `HomeserverTestCase`. ([\#9736](matrix-org/synapse#9736)) - Start linting mypy with `no_implicit_optional`. ([\#9742](matrix-org/synapse#9742)) - Add missing type hints to federation handler and server. ([\#9743](matrix-org/synapse#9743)) - Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](matrix-org/synapse#9753)) - Fix incompatibility with `tox` 2.5. ([\#9769](matrix-org/synapse#9769)) - Enable Complement tests for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary API. ([\#9771](matrix-org/synapse#9771)) - Use mock from the standard library instead of a separate package. ([\#9772](matrix-org/synapse#9772)) - Update Black configuration to target Python 3.6. ([\#9781](matrix-org/synapse#9781)) - Add option to skip unit tests when building Debian packages. ([\#9793](matrix-org/synapse#9793)) Synapse 1.31.0 (2021-04-06) =========================== **Note:** As announced in v1.25.0, and in line with the deprecation policy for platform dependencies, this is the last release to support Python 3.5 and PostgreSQL 9.5. Future versions of Synapse will require Python 3.6+ and PostgreSQL 9.6+, as per our [deprecation policy](docs/deprecation_policy.md). This is also the last release that the Synapse team will be publishing packages for Debian Stretch and Ubuntu Xenial. Improved Documentation ---------------------- - Add a document describing the deprecation policy for platform dependencies. ([\#9723](matrix-org/synapse#9723)) Internal Changes ---------------- - Revert using `dmypy run` in lint script. ([\#9720](matrix-org/synapse#9720)) - Pin flake8-bugbear's version. ([\#9734](matrix-org/synapse#9734)) Synapse 1.31.0rc1 (2021-03-30) ============================== Features -------- - Add support to OpenID Connect login for requiring attributes on the `userinfo` response. Contributed by Hubbe King. ([\#9609](matrix-org/synapse#9609)) - Add initial experimental support for a "space summary" API. ([\#9643](matrix-org/synapse#9643), [\#9652](matrix-org/synapse#9652), [\#9653](matrix-org/synapse#9653)) - Add support for the busy presence state as described in [MSC3026](matrix-org/matrix-spec-proposals#3026). ([\#9644](matrix-org/synapse#9644)) - Add support for credentials for proxy authentication in the `HTTPS_PROXY` environment variable. ([\#9657](matrix-org/synapse#9657)) Bugfixes -------- - Fix a longstanding bug that could cause issues when editing a reply to a message. ([\#9585](matrix-org/synapse#9585)) - Fix the `/capabilities` endpoint to return `m.change_password` as disabled if the local password database is not used for authentication. Contributed by @dklimpel. ([\#9588](matrix-org/synapse#9588)) - Check if local passwords are enabled before setting them for the user. ([\#9636](matrix-org/synapse#9636)) - Fix a bug where federation sending can stall due to `concurrent access` database exceptions when it falls behind. ([\#9639](matrix-org/synapse#9639)) - Fix a bug introduced in Synapse 1.30.1 which meant the suggested `pip` incantation to install an updated `cryptography` was incorrect. ([\#9699](matrix-org/synapse#9699)) Updates to the Docker image --------------------------- - Speed up Docker builds and make it nicer to test against Complement while developing (install all dependencies before copying the project). ([\#9610](matrix-org/synapse#9610)) - Include [opencontainers labels](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) in the Docker image. ([\#9612](matrix-org/synapse#9612)) Improved Documentation ---------------------- - Clarify that `register_new_matrix_user` is present also when installed via non-pip package. ([\#9074](matrix-org/synapse#9074)) - Update source install documentation to mention platform prerequisites before the source install steps. ([\#9667](matrix-org/synapse#9667)) - Improve worker documentation for fallback/web auth endpoints. ([\#9679](matrix-org/synapse#9679)) - Update the sample configuration for OIDC authentication. ([\#9695](matrix-org/synapse#9695)) Internal Changes ---------------- - Preparatory steps for removing redundant `outlier` data from `event_json.internal_metadata` column. ([\#9411](matrix-org/synapse#9411)) - Add type hints to the caching module. ([\#9442](matrix-org/synapse#9442)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9499](matrix-org/synapse#9499), [\#9659](matrix-org/synapse#9659)) - Add additional type hints to the Homeserver object. ([\#9631](matrix-org/synapse#9631), [\#9638](matrix-org/synapse#9638), [\#9675](matrix-org/synapse#9675), [\#9681](matrix-org/synapse#9681)) - Only save remote cross-signing and device keys if they're different from the current ones. ([\#9634](matrix-org/synapse#9634)) - Rename storage function to fix spelling and not conflict with another function's name. ([\#9637](matrix-org/synapse#9637)) - Improve performance of federation catch up by sending the latest events in the room to the remote, rather than just the last event sent by the local server. ([\#9640](matrix-org/synapse#9640), [\#9664](matrix-org/synapse#9664)) - In the `federation_client` commandline client, stop automatically adding the URL prefix, so that servlets on other prefixes can be tested. ([\#9645](matrix-org/synapse#9645)) - In the `federation_client` commandline client, handle inline `signing_key`s in `homeserver.yaml`. ([\#9647](matrix-org/synapse#9647)) - Fixed some antipattern issues to improve code quality. ([\#9649](matrix-org/synapse#9649)) - Add a storage method for pulling all current user presence state from the database. ([\#9650](matrix-org/synapse#9650)) - Import `HomeServer` from the proper module. ([\#9665](matrix-org/synapse#9665)) - Increase default join ratelimiting burst rate. ([\#9674](matrix-org/synapse#9674)) - Add type hints to third party event rules and visibility modules. ([\#9676](matrix-org/synapse#9676)) - Bump mypy-zope to 0.2.13 to fix "Cannot determine consistent method resolution order (MRO)" errors when running mypy a second time. ([\#9678](matrix-org/synapse#9678)) - Use interpreter from `$PATH` via `/usr/bin/env` instead of absolute paths in various scripts. ([\#9689](matrix-org/synapse#9689)) - Make it possible to use `dmypy`. ([\#9692](matrix-org/synapse#9692)) - Suppress "CryptographyDeprecationWarning: int_from_bytes is deprecated". ([\#9698](matrix-org/synapse#9698)) - Use `dmypy run` in lint script for improved performance in type-checking while developing. ([\#9701](matrix-org/synapse#9701)) - Fix undetected mypy error when using Python 3.6. ([\#9703](matrix-org/synapse#9703)) - Fix type-checking CI on develop. ([\#9709](matrix-org/synapse#9709)) Synapse 1.30.1 (2021-03-26) =========================== This release is identical to Synapse 1.30.0, with the exception of explicitly setting a minimum version of Python's Cryptography library to ensure that users of Synapse are protected from the recent [OpenSSL security advisories](https://mta.openssl.org/pipermail/openssl-announce/2021-March/000198.html), especially CVE-2021-3449. Note that Cryptography defaults to bundling its own statically linked copy of OpenSSL, which means that you may not be protected by your operating system's security updates. It's also worth noting that Cryptography no longer supports Python 3.5, so admins deploying to older environments may not be protected against this or future vulnerabilities. Synapse will be dropping support for Python 3.5 at the end of March. Updates to the Docker image --------------------------- - Ensure that the docker container has up to date versions of openssl. ([\#9697](matrix-org/synapse#9697)) Internal Changes ---------------- - Enforce that `cryptography` dependency is up to date to ensure it has the most recent openssl patches. ([\#9697](matrix-org/synapse#9697)) Synapse 1.30.0 (2021-03-22) =========================== Note that this release deprecates the ability for appservices to call `POST /_matrix/client/r0/register` without the body parameter `type`. Appservice developers should use a `type` value of `m.login.application_service` as per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions). In future releases, calling this endpoint with an access token - but without a `m.login.application_service` type - will fail. No significant changes. Synapse 1.30.0rc1 (2021-03-16) ============================== Features -------- - Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](matrix-org/synapse#9510), [\#9511](matrix-org/synapse#9511), [\#9573](matrix-org/synapse#9573)) - Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](matrix-org/synapse#9540)) - Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](matrix-org/synapse#9549)) - Optimise handling of incomplete room history for incoming federation. ([\#9601](matrix-org/synapse#9601)) - Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9617](matrix-org/synapse#9617)) - Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](matrix-org/synapse#9626)) Bugfixes -------- - Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](matrix-org/synapse#9473)) - Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](matrix-org/synapse#9542), [\#9583](matrix-org/synapse#9583)) - Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](matrix-org/synapse#9567)) - Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](matrix-org/synapse#9587)) - Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](matrix-org/synapse#9597)) - Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](matrix-org/synapse#9620)) - Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](matrix-org/synapse#9623)) Updates to the Docker image --------------------------- - Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](matrix-org/synapse#8553)) Improved Documentation ---------------------- - Add relayd entry to reverse proxy example configurations. ([\#9508](matrix-org/synapse#9508)) - Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](matrix-org/synapse#9550)) - Link to the "List user's media" admin API from the media admin API docs. ([\#9571](matrix-org/synapse#9571)) - Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](matrix-org/synapse#9580)) - Clarify the sample configuration for `stats` settings. ([\#9604](matrix-org/synapse#9604)) Deprecations and Removals ------------------------- - The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](matrix-org/synapse#9540)) - Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](matrix-org/synapse#9559)) Internal Changes ---------------- - Add tests to ResponseCache. ([\#9458](matrix-org/synapse#9458)) - Add type hints to purge room and server notice admin API. ([\#9520](matrix-org/synapse#9520)) - Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](matrix-org/synapse#9523)) - Fix incorrect type hints. ([\#9528](matrix-org/synapse#9528), [\#9543](matrix-org/synapse#9543), [\#9591](matrix-org/synapse#9591), [\#9608](matrix-org/synapse#9608), [\#9618](matrix-org/synapse#9618)) - Add an additional test for purging a room. ([\#9541](matrix-org/synapse#9541)) - Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](matrix-org/synapse#9560)) - Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](matrix-org/synapse#9561)) - Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](matrix-org/synapse#9562)) - Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](matrix-org/synapse#9563)) - Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](matrix-org/synapse#9568)) - Improve efficiency of calculating the auth chain in large rooms. ([\#9576](matrix-org/synapse#9576)) - Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](matrix-org/synapse#9586)) - Add logging for redis connection setup. ([\#9590](matrix-org/synapse#9590)) - Improve logging when processing incoming transactions. ([\#9596](matrix-org/synapse#9596)) - Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](matrix-org/synapse#9604)) - Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](matrix-org/synapse#9619))

@dklimpel

Synapse 1.32.0 (2021-04-20) =========================== **Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+. This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities. This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date. Bugfixes -------- - Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](matrix-org/synapse#9829)) Synapse 1.32.0rc1 (2021-04-13) ============================== Features -------- - Add a Synapse module for routing presence updates between users. ([\#9491](matrix-org/synapse#9491)) - Add an admin API to manage ratelimit for a specific user. ([\#9648](matrix-org/synapse#9648)) - Include request information in structured logging output. ([\#9654](matrix-org/synapse#9654)) - Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](matrix-org/synapse#9691)) - Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](matrix-org/synapse#9700)) - Add experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9717](matrix-org/synapse#9717), [\#9735](matrix-org/synapse#9735)) - Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](matrix-org/synapse#9710)) - Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](matrix-org/synapse#9766)) Bugfixes -------- - Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](matrix-org/synapse#8926)) - Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](matrix-org/synapse#9711)) - Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](matrix-org/synapse#9725)) - Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](matrix-org/synapse#9770)) - Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](matrix-org/synapse#9780)) Updates to the Docker image --------------------------- - Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](matrix-org/synapse#9765)) Improved Documentation ---------------------- - Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](matrix-org/synapse#9719)) Deprecations and Removals ------------------------- - Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](matrix-org/synapse#9401)) - Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](matrix-org/synapse#9548)) Internal Changes ---------------- - Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](matrix-org/synapse#9718)) - Experiment with GitHub Actions for CI. ([\#9661](matrix-org/synapse#9661)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](matrix-org/synapse#9682)) - Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](matrix-org/synapse#9685)) - Improve Jaeger tracing for `to_device` messages. ([\#9686](matrix-org/synapse#9686)) - Add release helper script for automating part of the Synapse release process. ([\#9713](matrix-org/synapse#9713)) - Add type hints to expiring cache. ([\#9730](matrix-org/synapse#9730)) - Convert various testcases to `HomeserverTestCase`. ([\#9736](matrix-org/synapse#9736)) - Start linting mypy with `no_implicit_optional`. ([\#9742](matrix-org/synapse#9742)) - Add missing type hints to federation handler and server. ([\#9743](matrix-org/synapse#9743)) - Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](matrix-org/synapse#9753)) - Fix incompatibility with `tox` 2.5. ([\#9769](matrix-org/synapse#9769)) - Enable Complement tests for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary API. ([\#9771](matrix-org/synapse#9771)) - Use mock from the standard library instead of a separate package. ([\#9772](matrix-org/synapse#9772)) - Update Black configuration to target Python 3.6. ([\#9781](matrix-org/synapse#9781)) - Add option to skip unit tests when building Debian packages. ([\#9793](matrix-org/synapse#9793))

callahad mentioned this pull request Mar 22, 2021

Proof of concept for GitHub Actions #9660

Closed

4 tasks

Proof of concept for GitHub Actions

9572716

Signed-off-by: Dan Callahan <[email protected]>

callahad force-pushed the callahad/actions branch from 61915e8 to 9572716 Compare March 22, 2021 08:55

ShadowJonathan reviewed Mar 22, 2021

View reviewed changes

Fixup SyTest behavior on failure

4561f18

Signed-off-by: Dan Callahan <[email protected]>

callahad requested a review from a team April 1, 2021 15:08

ShadowJonathan reviewed Apr 6, 2021

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

ShadowJonathan reviewed Apr 6, 2021

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

Reflow yaml

6a0951b

callahad force-pushed the callahad/actions branch from 2b20fc4 to 6a0951b Compare April 6, 2021 10:10

callahad added 3 commits April 6, 2021 11:38

Address review feedback

e955013

Signed-off-by: Dan Callahan <[email protected]>

Fixup log dumping

54a1e14

Signed-off-by: Dan Callahan <[email protected]>

Fixup log dumping for Sytest

ba038bb

Signed-off-by: Dan Callahan <[email protected]>

Upload Sytest logs as artifacts

532b6c4

Dumping them to stdout slowed GitHub Actions' log viewer to a crawl Signed-off-by: Dan Callahan <[email protected]>

richvdh reviewed Apr 6, 2021

View reviewed changes

callahad added 2 commits April 6, 2021 16:17

Temporarily always run SyTest

6666985

Signed-off-by: Dan Callahan <[email protected]>

s/uploat/upload/

574da27

Signed-off-by: Dan Callahan <[email protected]>

callahad added 4 commits April 6, 2021 16:27

...fix Sytest invocation

f6b43aa

Signed-off-by: Dan Callahan <[email protected]>

Fix newsfile linting conditions

a20e12b

Signed-off-by: Dan Callahan <[email protected]>

Correct test ordering

49686c4

Signed-off-by: Dan Callahan <[email protected]>

Merge remote-tracking branch 'origin/develop' into callahad/actions

30f5708

Address review comments

9a26295

Signed-off-by: Dan Callahan <[email protected]>

callahad force-pushed the callahad/actions branch from 665e8cc to 9a26295 Compare April 7, 2021 21:51

callahad requested a review from a team April 7, 2021 22:14

Drop Py35

368e20b

Signed-off-by: Dan Callahan <[email protected]>

richvdh approved these changes Apr 8, 2021

View reviewed changes

Merge remote-tracking branch 'origin/develop' into callahad/actions

92174d0

callahad merged commit 0277b8f into develop Apr 9, 2021

callahad deleted the callahad/actions branch April 9, 2021 09:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proof of concept for GitHub Actions #9661

Proof of concept for GitHub Actions #9661

callahad commented Mar 22, 2021 •

edited

Loading

ShadowJonathan Mar 22, 2021

callahad Mar 22, 2021

ShadowJonathan Mar 22, 2021

ShadowJonathan commented Mar 22, 2021

callahad commented Mar 22, 2021 •

edited

Loading

callahad commented Mar 22, 2021

ShadowJonathan commented Mar 22, 2021

callahad commented Mar 22, 2021

callahad commented Apr 1, 2021 •

edited

Loading

babolivier commented Apr 1, 2021 •

edited

Loading

clokep commented Apr 1, 2021

richvdh commented Apr 6, 2021

callahad commented Apr 6, 2021

ShadowJonathan commented Apr 6, 2021 •

edited

Loading

richvdh commented Apr 6, 2021

richvdh commented Apr 6, 2021

callahad commented Apr 6, 2021

callahad commented Apr 6, 2021

ShadowJonathan commented Apr 6, 2021

callahad commented Apr 7, 2021

callahad commented Apr 7, 2021

callahad commented Apr 8, 2021

callahad commented Apr 8, 2021

richvdh left a comment

Proof of concept for GitHub Actions #9661

Proof of concept for GitHub Actions #9661

Conversation

callahad commented Mar 22, 2021 • edited Loading

Pull Request Checklist

ShadowJonathan Mar 22, 2021

Choose a reason for hiding this comment

callahad Mar 22, 2021

Choose a reason for hiding this comment

ShadowJonathan Mar 22, 2021

Choose a reason for hiding this comment

ShadowJonathan commented Mar 22, 2021

callahad commented Mar 22, 2021 • edited Loading

callahad commented Mar 22, 2021

ShadowJonathan commented Mar 22, 2021

callahad commented Mar 22, 2021

callahad commented Apr 1, 2021 • edited Loading

babolivier commented Apr 1, 2021 • edited Loading

clokep commented Apr 1, 2021

richvdh commented Apr 6, 2021

callahad commented Apr 6, 2021

ShadowJonathan commented Apr 6, 2021 • edited Loading

richvdh commented Apr 6, 2021

richvdh commented Apr 6, 2021

callahad commented Apr 6, 2021

callahad commented Apr 6, 2021

ShadowJonathan commented Apr 6, 2021

callahad commented Apr 7, 2021

callahad commented Apr 7, 2021

callahad commented Apr 8, 2021

callahad commented Apr 8, 2021

richvdh left a comment

Choose a reason for hiding this comment

callahad commented Mar 22, 2021 •

edited

Loading

callahad commented Mar 22, 2021 •

edited

Loading

callahad commented Apr 1, 2021 •

edited

Loading

babolivier commented Apr 1, 2021 •

edited

Loading

ShadowJonathan commented Apr 6, 2021 •

edited

Loading