We need a way to wake up an AppService push task manually

[Yoric]

Description:

Current scenario:

1. I'm starting a Synapse server with an AppService.
2. For some reason, the AppService goes down, maybe for 24h.
3. Synapse goes into exponential backoff and retry. I don't know if there is a limit to this backoff delay, but I've witnessed it growing pretty large. For this example, I'm going to assume that it can grow beyond 48h.
4. When we restart the AppService, it takes 48h to find out whether things work correctly.

I'd like the ability to replace 4. with

4. Use some (presumably admin) command to get Synapse to resume AppService immediately, as if nothing had happened.

As discussed over #synapse-dev, there are also use cases for CI testing an AppService, as CI tests can be very sensitive to timeouts.

[reivilibre]

Note that the maximum backoff is 8.5 minutes

synapse/synapse/appservice/scheduler.py

Lines 490 to 493 in d736d5c

	# cap the backoff to be around 8.5min => (2^9) = 512 secs
	if self.backoff_counter < 9:
	self.backoff_counter += 1
	self.recover()

so whilst it might still be nice to have a way to wake the AS sender when the AS pokes the homeserver, I wouldn't worry about outages on the order of 48 hours or even an hour :)

The equivalent approach we take with federation is to clear the backoff when we receive anything from the remote homeserver. We might consider something like that for application services as well; e.g. when the AS makes a request to the homeserver we clear the backoff for that appservice.

[DMRobertson]

Sounds like the same kind of problem that #12685 wanted to address?

[Yoric]

so whilst it might still be nice to have a way to wake the AS sender when the AS pokes the homeserver, I wouldn't worry about outages on the order of 48 hours or even an hour :)

That's much better. Still a problem in both CI and production, though.

The equivalent approach we take with federation is to clear the backoff when we receive anything from the remote homeserver. We might consider something like that for application services as well; e.g. when the AS makes a request to the homeserver we clear the backoff for that appservice.

That sounds like the ideal solution, yes.

[H-Shay]

I think this is a dup of #1334 but I am closing that in favor of this as this has more info/is better structured.

[gabrc52]

If the application service cannot be reached, the homeserver SHOULD backoff exponentially until the application service is reachable again.

This is also useful if you are building an application service yourself. When it's still not working, it's not going to be reachable or it might crash. So exponential backoff, while a reasonable part of the spec, can be annoying for people wanting to write Matrix integrations or bridges, and would be nice to disable.