Reload broken in compatibility mode

[dpmatthews]

With this suite.rc:

[scheduling]
    [[graph]]
        R1 = "foo => bar"
[runtime]
    [[foo]]
        script = "echo Hello; false"
    [[bar]]

Play the workflow and let it stall.
Then reload.
Then trigger foo.
I get the following in the scheduler log:

2022-08-25T18:57:43+01:00 INFO - Processing 1 queued command(s)
2022-08-25T18:57:43+01:00 INFO - [1/foo failed job:01 flows:1] => waiting
2022-08-25T18:57:43+01:00 INFO - Command succeeded: force_trigger_tasks(['1/foo'], flow=['all'], flow_wait=False, flow_descr=None)
2022-08-25T18:57:43+01:00 INFO - [1/foo waiting job:02 flows:1] => preparing
2022-08-25T18:57:43+01:00 INFO - [1/foo preparing job:02 flows:1] host=XXX
2022-08-25T18:57:43+01:00 WARNING - stall timer stopped
2022-08-25T18:57:44+01:00 WARNING - Unhandled jobs-submit output: 2022-08-25T18:57:44+01:00|1/foo/02|0|78545
2022-08-25T18:57:44+01:00 WARNING - ('1', 'foo', '02')
2022-08-25T18:57:44+01:00 WARNING - Unhandled jobs-submit output: 2022-08-25T18:57:44+01:00|1/foo/02|[STDOUT] 78545
2022-08-25T18:57:44+01:00 WARNING - ('1', 'foo', '02')
2022-08-25T18:57:44+01:00 ERROR - [jobs-submit cmd] cylc jobs-submit --clean-env --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/test/reload/run1/log/job' 1/foo/02
    [jobs-submit ret_code] 1
    [jobs-submit out] 2022-08-25T18:57:44+01:00|1/foo/01|1
2022-08-25T18:57:44+01:00 INFO - [1/foo failed job:01 flows:1] (internal)submission failed at 2022-08-25T18:57:44+01:00
2022-08-25T18:57:44+01:00 CRITICAL - [1/foo failed job:01 flows:1] submission failed
2022-08-25T18:57:44+01:00 INFO - [1/foo failed job:01 flows:1] => submit-failed
2022-08-25T18:57:46+01:00 INFO - [1/foo preparing job:02 flows:1] (received)started at 2022-08-25T18:57:45+01:00

Something is going badly wrong with the the job submission.
Despite the warnings, job 02 does get submitted and runs.
However, if I modify the task config and reinstall before doing the reload, job 02 does not use the updated config so it's very broken.

The problem does not occur if I:

1. rename suite.rc to flow.cylc
2. Change the graph to R1 = "foo"

[hjoliver]

It looks like we're ending up with two instances of the task proxy - job:01 and job:02 ... [sort of]

[hjoliver]

Tentative fix coming.

[oliver-sanders]

It looks like we're ending up with two instances of the task proxy - job:01 and job:02 ... [sort of]

That's exactly the same cause of: #4974

This reload mechanism is quite problematic. Longer term I've suggested not replacing the TaskProxy instance, just the TaskDef it uses.