Fix spawning of mixed-parentage tasks. #4906

hjoliver · 2022-06-09T07:10:11Z

(Those that have parents in some cycles but not in others).

Before spawn-on-demand, all tasks depended implicitly on their own previous instance submitting (same task at previous cycle point), which had several unpleasant consequences such as the inability of same-task instances to run out of order.

This fixes the last vestige of previous-instance dependence in Cylc 8: if a task instance has no parents to spawn it, we spawn it automatically when its previous instance is released from runahead. This (correctly) makes parentless tasks spawn out to the runahead limit immediately, but only if they have no parents in all cycle points. If they have parents at some points, those points temporarily block further spawning (because the parented instance does not get released from runahead until after it gets spawned by its parent).

This PR also tidies spawning-related code a lot (it was really hard to follow after my incremental shoe-horning of SOD into the old codebase ~2 yrs ago 🤦 :), and computes the runahead limit much less often.

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 10
    [[xtriggers]]
        never = xrandom("0")  # never satisfied
    [[graph]]
        # foo has no parents, should spawn to runahead limit
        P1 = "foo => bar"
        # except at point 3, where it should be spawned by baz
        R1/3 = "@never => baz => foo"
[runtime]
    [[foo, bar, baz]]
        script = sleep 10

On master, foo should auto-spawn to the runahead limit except for point 3 where it will be spawned by 3/baz, but spawning is blocked at point 3:

On this branch:

Also: close #4913

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg and conda-environment.yml.
Appropriate tests are included (unit and/or functional).
Appropriate change log entry included.
(master branch) I have opened a documentation PR at Update runahead limit documentation. cylc-doc#479 (for the runahead fix)

cylc/flow/id_match.py

hjoliver · 2022-06-13T02:07:30Z

cylc/flow/rundb.py

@@ -820,7 +820,8 @@ def select_task_pool_for_restart(self, callback):
            LEFT OUTER JOIN
                %(task_outputs)s
            ON  %(task_pool)s.cycle == %(task_outputs)s.cycle AND
-                %(task_pool)s.name == %(task_outputs)s.name
+                %(task_pool)s.name == %(task_outputs)s.name AND
+                %(task_pool)s.flow_nums == %(task_outputs)s.flow_nums


Bug fix! Loading the task pool from the DB at restart, on master I get multiple rows per task proxy if a waiting task had also run in previous flows (and hence attempt to spawn multiple copies of the same task).

hjoliver · 2022-06-14T23:24:35Z

~~(NB: I've just pushed a commit to fix the runahead off-by-one error #4913; this will require a minor adjustment to a whole bunch of tests... which were all passing)~~ DONE

dwsutherland

Reviewed, and found working (could reproduce the issue and fix) 👍
I reviewed the code, followed most of it...

cylc/flow/scheduler.py

hjoliver · 2022-06-23T03:30:11Z

~~NB: I have one thing to investigate on this branch: it might stop the inactivity timer working properly (if so, should be easy to fix).~~ [done]

oliver-sanders · 2022-06-30T12:47:34Z

@hjoliver What font are you using in those terminal screenshots?

hjoliver · 2022-06-30T21:27:46Z

@hjoliver What font are you using in those terminal screenshots?

Ubuntu Mono Derivative. You like or no like?

hjoliver · 2022-07-01T04:50:09Z

@MetRonnie - this is good to go now.

MetRonnie · 2022-07-01T15:27:41Z

I'm happy to review but will be fairly busy this upcoming week with workshops, so someone might like to beat me to it

hjoliver · 2022-07-03T21:35:56Z

That's fine (I just pinged you as you'd self-assigned as a reviewer above).

MetRonnie

Partial review

cylc/flow/cfgspec/workflow.py

cylc/flow/task_pool.py

cylc/flow/scheduler.py

(Those that have parents in some cycles but not others). Also tidy up task spawning and runahead computation.

Adjust tests accordingly.

Co-authored-by: Ronnie Dutta <[email protected]>

oliver-sanders · 2022-07-11T15:30:54Z

Ubuntu Mono Derivative. You like or no like?

Me likey, is very nice for Tui.

MetRonnie

Halfway through...

cylc/flow/taskdef.py

Co-authored-by: Ronnie Dutta <[email protected]>

CHANGES.md

MetRonnie · 2022-07-15T15:23:27Z

tests/functional/runahead/default-future/flow.cylc

    [[events]]
-        inactivity timeout = PT10S
+        inactivity timeout = PT20S


What's the reason for this change?

I was getting some early aborts due to this, under load, when running many tests at once. Pretty sure not related to this branch.

tests/functional/spawn-on-demand/16-parents-yes-no/flow.cylc

hjoliver · 2022-07-20T05:37:49Z

(All review comments addressed)

hjoliver added bug Something is wrong :( sod-follow-up labels Jun 9, 2022

hjoliver added this to the cylc-8.0rc4 milestone Jun 9, 2022

hjoliver self-assigned this Jun 9, 2022

hjoliver force-pushed the sod-parentless-spawn branch from 83decfb to fd38b0a Compare June 12, 2022 21:27

hjoliver commented Jun 12, 2022

View reviewed changes

cylc/flow/id_match.py Outdated Show resolved Hide resolved

hjoliver commented Jun 13, 2022

View reviewed changes

hjoliver mentioned this pull request Jun 14, 2022

Runahead limit consistency: P1 vs PT1H #4913

Closed

hjoliver force-pushed the sod-parentless-spawn branch 2 times, most recently from 938e4d9 to 4c98f6a Compare June 14, 2022 08:41

hjoliver force-pushed the sod-parentless-spawn branch 2 times, most recently from e41a656 to 5183cdd Compare June 15, 2022 23:09

hjoliver mentioned this pull request Jun 16, 2022

Update runahead limit documentation. cylc/cylc-doc#479

Merged

hjoliver marked this pull request as ready for review June 16, 2022 11:18

hjoliver mentioned this pull request Jun 17, 2022

Cylc 728 (with linter) #4900

Merged

7 tasks

MetRonnie self-requested a review June 21, 2022 10:15

dwsutherland approved these changes Jun 22, 2022

View reviewed changes

cylc/flow/scheduler.py Show resolved Hide resolved

This was referenced Jun 27, 2022

2022 Cylc Meetings cylc/cylc-admin#143

Closed

runahead limit not being cleared #4817

Closed

hjoliver modified the milestones: cylc-8.0rc4, cylc-8.0.0 Jul 5, 2022

MetRonnie reviewed Jul 7, 2022

View reviewed changes

Fix spawning of parent/no-parent tasks.

f4e0bf3

(Those that have parents in some cycles but not others). Also tidy up task spawning and runahead computation.

hjoliver and others added 10 commits July 8, 2022 20:58

Fix runahead off-by-one error.

7eea4bd

Adjust tests accordingly.

Update runahead limit docstring.

2e33afe

Add new func test.

e05721b

Move timer reset.

dd0ebbf

Update change log.

99a6bb3

Address review feedback.

f16b6ff

Nail some TODOs.

ed0f03a

Update cylc/flow/task_pool.py

9505201

Co-authored-by: Ronnie Dutta <[email protected]>

Fix post-rebase problems.

a23528b

Final branch tidy.

965d356

hjoliver force-pushed the sod-parentless-spawn branch from c9e276c to 965d356 Compare July 9, 2022 02:35

hjoliver added 2 commits July 9, 2022 15:54

Reload runhead compute on demand.

c79bb54

Address another TODO.

fa05ab8

MetRonnie reviewed Jul 14, 2022

View reviewed changes

cylc/flow/taskdef.py Outdated Show resolved Hide resolved

cylc/flow/taskdef.py Outdated Show resolved Hide resolved

hjoliver and others added 2 commits July 15, 2022 12:07

Update cylc/flow/taskdef.py [skip ci]

feb4a74

Co-authored-by: Ronnie Dutta <[email protected]>

Update cylc/flow/taskdef.py [skip ci]

3b72a6e

Co-authored-by: Ronnie Dutta <[email protected]>

MetRonnie reviewed Jul 15, 2022

View reviewed changes

hjoliver added 3 commits July 20, 2022 16:13

Update change log.

147ce2b

Remove unused xtrigger in test.

9dee129

Merge branch 'master' into sod-parentless-spawn

2623d35

MetRonnie approved these changes Jul 20, 2022

View reviewed changes

hjoliver merged commit 37ea94b into cylc:master Jul 20, 2022

hjoliver deleted the sod-parentless-spawn branch July 20, 2022 21:56

MetRonnie mentioned this pull request Jul 21, 2022

Parentless tasks try to spawn again after already finishing #5002

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix spawning of mixed-parentage tasks. #4906

Fix spawning of mixed-parentage tasks. #4906

hjoliver commented Jun 9, 2022 •

edited

Loading

hjoliver Jun 13, 2022

hjoliver commented Jun 14, 2022 •

edited

Loading

dwsutherland left a comment

hjoliver commented Jun 23, 2022 •

edited

Loading

oliver-sanders commented Jun 30, 2022

hjoliver commented Jun 30, 2022 •

edited

Loading

hjoliver commented Jul 1, 2022

MetRonnie commented Jul 1, 2022

hjoliver commented Jul 3, 2022

MetRonnie left a comment

oliver-sanders commented Jul 11, 2022 •

edited

Loading

MetRonnie left a comment

MetRonnie Jul 15, 2022

hjoliver Jul 18, 2022

hjoliver commented Jul 20, 2022

Fix spawning of mixed-parentage tasks. #4906

Fix spawning of mixed-parentage tasks. #4906

Conversation

hjoliver commented Jun 9, 2022 • edited Loading

hjoliver Jun 13, 2022

Choose a reason for hiding this comment

hjoliver commented Jun 14, 2022 • edited Loading

dwsutherland left a comment

Choose a reason for hiding this comment

hjoliver commented Jun 23, 2022 • edited Loading

oliver-sanders commented Jun 30, 2022

hjoliver commented Jun 30, 2022 • edited Loading

hjoliver commented Jul 1, 2022

MetRonnie commented Jul 1, 2022

hjoliver commented Jul 3, 2022

MetRonnie left a comment

Choose a reason for hiding this comment

oliver-sanders commented Jul 11, 2022 • edited Loading

MetRonnie left a comment

Choose a reason for hiding this comment

MetRonnie Jul 15, 2022

Choose a reason for hiding this comment

hjoliver Jul 18, 2022

Choose a reason for hiding this comment

hjoliver commented Jul 20, 2022

hjoliver commented Jun 9, 2022 •

edited

Loading

hjoliver commented Jun 14, 2022 •

edited

Loading

hjoliver commented Jun 23, 2022 •

edited

Loading

hjoliver commented Jun 30, 2022 •

edited

Loading

oliver-sanders commented Jul 11, 2022 •

edited

Loading