data store: blocked tasks outside of n=1 are missing from the store

[oliver-sanders]

This applies to Cylc 7 compat mode, regular Cylc 8 workflows are protected from this by required outputs.

Explanation

Unsatisfiable tasks will become invisible in the UIs if they drift outside of the N=1 window.

• For regular Cylc 8 workflows the cause of the problem is an incomplete task which will stay in the N=0 window (I think) preventing blocked downstream tasks from leaving the N=1 window. Consequently the blocked tasks remain visible.
• However, in Cylc 7 compat mode we don't have required outputs to save us so if the blocked task(s) drift outside the N=1 window they become invisible to the user making it hard to work out what's going wrong.

Example

This example is missing the suicide trigger which would remove the recover task:

# suite.rc
[scheduler]
    allow implicit tasks = True

[scheduling]
    initial cycle point = 1
    cycling mode = integer
    [[graph]]
        P1 = """
            foo:fail => recover
            foo | recover => bar
        """

When we run this workflow the first 5 cycles run as normal, the sixth cycle is runahead limited.

The blocked "recover" tasks from the first five cycles are not in the N=1 window (note no inter-cycle dependencies here) so disappear from the UIs:

foo - running
TUI is experimental and may break with large flows

- foo
   - ̊○ 6
        ○ bar      # ideally we would mark this task as runahead limited too?
        ̊○ foo
        ○ recover  # ideally we would mark this task as runahead limited too?

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

[hjoliver]

Somewhat off-topic, but regarding this:

"ideally we would mark this task as runahead limited too"

What you're suggesting amounts to marking all future tasks as runahead limited, right?

The status quo seems pretty sensible to me and (I would hope) not difficult for users to understand:

A task is runahead limited if it is ready to run, but held back by the runahead limit.

That usefully distinguishes foo from bar and recover in your example above, in that foo is ready to run as soon as the runahead limit moves on, but the other tasks are not.

[oliver-sanders]

Ok, if that's intended, yes totally off topic then.

[oliver-sanders]

(IMO this is not release blocking so have listed against 8.0.1)

[oliver-sanders]

@hjoliver I think your cylc set PR might close this?

[dwsutherland]

wouldn't it be the case of keeping them active so they stay in N=0?

[oliver-sanders]

The cylc set PR removes the hidden pool which should make them N=0 I think.

[oliver-sanders]

At some point along the road, this issue was fixed by some other change.

The n=1 window now contains the recover tasks (which were previously invisible), however, I'm not sure bar should be appearing?

      - ○ 1
         + ● ■ bar
         + ● ■ foo
           ○ recover
      - ○ 2
         + ● ■ bar
         + ● ■ foo
           ○ recover
      - ○ 3
         + ● ■ bar
         + ● ■ foo
           ○ recover
      - ○ 4
         + ● ■ bar
         + ● ■ foo
           ○ recover
      - ○ 5
         + ● ■ bar
         + ● ■ foo
           ○ recover
      - ̊○ 6
           ○ bar
           ̊○ foo
           ○ recover