stuck in preparing state

[oliver-sanders]

Tasks can get "stuck" in the preparing state, seen in the wild, twice, but struggling to reproduce.

Possibly related to reloads, possibly caused by a mismatch of submission number (we increment sub num on submission so a subsequent preparing job will have the same sub num as a previous instance), dunno.

Order of events:

• submission 01 preparing
• submission 01 submit-failed
• reload # no change
• submission 02 preparing
• unhandled jobs-submit output ... same-task/02 ...
• [jobs-submit cmd] ... same-task/02
  [jobs-submit ret_code] 1
  [jobs-submit out] ...|cycle/task/01|1
• submission 01 (polled-ignored) submission failed # i.e. previous job showed up but was ignored

At the end of this, job 02 is still marked as preparing, however, is no longer preparing anything, submission seemingly having failed long ago just after the unhandled jobs-submit output. The task is not killable (because the preparing state is not killable).

[oliver-sanders]

It looks like the cause is a mix up of submit numbers.

When we perform reload we make a copy of the task:

cylc-flow/cylc/flow/task_pool.py

Lines 845 to 848 in 8ff0c80

	new_task = TaskProxy(
	self.config.get_taskdef(itask.tdef.name),
	itask.point, itask.flow_nums, itask.state.status)
	itask.copy_to_reload_successor(new_task)

The copy is then triggered incrementing its submit number. If something out there somewhere is maintaining a reference to the original task, however, that could explain the submit-number disparity.

A possible mechanism is pre_prep_tasks which is not updated with reloads:

• Run the task.
• Pause the workflow.
• Get the task into pre_prep_tasks somehow.
  • Probably not possible whilst the workflow is paused but there may be other mechanisms beyond workflow-paused, slow remote-init? auto retries? pausing and triggering in the same main loop cycle? pre_prep_tasks length above batched submission limit?
• Reload the workflow.
• Release the workflow.
• Now the task in pre_prep_tasks is the pre-reload copy vs the newer version in the task_pool!!!

[oliver-sanders]

The issue was resolved via a restart which suggests that the issue was due to scheduler state not mirrored in the DB so the pre_prep_tasks theory is likely, any easy fix if so...

[oliver-sanders]

Haven't managed to reproduce, however, this test shows how reload can interfere with the pre_prep_tasks pipeline:

async def test_pre_prep_reload(one, start):
    """Ensure reload does not interfere with the task preparation pipeline."""                         
    async with start(one):
        one.resume_workflow()
        one.pool.release_runahead_tasks()
        one.release_queued_tasks()
        assert len(one.pre_prep_tasks) == 1
        one.command_reload_workflow()           
        assert one.pool.get_tasks()[0] == one.pre_prep_tasks   

[oliver-sanders]

The underlying issue here is that the submit_num is incremented during the job submission process. Because of the way reload works, this means that if a job is awaiting submission (i.e. if it is preparing or in the pre_prep_tasks list) when a reload occurs the submit_num will be incremented on the pre-reload copy of the task (which exists in the submission pipeline) and not the post-reload copy (which exists in the task pool).

This causes the submit-number to be 1 out of line which means job polling will be looking for the wrong submission and job messaging will be associated with the wrong submission. If something goes wrong with preparation/submission the task can become a ghost so to speak, resulting in the stuck state reported.

Solutions:

1. Remove the pre_prep_tasks list.
   • I don't think we need to maintain this from cycle-to-cycle, we can build it when we need it.
   • This solves the issue for reloads on paused workflows (because released tasks build up in pre_prep_tasks).
2. Consider incrementing the submission number before preparation rather than during/after submission.
   • I think this makes more logical sense, especially since a failure in preparation results in the submit-failed state (and an increment in the submission number anyway).
   • At the moment the logs are confusing to read, you will see entries like this all referring to the same submission:
     • Queue released: 1/foo
     • [1/foo waiting job:00 flows:1] => preparing (00 preparing)
     • [1/foo preparing job:01 flows:1] host=<host> (01 preparing ???)
     • [1/foo preparing job:01 flows:1] (internal)submitted at <time> (01 submitted)
   • Would also make the submit_num logic a bit more water-tight as we would only be incrementing the number in one places rather than many (e.g. we actually decrement the submission number after SSH failures, then re-increment it once we've exhausted available platform-hosts ATM).
3. Don't re-create tasks from scratch at reload, just swap out the TaskDef and perform any required updates.
   • Potentially harder, we would need to update prepreqs/outputs.
   • This avoids the issue of having two copies of the task kicking around with all of the potential headaches that can cause.
4. Other?

I think we should do (1) (quick n' easy), to close this issue we would also need to do either (2) or (3).

[dwsutherland]

I was thinking (2) before I read your solutions..

I can't imagine it would be too hard to update the task pool first during the submission process?