clock-expire: prevent active tasks from being clock-expired #6025
Comments
oliver-sanders
added
the
question
oliver-sanders
changed the title
oliver-sanders
added
small
and removed
question
|
Relatedly, we should probably not allow any final output (i.e., expired, submit-failed, failed, succeeded) to be manually set on an active task. Setting these outputs also changes the task state to a final state, which is inconsistent with having a submitted or running job. And we have previously decided that only We could decide not to change the state of an active task via |
No, the proposal explicitly permits this. |
|
No it doesn't 😁 - it only explicitly states that we should be able to cancel a retry chain by setting a final output. Typically you would do that after a job fails and is waiting for a retry, which is fine. By active, I meant literally active, i.e. "submitted" or "running", not just "present in the task pool". Setting an active task to failed leaves an orphaned job out in the world, which can be confusing (I see a bit of this from users, for other reasons), or even dangerous (if the user triggers the task again they'll get duplicate jobs fighting it out on the filesystem). So, if the task is active, I think we should have to kill it first, then set it to failed (which is always possible, even for short retry delays, because a killed task gets held). If the job can't be killed, then we have no choice but to orphan it, but that's an edge case (it could be handled automatically on failure to kill, or with |
Yes it does, see also: |
|
Ah OK, I stand corrected - you linked to the wrong bit of the proposal. However, my argument above still stands! The proposal neglects to address the problem of orphaned jobs, which can be avoided simply by requiring a job kill before a state reset (where possible - note I did also state above that we need the ability to force it if the job can't be killed). |
|
This issue covers the mechanism of clock-expiry which is simple and uncontroversial, more significant change will require a new proposal. |
|
This is closely related, hence I raised it here for discussion. If it wasn't controversial (I didn't expect it to be) we could simply change the issue title slightly. But fine - I'll shift it to another issue. We don't need a new proposal for every small detail... #6033 |
Full proposal, no, but we should put up an amendment if a change directly contradicts an existing proposal, especially a fresh one that is still being implemented. |
|
The proposal doesn't make mention of the interaction between One related part of the proposal is point 10:
Suggest tackling that along with this issue rather than shoehorning it into #5640 as the solution will likely overlap. |
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
* Active tasks should not be considered for clock-expiry. Closes cylc#6025 * Manually triggered tasks should not be considered for clock-expiry. Implements proposal point 10 https://cylc.github.io/cylc-admin/proposal-optional-output-extension.html#proposal
From discussion on #6020
Clock expiry can complete a task.
If the task is already active then it should not be completed in this way.
Note, I think we would expect a clock-expired task to be able to expire between retries.
The text was updated successfully, but these errors were encountered: