ihs: catch NoHostsError #5195

oliver-sanders · 2022-10-14T12:46:53Z

Catch NoHostsError in the code it can occur in.
Attempt to handle the error in these contexts.
Mark the functions where the error can crop up for future warning.

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg and conda-environment.yml.
Tests are included (or explain why tests are not needed).
CHANGES.md entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

cylc/flow/task_job_mgr.py

cylc/flow/task_remote_mgr.py

wxtim

Read code
Checked out, run new tests
Proof of pudding test, played with code.

tests/functional/events/51-task-event-job-logs-retrieve-4.t

wxtim · 2022-10-18T14:32:53Z

tests/unit/test_host_select.py

+    caplog.set_level(logging.WARN, 'cylc')
+    host_stats, data = _get_metrics(['not-a-host'], None)
+    # a warning should be logged
+    assert len(caplog.records) == 1


Any warning at all?

Yup.

Note: This warning could only come from _get_metrics which is a small simple function.

cylc/flow/task_events_mgr.py

cylc/flow/task_job_mgr.py

MetRonnie

I have found this does not address Dave's NoHostsError during shutdown

Traceback (most recent call last):
  File "~/cylc-flow/cylc/flow/scheduler.py", line 1742, in _shutdown
    self.proc_pool.terminate()
  File "~/cylc-flow/cylc/flow/subprocpool.py", line 345, in terminate
    self.process()
  File "~/cylc-flow/cylc/flow/subprocpool.py", line 224, in process
    self._proc_exit(
  File "~/cylc-flow/cylc/flow/subprocpool.py", line 206, in _proc_exit
    self._run_command_exit(
  File "~/cylc-flow/cylc/flow/subprocpool.py", line 529, in _run_command_exit
    res = _run_callback(callback_255, callback_255_args)
  File "~/cylc-flow/cylc/flow/subprocpool.py", line 481, in _run_callback
    callback(ctx, *args_)
  File "~/cylc-flow/cylc/flow/task_job_mgr.py", line 820, in _poll_task_jobs_callback_255
    self._manip_task_jobs_callback(
  File "~/cylc-flow/cylc/flow/task_job_mgr.py", line 806, in _manip_task_jobs_callback
    summary_callback(workflow, itask, ctx, line)
  File "~/cylc-flow/cylc/flow/task_job_mgr.py", line 829, in _poll_task_job_callback_255
    self.poll_task_jobs(workflow, [itask])
  File "~/cylc-flow/cylc/flow/task_job_mgr.py", line 210, in poll_task_jobs
    self._run_job_cmd(
  File "~/cylc-flow/cylc/flow/task_job_mgr.py", line 948, in _run_job_cmd
    host = get_host_from_platform(
  File "~/cylc-flow/cylc/flow/platforms.py", line 516, in get_host_from_platform
    raise NoHostsError(platform)

(line numbers here are as on master not this branch)

tests/unit/test_host_select.py

oliver-sanders · 2022-10-25T13:56:24Z

Rebased, added a try/except for kill & poll to address the reported traceback.

MetRonnie · 2022-10-26T09:26:09Z

Hmm, with this I now get new traceback for an invalid ssh config

INFO - platform: exvcylcdev01 - remote init (on exvcylcdev01)
ERROR - [Errno 2] No such file or directory: 'jobs-poll'
    Traceback (most recent call last):
      File "~/cylc-flow/cylc/flow/subprocpool.py", line 429, in _run_command_init
        proc = procopen(
      File "~/cylc-flow/cylc/flow/cylc_subproc.py", line 55, in procopen
        process = Popen(command, bufsize, executable, stdin, stdout,  # nosec
      File "~/.conda/envs/cylc8/lib/python3.9/subprocess.py", line 951, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "~/.conda/envs/cylc8/lib/python3.9/subprocess.py", line 1821, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'jobs-poll'
ERROR - [jobs-poll cmd] jobs-poll -- '$HOME/cylc-run/remote-example/run1/log/job'
    [jobs-poll ret_code] 1
    [jobs-poll err] [Errno 2] No such file or directory: 'jobs-poll'

and the workflow seems to hang.

Might be why tests/f/restart/26-remote-kill.t is failing, it also seems to hang

* Catch NoHostsError in the code it can occur in. * Attempt to handle the error in these contexts. * Mark the functions where the error can crop up for future warning. * Fix job log retrieval retries.

oliver-sanders · 2022-12-06T16:19:13Z

Hmm, with this I now get new traceback for an invalid ssh config

Sorted. I had made a mistake with the error handling in one place, all tests should now be passing.

Co-authored-by: Ronnie Dutta <[email protected]>

hjoliver

Looks good, tested, no problems found 👍

hjoliver · 2022-12-07T00:43:56Z

tests/unit/test_host_select.py

+def test_get_metrics_no_hosts_error(caplog):
+    """It should handle SSH errors.
+
+    If a host is not contactable then it should be shipped.


"shipped/skipped"? (Obvious enough, I guess!)

hjoliver · 2022-12-07T00:46:10Z

Tests didn't run on last commit? Kicking...

hjoliver · 2022-12-07T01:30:36Z

Passed. Patch-diff coverage only 75%, but it'll do for now.

oliver-sanders added the bug Something is wrong :( label Oct 14, 2022

oliver-sanders added this to the cylc-8.0.4 milestone Oct 14, 2022

oliver-sanders self-assigned this Oct 14, 2022

oliver-sanders force-pushed the no-hosts-error branch from f267d24 to 31857e9 Compare October 18, 2022 12:20

oliver-sanders commented Oct 18, 2022

View reviewed changes

cylc/flow/task_job_mgr.py Show resolved Hide resolved

oliver-sanders commented Oct 18, 2022

View reviewed changes

cylc/flow/task_remote_mgr.py Show resolved Hide resolved

oliver-sanders force-pushed the no-hosts-error branch from 31857e9 to 6979613 Compare October 18, 2022 12:24

oliver-sanders requested review from MetRonnie and wxtim October 18, 2022 12:25

oliver-sanders marked this pull request as ready for review October 18, 2022 12:26

oliver-sanders force-pushed the no-hosts-error branch 2 times, most recently from 02cd1b8 to e0fca23 Compare October 18, 2022 14:26

wxtim approved these changes Oct 18, 2022

View reviewed changes

MetRonnie reviewed Oct 18, 2022

View reviewed changes

cylc/flow/task_events_mgr.py Show resolved Hide resolved

cylc/flow/task_job_mgr.py Show resolved Hide resolved

MetRonnie reviewed Oct 18, 2022

View reviewed changes

tests/unit/test_host_select.py Outdated Show resolved Hide resolved

oliver-sanders mentioned this pull request Oct 20, 2022

auto-restart: enable for stopping workflows #5203

Open

oliver-sanders force-pushed the no-hosts-error branch from e0fca23 to 16fafc9 Compare October 25, 2022 13:55

oliver-sanders marked this pull request as draft October 26, 2022 12:46

ihs: catch NoHostsError

dc15ce4

* Catch NoHostsError in the code it can occur in. * Attempt to handle the error in these contexts. * Mark the functions where the error can crop up for future warning. * Fix job log retrieval retries.

oliver-sanders force-pushed the no-hosts-error branch from 16fafc9 to dc15ce4 Compare December 6, 2022 15:36

oliver-sanders marked this pull request as ready for review December 6, 2022 16:18

Update tests/unit/test_host_select.py

818887a

Co-authored-by: Ronnie Dutta <[email protected]>

oliver-sanders force-pushed the no-hosts-error branch from d3a7b49 to 818887a Compare December 6, 2022 16:35

hjoliver self-requested a review December 6, 2022 21:40

hjoliver approved these changes Dec 7, 2022

View reviewed changes

hjoliver closed this Dec 7, 2022

hjoliver reopened this Dec 7, 2022

hjoliver merged commit d8d4946 into cylc:8.0.x Dec 7, 2022

oliver-sanders deleted the no-hosts-error branch December 7, 2022 09:56

oliver-sanders mentioned this pull request Dec 16, 2022

polling: infinite recursion loop on 255 error #5276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ihs: catch NoHostsError #5195

ihs: catch NoHostsError #5195

oliver-sanders commented Oct 14, 2022 •

edited

Loading

wxtim left a comment

wxtim Oct 18, 2022

oliver-sanders Oct 18, 2022

MetRonnie left a comment

oliver-sanders commented Oct 25, 2022

MetRonnie commented Oct 26, 2022 •

edited

Loading

oliver-sanders commented Dec 6, 2022 •

edited

Loading

hjoliver left a comment

hjoliver Dec 7, 2022

hjoliver commented Dec 7, 2022

hjoliver commented Dec 7, 2022

ihs: catch NoHostsError #5195

ihs: catch NoHostsError #5195

Conversation

oliver-sanders commented Oct 14, 2022 • edited Loading

wxtim left a comment

Choose a reason for hiding this comment

wxtim Oct 18, 2022

Choose a reason for hiding this comment

oliver-sanders Oct 18, 2022

Choose a reason for hiding this comment

MetRonnie left a comment

Choose a reason for hiding this comment

oliver-sanders commented Oct 25, 2022

MetRonnie commented Oct 26, 2022 • edited Loading

oliver-sanders commented Dec 6, 2022 • edited Loading

hjoliver left a comment

Choose a reason for hiding this comment

hjoliver Dec 7, 2022

Choose a reason for hiding this comment

hjoliver commented Dec 7, 2022

hjoliver commented Dec 7, 2022

oliver-sanders commented Oct 14, 2022 •

edited

Loading

MetRonnie commented Oct 26, 2022 •

edited

Loading

oliver-sanders commented Dec 6, 2022 •

edited

Loading