Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/8.0.x' into fix.rose-re-instal…
Browse files Browse the repository at this point in the history
…l-failure.bk

* upstream/8.0.x:
  remote: ensure all remote commands use a platform config (cylc#5152)
  Db store force triggered (cylc#5023)
  Run GH Actions tests on push to `8.*.x` branches
  Auto bump dev version on release
  remote-install: add "ana/" to the default install list (cylc#5137)
  A no-flow task should not merge and retrigger incomplete children (cylc#5146)
  `log_vc_info`: Redirect diff straight to file to avoid blocking pipe (cylc#5139)
  fix reversed data-store edge source-target (cylc#5156)
  Fix Jinja2 support if HOME undefined.
  Assume Jinja2 might be used in global-tests.cylc.
  Fix post-reload trigger. (cylc#5104)
  Bump dev version
  Update changelog
  workflow_state xtrigger: infer run num
  Type annotations
  Prepare release 8.0.2
  Remove HOME Env Variable from get_remote_workflow_run_dir (cylc#5115)
  • Loading branch information
wxtim committed Oct 3, 2022
2 parents 45ab9b7 + ece609d commit 6f83653
Show file tree
Hide file tree
Showing 41 changed files with 667 additions and 283 deletions.
7 changes: 7 additions & 0 deletions .github/workflows/2_auto_publish_release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,10 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
release-url: ${{ steps.create-release.outputs.html_url }}

- name: Bump dev version
uses: cylc/release-actions/stage-2/bump-dev-version@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
init-file: 'cylc/flow/__init__.py'
4 changes: 3 additions & 1 deletion .github/workflows/bash.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ on:
- '**.md'
- '**/README*/**'
push:
branches: [master]
branches:
- master
- '8.*.x'
paths-ignore:
- '.github/workflows/*.ya?ml'
- '!.github/workflows/bash.yml'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/test_fast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ on:
pull_request:
workflow_dispatch:
push:
branches: [master]
branches:
- master
- '8.*.x'

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/test_functional.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ on:
- '**.md'
- '**/README*/**'
push:
branches: [master]
branches:
- master
- '8.*.x'
paths-ignore:
- '.github/workflows/*.ya?ml'
- '!.github/workflows/test_functional.yml'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/test_tutorial_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ name: test-tutorial-workflow

on:
push:
branches: [master]
branches:
- master
- '8.*.x'
pull_request:
paths-ignore:
- '.github/workflows/*.ya?ml'
Expand Down
36 changes: 34 additions & 2 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,34 @@ creating a new release entry be sure to copy & paste the span tag with the
`actions:bind` attribute, which is used by a regex to find the text to be
updated. Only the first match gets replaced, so it's fine to leave the old
ones in. -->

-------------------------------------------------------------------------------
## __cylc-8.0.3 (<span actions:bind='release-date'>Pending YYYY-MM-DD</span>)__

Maintenance release.

### Fixes

[#5023](https://github.com/cylc/cylc-flow/pull/5023) - tasks force-triggered
after a shutdown was ordered should submit to run immediately on restart.

[#5137](https://github.com/cylc/cylc-flow/pull/5137) -
Install the `ana/` directory to remote platforms by default.

[#5146](https://github.com/cylc/cylc-flow/pull/5146) - no-flow tasks should not
retrigger incomplete children.

[#5104](https://github.com/cylc/cylc-flow/pull/5104) - Fix retriggering of
failed tasks after a reload.

[#5139](https://github.com/cylc/cylc-flow/pull/5139) - Fix bug where
`cylc install` could hang if there was a large uncommitted diff in the
source dir (for git/svn repos).

[#5131](https://github.com/cylc/cylc-flow/pull/5131) - Infer workflow run number
for `workflow_state` xtrigger.

-------------------------------------------------------------------------------
## __cylc-8.1.0 (<span actions:bind='release-date'>Upcoming</span>)__

### Enhancements
Expand All @@ -42,15 +68,21 @@ numbers of Cylc Lint's style issues and allow users to ignore Cylc Lint issues
using `--ignore <Issue Code>`.

-------------------------------------------------------------------------------
## __cylc-8.0.2 (<span actions:bind='release-date'>Upcoming</span>)__

Maintenance release.
## __cylc-8.0.3 (<span actions:bind='release-date'>Upcoming</span>)__

### Fixes

[#5125](https://github.com/cylc/cylc-flow/pull/5125) - Allow rose-suite.conf
changes to be considered by ``cylc reinstall``.

-------------------------------------------------------------------------------
## __cylc-8.0.2 (<span actions:bind='release-date'>Released 2022-09-12</span>)__

Maintenance release.

### Fixes

[#5115](https://github.com/cylc/cylc-flow/pull/5115) - Updates rsync commands
to make them compatible with latest rsync releases.

Expand Down
2 changes: 1 addition & 1 deletion cylc/flow/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def environ_init():

environ_init()

__version__ = '8.1.0.dev'
__version__ = '8.0.3.dev'


def iter_entry_points(entry_point_name):
Expand Down
16 changes: 12 additions & 4 deletions cylc/flow/cfgspec/globalcfg.py
Original file line number Diff line number Diff line change
Expand Up @@ -1614,18 +1614,26 @@ def default_for(
with Conf('localhost', meta=Platform, desc='''
A default platform for running jobs on the the scheduler host.
.. attention::
This platform configures the host on which
:term:`schedulers <scheduler>` run. By default this is the
host where ``cylc play`` is run, however, we often configure
Cylc to start schedulers on dedicated hosts by configuring
:cylc:conf:`global.cylc[scheduler][run hosts]available`.
It is common practice to start Cylc schedulers on dedicated
hosts, in which case **"localhost" is the scheduler host and
not necessarily where you ran "cylc play"**.
This platform affects connections made to the scheduler host and
any jobs run on it.
.. versionadded:: 8.0.0
'''):
Conf('hosts', VDR.V_STRING_LIST, ['localhost'], desc='''
List of hosts for the localhost platform. You are unlikely to
need to change this.
The scheduler hosts are configured by
:cylc:conf:`global.cylc[scheduler][run hosts]available`.
See :ref:`Submitting Workflows To a Pool Of Hosts` for
more information.
.. seealso::
:cylc:conf:`global.cylc[platforms][<platform name>]hosts`
Expand Down
15 changes: 11 additions & 4 deletions cylc/flow/cfgspec/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,10 +222,17 @@ def get_script_common_text(this: str, example: Optional[str] = None):
The following directories already get installed by default:
* ``app/``
* ``bin/``
* ``etc/``
* ``lib/``
``ana/``
Rose ana analysis modules
``app/``
Rose applications
``bin/``
Cylc bin directory (added to ``PATH``)
``etc/``
Miscellaneous resources
``lib/``
Cylc lib directory (``lib/python`` added to ``PYTHONPATH``
for workflow config)
These should be located in the top level of your Cylc workflow,
i.e. the directory that contains your ``flow.cylc`` file.
Expand Down
17 changes: 12 additions & 5 deletions cylc/flow/data_store_mgr.py
Original file line number Diff line number Diff line change
Expand Up @@ -733,11 +733,18 @@ def _expand_graph_window(
and e_id not in self.added[EDGES]
and edge_distance <= self.n_edge_distance
):
self.added[EDGES][e_id] = PbEdge(
id=e_id,
source=s_tokens.id,
target=t_tokens.id
)
if is_parent:
self.added[EDGES][e_id] = PbEdge(
id=e_id,
source=t_tokens.id,
target=s_tokens.id
)
else:
self.added[EDGES][e_id] = PbEdge(
id=e_id,
source=s_tokens.id,
target=t_tokens.id
)
# Add edge id to node field for resolver reference
self.updated[TASK_PROXIES].setdefault(
t_tokens.id,
Expand Down
4 changes: 2 additions & 2 deletions cylc/flow/host_select.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
from cylc.flow.cfgspec.glbl_cfg import glbl_cfg
from cylc.flow.exceptions import CylcError, HostSelectException
from cylc.flow.hostuserutil import get_fqdn_by_host, is_remote_host
from cylc.flow.remote import _remote_cylc_cmd, run_cmd
from cylc.flow.remote import run_cmd, cylc_server_cmd
from cylc.flow.terminal import parse_dirty_json


Expand Down Expand Up @@ -533,7 +533,7 @@ def _get_metrics(hosts, metrics, data=None):
}
for host in hosts:
if is_remote_host(host):
proc_map[host] = _remote_cylc_cmd(cmd, host=host, **kwargs)
proc_map[host] = cylc_server_cmd(cmd, host=host, **kwargs)
else:
proc_map[host] = run_cmd(['cylc'] + cmd, **kwargs)

Expand Down
98 changes: 61 additions & 37 deletions cylc/flow/install_plugins/log_vc_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,9 @@
import json
from pathlib import Path
from subprocess import Popen, DEVNULL, PIPE
from typing import Any, Dict, Iterable, List, Optional, TYPE_CHECKING, Union
from typing import (
Any, Dict, Iterable, List, Optional, TYPE_CHECKING, TextIO, Union, overload
)

from cylc.flow import LOG
from cylc.flow.exceptions import CylcError
Expand All @@ -80,8 +82,6 @@
GIT: ['describe', '--always', '--dirty']
}

# git ['show', '--quiet', '--format=short'],

STATUS_COMMANDS: Dict[str, List[str]] = {
SVN: ['status', '--non-interactive'],
GIT: ['status', '--short']
Expand Down Expand Up @@ -189,13 +189,40 @@ def get_vc_info(path: Union[Path, str]) -> Optional[Dict[str, Any]]:
return None


def _run_cmd(vcs: str, args: Iterable[str], cwd: Union[Path, str]) -> str:
"""Run a VCS command, return stdout.
@overload
def _run_cmd(
vcs: str, args: Iterable[str], cwd: Union[Path, str], stdout: int = PIPE
) -> str:
...


@overload
def _run_cmd(
vcs: str, args: Iterable[str], cwd: Union[Path, str], stdout: TextIO
) -> None:
...


def _run_cmd(
vcs: str,
args: Iterable[str],
cwd: Union[Path, str],
stdout: Union[TextIO, int] = PIPE
) -> Optional[str]:
"""Run a VCS command.
Args:
vcs: The version control system.
args: The args to pass to the version control command.
cwd: Directory to run the command in.
stdout: Where to redirect output (either PIPE or a
text stream/file object). Note: only use PIPE for
commands that will not generate a large output, otherwise
the pipe might get blocked.
Returns:
Stdout output if stdout=PIPE, else None as the output has been
written directly to the specified file.
Raises:
VCSNotInstalledError: The VCS is not found.
Expand All @@ -208,7 +235,7 @@ def _run_cmd(vcs: str, args: Iterable[str], cwd: Union[Path, str]) -> str:
cmd,
cwd=cwd,
stdin=DEVNULL,
stdout=PIPE,
stdout=stdout,
stderr=PIPE,
text=True,
)
Expand Down Expand Up @@ -275,41 +302,40 @@ def _parse_svn_info(info_text: str) -> Dict[str, Any]:
return ret


def get_diff(vcs: str, path: Union[Path, str]) -> Optional[str]:
"""Return the diff of uncommitted changes for a repository.
def write_diff(
vcs: str, repo_path: Union[Path, str], run_dir: Union[Path, str]
) -> Path:
"""Get and write the diff of uncommitted changes for a repository to the
workflow's vcs log dir.
Args:
vcs: The version control system.
path: The path to the repo.
"""
args_ = DIFF_COMMANDS[vcs]
if Path(path).is_absolute():
args_.append(str(path))
else:
args_.append(str(Path().cwd() / path))

try:
diff = _run_cmd(vcs, args_, cwd=path)
except (VCSNotInstalledError, VCSMissingBaseError):
return None
header = (
"# Auto-generated diff of uncommitted changes in the Cylc "
"workflow repository:\n"
f"# {path}")
return f"{header}\n{diff}"


def write_diff(diff: str, run_dir: Union[Path, str]) -> None:
"""Write a diff to the workflow's vcs log dir.
Args:
diff: The diff.
repo_path: The path to the repo.
run_dir: The workflow run directory.
Returns the path to diff file.
"""
args = DIFF_COMMANDS[vcs]
args.append(
str(repo_path) if Path(repo_path).is_absolute() else
str(Path().cwd() / repo_path)
)

diff_file = Path(run_dir, LOG_VERSION_DIR, DIFF_FILENAME)
diff_file.parent.mkdir(exist_ok=True)
with open(diff_file, 'w') as f:
f.write(diff)

with open(diff_file, 'a') as f:
f.write(
"# Auto-generated diff of uncommitted changes in the Cylc "
"workflow repository:\n"
f"# {repo_path}\n"
)
f.flush()
try:
_run_cmd(vcs, args, repo_path, stdout=f)
except VCSMissingBaseError as exc:
f.write(f"# No diff - {exc}")
return diff_file


# Entry point:
Expand All @@ -331,8 +357,6 @@ def main(
if vc_info is None:
return False
vcs = vc_info['version control system']
diff = get_diff(vcs, srcdir)
write_vc_info(vc_info, rundir)
if diff is not None:
write_diff(diff, rundir)
write_diff(vcs, srcdir, rundir)
return True
Loading

0 comments on commit 6f83653

Please sign in to comment.