[[localhost]] platform selects from top of platforms list

[wxtim]

Describe the bug
Localhost doesn't seem to follow the same platform selection rules as other platforms - it appears to select the top regex in the list, and not the bottom one!

Steps to reproduce the bug
Given the global.cylc:

[platforms]
    [[localhost|foo|bar]]
        global init-script = "echo 'defined  first'"
    [[localhost]]
        global init-script = "echo 'defined last'"
    [[foo|bar|baz]]
        global init-script = "echo 'defined first'"
    [[baz]]
        global init-script = "echo 'defined last'"

and the flow.cylc:

[scheduling]
    initial cycle point = 19831213T0600Z
    [[dependencies]]
        R1 = Quokka => Numbat
[runtime]
    [[Quokka]]
    [[Numbat]]
        platform = baz

then testing for the global init-script in the job script

grep "defined [first|last]" ~/cylc-run/localhost-investigation/runN/log/job/19831213T0600Z/*/NN/job

One would expect both to return "defined last" but in fact this is not the case for localhost.

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

[wxtim]

On further investigation, I'm not at all sure it's platforms that's at fault:

python -c "from cylc.flow.cfgspec.glbl_cfg import glbl_cfg; print(glbl_cfg().get(['platforms']).keys())"

Commenting out the default platform localhost fixes the problem.

Conclusion

What appears to be going on here is that the global config spec always adds localhost as a special case so it's always at the top of the list, even if you've defined it further down the list elsewhere.

Further examination suggests that even then you are getting the hard-set default localhost, and not overridding any of its settings - consider

python -c "from cylc.flow.cfgspec.glbl_cfg import glbl_cfg; print(glbl_cfg().get(['platforms', 'localhost', 'global init script']))"

Possible solutions

1. Document that users should not use [localhost], add a warning if it's defined in the user config and suggest that users use something like [[localhost|some_very_unlikely_platform_name]] as a workaround.
2. Add logic to parsec - allow the spec to specify cases where user setting can over-ride a hard set section like `[platforms][localhost].

@dpmatthews & @oliver-sanders - I'd appreciate your thoughts.

[oliver-sanders]

localhost is more of an alias than a host "name" per-se, it evaluates differently on every machine. I think it is ok if it doesn't work with platform pattern matches. Put everything into a [platform][localhost] section and I think you're fine?

So I'm suggesting we document not to use localhost in pattern matches.

What appears to be going on here is that the global config spec always adds localhost as a special case

We need localhost to be defined by default so we specify it with sensible defaults as a "meta" section of the config:

https://cylc.github.io/cylc-doc/latest/html/reference/config/global.html#global.cylc[platforms][localhost]

Further examination suggests that even then you are getting the hard-set default localhost, and not overridding any of its settings

Overriding [localhost] settings in the global.cylc works fine for me:

# global.cylc
[platforms]
    [[localhost]]
        hosts = foo

$ cylc config -i '[platforms][localhost]'
hosts = foo

Document that users should not use [localhost]

Isn't this the wrong way around?

[wxtim]

Do I interpret this as you saying that

if "localhost" in platform_regex and platform_regex is not "localhost":
    raise CylcException("Don't use 'localhost' in a broader regex.")

[oliver-sanders]

Yep, perhaps a warning rather than an exception, if we run the regex we can catch things like local.... and .* so this might be worthwhile:

if (
    # if the platform contains special regular expression characters
    re.escape(platform_regex) != platform_regex
    # and 
    and re.match(platform_regex, 'localhost')
):
    LOG.warn(f'The platform regex {platform_regex} matches "localhost", however, ...')

[oliver-sanders]

Swapping "bug" for "doc", this behaviour is fine and easy to work around, we just need to document it somewhere. Either a warning in the code, or a line in cylc-doc or both.

[wxtim]

In a conversation with @dpmatthews this morning, it looks like he favours making this an error, not a warning.

Also convinced me that it's probably worth bringing the comma separated list and expansion behaviour of the global config more into line with Cylc config.

[oliver-sanders]

Items in the [graph] and [runtime] sections can be comma separated e.g. [runtime][foo, bar], however, this is not a generic parsec feature, it is enabled on a case-by-case basis so was not available to the global.cylc[platforms] section.

The [graph] and [runtime] section use different approaches, so I added another one 🙈 in parsec.utils which implements this splitting in a more generic fashion as we might want to turn it on for all __MANY__ sections.

https://github.com/oliver-sanders/cylc-flow/pull/new/parsec-section-expand-many