Command-line option to statically configure places

[sjg20]

See #1069 (but I was unable to reopen it, so created a new issue with more details)

The current approach requires these steps when starting up the lab:

crossbar start --config ../cfg/crossbar-kea.yaml
LG_CROSSBAR=ws://kea:20408/ws labgrid-exporter ../cfg/export_kea.yaml
LG_CROSSBAR=ws://kea:20408/ws ../labgrid/contrib/sync-places.py sync cfg/places_kea.yaml

It would be better to have labgrid-exporter handle the sync automatically, e.g. by loading the places file.

But even better to not have a places file. My export_kea.yaml file looks like this:

rpi3:
  location: 'USB hub C'
  USBSerialPort:
    ## usbhubc-port4
    match:
      '@ID_PATH': 'pci-0000:03:00.2-usb-0:5.2.4:1.0'

  USBSDWireDevice:
    ## usbhubc-port9
    match:
      ## '@ID_PATH': 'pci-0000:03:00.2-usb-0:5.4.1.2'
      'ID_SERIAL_SHORT': 'sdwire-18'

  YKUSHPowerPort:
    serial: 'YK17698'
    index: 1

and places_kea.yaml is just:

places:
  rpi3:
    matches:
      - kea/rpi3/*

which seems pretty redundant to me.

So could we perhaps have an option to create a place for each exported target as a place?

Yes, I am aware that most uses of labgrid create ad-hoc labs consisting of one or two targets. But I have a lab of about 34 targets so far, so ad-hoc creation isn't really practical.

[Emantor]

The intention behind the coordinator is to have it run continuously in the background, it will correctly store and restore the current place information if it has persistent storage.

A one-time sync to create the places and than modifying them with labgrid-client is not enough here?

[JoshuaWatt]

Our lab is also 40+ devices. You might look at contrib/sync-places.py. It will let you define your places in a YAML file, then sync the configuration of the coordinator to match. We use it to keep our place configuration in source control, then use an Ansible task to run the script to keep the coordinator in sync.

[JoshuaWatt]

Also of note, we have 1 exporter for each DUT, so our coordinator doesn't have access to all exporter configs that would make that even work properly. It gets even more complicated if you have multiple resources for the same DUT spread out over multiple exporters (which we also do in some limited cases). Labgrid is quite flexible in how you can organize it, but it does mean you have to be explicit about what the coordinator is doing vs. the exporters.

[sjg20]

Thanks for the info. Yes I am using sync-places.py but still need to write the places file.

@JoshuaWatt I don't quite understand your comment. What new information is in the places file that is not in the collective exporters? Is it trying to present only a subset of what is available?

[JoshuaWatt]

Exporters aren't really aware of "places". In your exporter config file, the top level key rpi3 is actually the "group name" which is not the same as the place. Exporters can really use any group name you want there. A place is determined by the coordinator, and is a collection of resources that match the provided globs; typical convention is to make the group name and the place name the same, because then your place resource glob can be */rpi3/* and this will pick up any resource with the group rpi3 on any exporter and associate it with the rpi3 place, which makes it easy to add new resources to a place by only modifying the exporter config. However, this is not the only way this can work. Imagine a single resource shared by multiple DUTs, or a "place" that encompasess multiple DUTs and maybe overlaps with other places. These can all be done because there is not a strict linking between the exporter group names and place names; the place can have as simple or complex a regex as required to match what it needs, but that necessarily means these are independent and the one cannot be used in place of the other.

[sjg20]

Thank you for the explanation. IMO this is all quite confusing.

(aside: it seems to me that 'place' and 'group' should be switched, since a 'place' is an arbitrary grouping of things that might come from any exporter whereas a 'group' is a logically connected group of devices from one exporter and presumably in one place)

It would be great to add your notes above in the docs here:

https://labgrid.readthedocs.io/en/latest/overview.html#remote-resources-and-places

Perhaps we could have a mention that the places and groups are generally the same, which makes things easier to understand, but that (for advanced use) they can be different?

[jluebbe]

Thank you for the explanation. IMO this is all quite confusing.

(aside: it seems to me that 'place' and 'group' should be switched, since a 'place' is an arbitrary grouping of things that might come from any exporter whereas a 'group' is a logically connected group of devices from one exporter and presumably in one place)

This again shows how little standard terminology exists in this space. I still remember that a large chunk of the first Automated Testing Summit in 2018 was spent figuring out a shared terminology... And documenting this in a way understandable by someone who already uses different terminology seems really hard. :/

While @JoshuaWatt mentions that his convention is to use the same name for 'group' and 'place', we use it differently in our lab. One rack of boards contains one or two exporters (running on physical machines) for USB busses. Additionally, we have a shared exporter running in a VM which exports PDUs, MQTT power plugs running Tasmota, serial port servers and PoE switches.

• For USB, the resource groups are named after labeled ports on USB hubs.
• For PDUs and serial service, the group is something linke 'lab-a-12' and it contains only one resource of each type.

This allows us to easily combine the resources from multiple exporters into one place and needed by a specific board when we place it on a rack shelf and wire it up. When done, it looks like this

Place 'distrokit-tq-mba8mpxl':
  matches:
    rlabD-srv/d-10/NetworkPowerPort
    rlabD-srv/d-usb-3-p3-1.3/NetworkSerialPort
    rlabD-srv/d-usb-3-p4/NetworkUSBSDMuxDevice
  acquired: None
  acquired resources:
  created: 2024-01-09 14:33:11.611892
  changed: 2024-05-03 04:04:30.917760
Matching resource 'NetworkPowerPort' (rlabD-srv/d-10/NetworkPowerPort/NetworkPowerPort):
  {'acquired': None,
   'avail': True,
   'cls': 'NetworkPowerPort',
   'params': {'extra': {'proxy': 'rlabD-srv.labmgmt.stw.pengutronix.de',
                        'proxy_required': False},
              'host': 'rlabd-srv.labmgmt.stw.pengutronix.de',
              'index': 11,
              'model': 'gude24'}}
Matching resource 'USBSerialPort' (rlabD-srv/d-usb-3-p3-1.3/NetworkSerialPort/USBSerialPort):
  {'acquired': None,
   'avail': True,
   'cls': 'NetworkSerialPort',
   'params': {'extra': {'path': '/dev/ttyUSB18',
                        'proxy': 'rlabD-srv.labmgmt.stw.pengutronix.de',
                        'proxy_required': False},
              'host': 'rlabD-srv',
              'port': None,
              'speed': 115200}}
Matching resource 'USBSDMuxDevice' (rlabD-srv/d-usb-3-p4/NetworkUSBSDMuxDevice/USBSDMuxDevice):
  {'acquired': None,
   'avail': True,
   'cls': 'NetworkUSBSDMuxDevice',
   'params': {'busnum': 4,
              'control_path': '/dev/sg1',
              'devnum': 64,
              'extra': {'proxy': 'rlabD-srv.labmgmt.stw.pengutronix.de',
                        'proxy_required': False},
              'host': 'rlabD-srv',
              'model_id': 16449,
              'path': '/dev/sdb',
              'vendor_id': 1060}}

This allows us to add/change/remove places just by looking at the labeled USB ports and power cables.

It would be great to add your notes above in the docs here:

https://labgrid.readthedocs.io/en/latest/overview.html#remote-resources-and-places

Perhaps we could have a mention that the places and groups are generally the same, which makes things easier to understand, but that (for advanced use) they can be different?

Yes, but it would still need to make the distinction that groups are mainly a exporter-level concept while places only come into play with the coordinator.

[liambeguin]

This again shows how little standard terminology exists in this space. I still remember that a large chunk of the first Automated Testing Summit in 2018 was spent figuring out a shared terminology... And documenting this in a way understandable by someone who already uses different terminology seems really hard. :/

Not sure how helpful this is in our case, but following the links lead to this page: https://elinux.org/Test_Glossary