Add compatibility test

Signed-off-by: Erik Jaegervall <[email protected]>

README.md

@@ -42,6 +42,31 @@ Examples on tool usage can be found in the [VSS Makefile](https://github.com/COV
 
 All current tools are based on common Python functionality in the `vspec` folder to read, parse and expand a Vehicle Signal Specification files(*.vspec files). As an example, if the standard [VSS root file](https://github.com/COVESA/vehicle_signal_specification/blob/master/spec/VehicleSignalSpecification.vspec) is given as input then the Python tooling will read all included files, do a validation of the content, expand any instances used and create an in-memory representation which then can be used by specialized tools to generate the wanted output.
 
+## Compatibility with VSS
+
+The [COVESA VSS project repository](https://github.com/COVESA/vehicle_signal_specification) includes vss-tools as a submodule.
+The vss-tools version linked by the VSS repository is the preferred vss-tools version to use for that particular version of the VSS repository. It is not guaranteed that newer or older versions of vss-tools can successfully handle that particular version of the VSS repository. The table below gives an overview of basic version support for`vspec2json.py`,
+other exporters may have stricter requirements.
+
+VSS-tools version | Supported VSS versions | Comments
+-----------------|------------------------|----------------
+`v3.0`| `v3.0` - `v3.1.1`
+`v3.1`| `v3.0` -`v4.0`
+`v4.0`| `v4.0`
+`v4.1`| `v4.0` -
+`<latest source>`| `v4.0` -
+
+### Changes affecting compatibility
+
+Examples on changes affecting compatibility
+
+* VSS version `v4.1` introduced a new syntax for the unit files that cannot be handled by `vss-tools < v4.1`
+* From `v4.0` vss-tools expects unit file to be explicitly specified or provided in the same directory as the VSS input.
+  VSS `v3.1` is the first VSS version including a unit file in the VSS repository.
+  This means vss-tools from `v4.0` onwards cannot handle VSS-versions prior to VSS `v3.1`
+* VSS-tools `v3.1` only supported `default` for attributes, resulting in that newer VSS-versions is not supported.
+* VSS-tools `v4.0` requires case-sensitive for type, resulting in that VSS versions `v3.1` and earlier is not supported.
+
 ## Getting started
 
 ## Prerequisites

tests/backward_compatibility/test_compatibility.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2024 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+import pytest
+import os
+
+
+@pytest.fixture
+def change_test_dir(request, monkeypatch):
+    # To make sure we run from test directory
+    monkeypatch.chdir(request.fspath.dirname)
+
+
+# Test all VSS versions we support
+#
+# Intended workflow:
+#
+# ---------- After a new VSS release -----------------
+#
+# * Add the new tag to this test case
+# * Update compatibility section in README
+#
+# ----------- If this test case fails -----------------
+#
+# * Check if we can add backward compatibility with limited effort
+# * If not add limitation to compatibility section in README and remove '
+#   unsupported versions from the test case
+#
+@pytest.mark.parametrize("tag",
+                         [
+                          'v4',
+                          'v4.0',
+                          'v4.1'])
+def test_compatibility(tag, change_test_dir):
+    """
+    Test that we still can analyze wanted versions without error
+    """
+
+    os.system("rm -rf vehicle_signal_specification")
+    os.system("git clone --depth 1 --branch " + tag +
+              " https://github.com/COVESA/vehicle_signal_specification")
+
+    result = os.system("../../vspec2json.py --json-pretty "
+                       "vehicle_signal_specification/spec/VehicleSignalSpecification.vspec "
+                       "out.json 1>out.txt 2>&1")
+    os.system("cat out.txt")
+    assert os.WIFEXITED(result)
+    assert os.WEXITSTATUS(result) == 0
+
+    os.system("cd ..")
+    os.system("rm -rf vehicle_signal_specification")