Example downstream manifest+module repo in zephyrproject-rtos

[carlescufi]

Create a new repository in the https://github.com/zephyrproject-rtos organizationt that can serve both as reference and as a starting point for users wanting to create their own manifest+module repo (later this can be extended with additional sample module repos).

https://github.com/zephyrproject-rtos/zephyr-downstream-application
which would have:

• A west.yml so it serves as a manifest repo. This will import zephyr's manifest.
• A zephyr/module.yml so it serves as an actual downstream module. This should include:
  • A custom CMakeLists.txt
  • A custom Kconfig.module
  • A custom board, using board_root and dts_root
• A library configurable with Kconfig
• A driver configurable with Kconfig and Devicetree
• A sample that uses that library and driver (and other Zephyr ones)
• A test that tests the library

We could then move https://github.com/zephyrproject-rtos/zephyr/tree/master/samples/application_development/out_of_tree_driver/hello_world_module/zephyr to the new repository.

[carlescufi]

This will require integration with our CI system to make sure we don't break downstream modules and applications, either as a new GitHub Workflow or in BuildKite.

[mbolivar-nordic]

self-assigned at least for any west instrumentation changes needed to get this working in CI on each PR

[gregshue]

For clarification,is this an example of what could be done, what is expected to be done, or what will be supported for the mid- to long- term?

[mbolivar-nordic]

A driver configurable with Kconfig

... and dts, I assume?

[tejlmand]

For clarification,is this an example of what could be done, what is expected to be done, or what will be supported for the mid- to long- term?

imo this will be a repo that can be used to ensure that Zephyr module features for out-of-tree are working can be tested by CI so that it is ensured that it work as expected.
At the same time, it can serve as a simple reference point for downstream users.

That way we ensure:

• A reference point of a single use case
• CI testing of Zephyr module features
• Guarantee that the reference point will always be in sync with the code, else CI will fail

[gregshue]

@tejlmand, I agree with the need to support CI verification of Zephyr module features. I think this needs to be more than a "simple reference point" of how those features can be used. I want to know how the module is expected to be setup - in practice, a boilerplate. Extending my downstream modules as Zephyr module features get added is fine (and expected), but restructuring it to take advantage of a new feature is highly undesirable.

Since we need to support extending (potentially forks of) existing external repositories to be zephyr modules (e.g. github.com/project-chip/connectedhomeip), I propose:

• All(?) the zephyr unique files be kept under the necessary zephyr sub-directory, following the zephyr repo directory structure (e.g. manifests, those discovered by sanitycheck/CI, hello_ sample, west extension scripts, Kconfig files).
• Shared namespaces (e.g. include pathnames, CPP symbols, Kconfig vars, etc) be prepended with the module repository org/name, where dashes are replaced with underscores. (It's really too long, but we need to avoid namespace conflicts.)
• Automatic formatting rules are provided for all files beneath the zephyr sub-directory structure (e.g. .clang-format). This will simplify file comparisons against the boilerplate.
• A repo-level .gitignore be commented with why each setting is necessary, since some may conflict with pre-existing build tools.

[gregshue]

A couple more:

• Names/prefixes of tags and the main development branch and tags are recommended to be the manifest repository name. Since this is a manifest+module repository the main development branch should be renamed to the name of this repository. (This means there will not be a branch named "master", and a "zephyr" branch likely will not be meaningful or useful.)
• A module version and manifest version schemes, policies, and query/discovery mechanisms should be set up, since the zephyr version is no longer the manifest version.

[gregshue]

Please remember that the board "unit_testing" requires different build logic in the test files, so the module interface functionality verification needs to include an example of that.

[mbolivar-nordic]

I think this needs to be more than a "simple reference point" of how those features can be used. I want to know how the module is expected to be setup - in practice, a boilerplate.

@gregshue I'm not sure I get you on this one. I don't think we have expectations of our users in the form of "if you don't structure your module in this way, you're doing it wrong" in this case. I am assuming @tejlmand means "simple reference point" as "here is a way to do it that we think works well and can commit to maintaining". (Please LMK if I misunderstood that.)

So a simple reference point is a boilerplate for a "typical" application with an in-tree SoC, but out of tree board, application, and driver for an IC on the board that doesn't have an upstream driver.

Is it possible we are in violent agreement on this?

[mbolivar-nordic]

@gregshue my take on the other bullets:

All(?) the zephyr unique files be kept under the necessary zephyr sub-directory, following the zephyr repo directory structure (e.g. manifests, those discovered by sanitycheck/CI, hello_ sample, west extension scripts, Kconfig files).

-1 from me.

Flat is better than nested in this case, in my opinion. Users can always nest if their use case is suited for that.

Shared namespaces (e.g. include pathnames, CPP symbols, Kconfig vars, etc) be prepended with the module repository org/name, where dashes are replaced with underscores. (It's really too long, but we need to avoid namespace conflicts.)

I think it could be nicer to 'steal' some short names that are unlikely to conflict. I expect most users will change these anyway, to suit their own project naming conventions.

Karim Yaghmour does this in his Embedded Android book's chapters on adding board support for that operating system. He names the board the "coyote pad", and defines namespaces and files based on that. Microsoft does something similar with the fictional "Contoso" corporation in some of their publications. I personally find this practice easy to understand. It's just "foo" in another form.

Some specific ideas:

• board name: plank, so files like boards/arm/plank/Kconfig, etc. (this matches the board porting guide, where "plank" is a stand-in for the custom board name)
• sample name: killer-app, so files like killer-app/src/main.c
• driver: for the chippy device, so files like drivers/sensor/chippy/{chippy.c,chippy.h}, preprocessor header guards VND_CHIPPY_H, Kconfig option CONFIG_CHIPPY, etc.
• devicetree vendor prefix for chippy: vnd, so files like dts/bindings/sensor/vnd,chippy.yaml (vnd is used by the test case bindings as a stand-in for a real vendor)

Test case names would use these as well.

It would be straightforward to tell the story of the development of the new Plank device and its killer application in the documentation. This can also say that internally, the Plank includes an SoC already supported by Zephyr and the VND corporation's chippy sensor, which is not supported upstream. Etc.

Automatic formatting rules are provided for all files beneath the zephyr sub-directory structure (e.g. .clang-format). This will simplify file comparisons against the boilerplate.

Here be dragons; this is far from a settled question in the project, even just for line length.

I suggest that instead, we just make sure the linters that run on code in main zephyr repository also pass on this new module. We can discuss formatters if and when Zephyr itself starts to use them.

A repo-level .gitignore be commented with why each setting is necessary, since some may conflict with pre-existing build tools.

+1

Names/prefixes of tags and the main development branch and tags are recommended to be the manifest repository name. Since this is a manifest+module repository the main development branch should be renamed to the name of this repository. (This means there will not be a branch named "master", and a "zephyr" branch likely will not be meaningful or useful.)

While I am personally in favor of avoiding a main branch named "master", I think this is a project-wide question (which I believe the TSC is already discussing) and this module's main branch name should follow what the zephyr repository uses, to be changed if and when zephyr itself does.

A module version and manifest version schemes, policies, and query/discovery mechanisms should be set up, since the zephyr version is no longer the manifest version.

I don't know that I agree with a module-wide version right now; can you explain why that is worth having? I agree that a version numbering mechanism for killer-app makes good sense, though.

[gregshue]

@mbolivar-nordic , I had not equated "simple reference point" with "boilerplate". With your clarification I think we are in agreement. Thanks for sharing your understanding. (BTW, I never assumed approaches other than this boilerplate were "wrong", just that the downstream maintainer assumes responsibility for designing/implementing/verifying support as module interface features get extended. :-))

Regarding your comments above:

If we don't set up a pattern to manage namespaces, et. al. at scale and beyond the scope of GitHub, then we have not addressed the crux of the problem. Similarly, if we haven't provided an example of how to extend existing repositories to be modules, then we also have not addressed the crux of the problem.

In my experience reuse-and-extend is a much more powerful enabler and paradigm than leverage-and-potential-reuse. I expect users to not rename anything from zephyrproject-rtos. It is much more efficient to live within the existing naming and reuse all applicable analysis and code provenance than to re-apply unnecessary changes over the long support/maintenance lifetime of IoT products (est. 10+ yrs). For boards, samples, drivers, applications (samples/testcases), and device tree settings, there is no need to "rename" anything. ("Renaming" implies removing the original.) Zephyr has already been designed to support extensions for each of these. You can easily copy from an existing implementation of any of these to create an additional one (whether in-tree or out-of-tree), and leaving the original intact makes it easier to track updates and apply relevant fixes.

Here be dragons; this is far from a settled question in the project, even just for line length.

By providing the same .clang-format file as exists in Zephyr, users can choose to enable, change, or disable automatic formatting as they like. It's no different than currently in the zephyr repo. :-)

While I am personally in favor of avoiding a main branch named "master", I think this is a project-wide question (which I believe the TSC is already discussing) and this module's main branch name should follow what the zephyr repository uses, to be changed if and when zephyr itself does.

I have been participating in Process Working Group meetings and monitoring TSC discussions for several weeks, so am aware of the topic. The recommendation for the main branch name to be 'zephyr' came from the PWG, and it makes sense that all existing modules (which happen to be upstream of zephyr) have a main/integration branch name of zephyr. This issue is now introducing downstream modules, which necessarily provide their own manifest. Naming the downstream module's main branch as 'zephyr' doesn't set up a scalable pattern. But recommending that main integration branch names follow the manifest repo name does scale within at least GitHub, and fits with the existing TSC preference. Part of the reason for expressing it here is to help the TSC evaluate their current preference. :-)

I don't know that I agree with a module-wide version right now; can you explain why that is worth having? I agree that a version numbering mechanism for killer-app makes good sense, though

As I see it, one of the huge benefits is Zephyr is a Software Product Line. Zephyr Project is set up for reuse-and-extension, and some of that reuse will come from open source repos outside of zephyrproject-rtos (e.g. Project CHIP). In an SPL, a FW image is simply a set of build configuration settings and a few image-unique values. The vast majority of logic is or will be shared with at least one other image. Building out of the same source branch reduces maintenance across the whole SPL. Consider the case where a user wants to build dozens of different FW images for the same class of from the same branch of code. Or they need to support the same implementation over SOCs from a variety of vendors.

What does that have to do with module version numbers? Anyone wanting to integrate a downstream module into their product will want to use a released version for code stability, issue tracking, and eventually partial analysis for security certification submissions. In this case we are setting up a boilerplate for a downstream module, so we should provide a versioning mechanism tied into the Zephyr ecosystem.

Cheers!

[gregshue]

@mbolivar @tejlmand @carlescufi After looking into how the Zephyr build system currently extracts version information, it seems like the following is all that is needed to tie downstream manifest modules into the Zephyr build system version extraction mechanisms:

• Make sure this downstream repository has an initial annotated tag following the pattern used within the existing manifest repository (zephyr). The build system uses this to initialize the BUILD_VERSION cmake & preprocessor variables and reports it in the boot banner.

  • Q: What should the prefix of the tag be (since zephyr is not the manifest repository)?

• Provide a VERSION file. The build system uses this to initialize the PROJECT_VERSION_* cmake variables.

  • Q: Where should the VERSION file be located?
    • Proposal: module.yml should describe the location of the hardcoded file (or a script to generate the file)? For this repository the hardcoded file or generation script should be at some level beneath the zephyr folder.

These will reveal places where we have assumed ZEPHYR_BASE is also the manifest repository (which is no longer true), and drive distinction back into the zephyr repository.

[gregshue]

Considering module names (and associated namespaces), it would be beneficial if all zephyrproject-rtos downstream modules were easily identified within the name. Perhaps github.com/zephyrproject-rtos/oot_*, with this specific repository being named oot_boilerplate or oot_template?

[gregshue]

@tejlmand We will also need to verify the solution also supports the Forest topology, where the manifest repo contains no business logic. I expect this will require a separate OOT manifest(s)-only repository with:

• multiple west.yml files
  • one importing only zephyr
  • at least one including an OOT module with one or more applications
• VERSION data (unique per manifest file?)

Perhaps this additional repository should be named oot_manifest_only?

[tejlmand]

@tejlmand We will also need to verify the solution also supports the Forest topology, where the manifest repo contains no business logic. I expect this will require a separate OOT manifest(s)-only repository with:

Note, the main point of testing in this case is to ensure that Zephyr modules works.
West and west workspace topology should be tested by west, and not Zephyr.

This is done here:
https://github.com/zephyrproject-rtos/west/tree/master/tests

[gregshue]

@tejlmand The Zephyr build system also needs to be verified to work in different topologies, notably when zephyr is not the manifest repository. This is true whether the manifest repository has any prj.conf or is configured from an alternate manifest file. Since Zephyr explicitly supports the Forest topology separate from the Star topology (w/o Zephyr as manifest), an example of a manifest-only repository seems appropriate. This is especially important if the overall FW version or BUILD_VERSION should map to a distinct manifest.

[tejlmand]

The Zephyr build system also needs to be verified to work in different topologies, notably when zephyr is not the manifest repository.

@gregshue you seem to miss the point that west is an optional tool.
https://docs.zephyrproject.org/latest/guides/modules.html#without-west

What this issue should focus on is that the Zephyr module infrastructure works correctly.

That is, ensuring ZEPHYR_MODULES works.
One way to obtain a list of modules, is by using west list, and therefore it makes sense to combine this into a repo that can also work as a simple reference point, as that will serve two purposes:

• Makes it easier to integrate into CI
• Show to users a simple way they can structure their project

Doing testing of other typologies is therefore outside the scope of this.

[tejlmand]

In this case we are setting up a boilerplate for a downstream module, so we should provide a versioning mechanism tied into the Zephyr ecosystem.

Zephyr is already calling into the module's CMake build system.
That build system can decide how it wants to handle versioning.
It could decide to use a VERSION:
https://github.com/nrfconnect/sdk-nrf/blob/master/VERSION

or it could decide to fetch the version info using git (if available), e.g.:
git describe --tags

If we start imposing a version field in the module.yml, then we force users to update that field.
As long as Zephyr itself doesn't use the field, I see no reason to introduce this, and definitely not as part of this task.
(Feel free to open a new enhancement suggestion for this/)

Considering module names (and associated namespaces), it would be beneficial if all zephyrproject-rtos downstream modules were easily identified within the name. Perhaps github.com/zephyrproject-rtos/oot_*, with this specific repository being named oot_boilerplate or oot_template?

I don't think this is the right issue for such discussion.

These will reveal places where we have assumed ZEPHYR_BASE is also the manifest repository

west is an optional tool, so Zephyr knows nothing about being a manifest repo.

Shared namespaces (e.g. include pathnames, CPP symbols, Kconfig vars, etc) be prepended with the module repository org/name, where dashes are replaced with underscores. (It's really too long, but we need to avoid namespace conflicts.)

I think it could be nicer to 'steal' some short names that are unlikely to conflict. I expect most users will change these anyway, to suit their own project naming conventions.

agree with @mbolivar-nordic here.
Especially as usersin some cases might want to uses same names as another module in order to replace/overrule that module.
As example, they want to use the same set of MBEDTLS Kconfig names as Zephyr does, as this allows them to override default menus/default settings.

[gregshue]

@tejlmand , Thank you for your feedback. I absolutely agree that ZephyrProject specifies that west is optional.

What this issue should focus on is that the Zephyr module infrastructure works correctly.

I think you have recognized a general need that is tangential to this ticket. This ticket is about creating an example downstream module with simple business logic and covers the breadth of points of extensions and is a suitable pattern for extending existing 3rd-party open source repositories (e.g. https://github.com/project-chip/connectedhomeip , https://github.com/bacnet-stack/bacnet-stack, https://github.com/micro-ROS/micro_ros_zephyr_module).

As such, this ticket will set precedence (and likely direction) for the following:

• Downstream/OOT naming pattern for modules owned by zephyrproject-rtos
• Namespace management pattern for Downstream/OOT modules (for symbols in Kconfig, CMake, C pre-processor, linker, include paths, shell, logging, etc.)
• Directory structure organization
• Documentation organization
• How the Zephyr build system natively supports versioning of downstream/OOT projects

Why consider versioning? Because PROJECT_VERSION* does not refer to the image built from an out-of-tree prj.conf.

[gregshue]

I have created a proposed implementation for this example at https://github.com/gregshue/oot_module_example. It does not have a board, driver, or dts implementation, but provides the structure and documentation to hold them. I would submit this as a PR but the Zephyr Project repository has not yet been created.

I will post an RFC for this proposal below.

@carlescufi @tejlmand Is there anything else needed to move this along?

[gregshue]

At this point it is stalled waiting for alignment on the repository naming pattern, then for the empty repositories to get created to enable a PR with my draft content.

@cfriedt, I just looked at the Greybus for Zephyr staging branch. Before you release this to production you should look at https://github.com/gregshue/oot_module_example.

[cfriedt]

@cfriedt, I just looked at the Greybus for Zephyr staging branch. Before you release this to production you should look at https://github.com/gregshue/oot_module_example.

Ah, interesting - I'll take a look at that. I just hacked something together last night independently of the Greybus repo. It looks similar-ish to yours with the exception that you have everything within the zephyr directory.

https://github.com/cfriedt/zephyr-example-module

I'll take a look at your repo as well.

[gregshue]

@cfriedt Everything Zephyr-unique is kept beneath the zephyr/ subdirectory to allow extending any existing repository to also be a Zephyr module simply by copying this in and filling in the glue. In order to avoid a top-level west.yml I had to use west version >= 0.8a1. I expect my CMakeLists.txt files could be improved. :-)

I applied a non-documented version of this in the zephyr branch of https://github.com/bacnet-stack/bacnet-stack and migrated all the unit tests to ztest. sanitycheck nicely ran with code coverage metrics.

Thanks for sharing, and let me know how to improve mine.

[carlescufi]

Proposal for CI:

1. The west.yml in the external repo points to a release tag of zephyr, and the repo itself is only tested against that last release, on-PR to the external repo. Who would be responsible for updating the external repo to the latest release when an -rc1` is tagged?
2. We integrate building the external repo in our on-PR CI in the main zephyr tree, by adding something similar to:

zephyr-example-module/west.yml:
  projects:
    - name: zephyr
      remote: zephyrproject-rtos
      revision: master
      import: true
  self:
     path: example-repo

west init -m https://github.com/zephyrproject-rtos/zephyr-example-module
west update
west build app

Since we do not want the external repo to be part of upstream zephyr's west.yml (that would be "cheating"), we need to figure out a strategy of keeping the revision of the last known working external repo revision somewhere, or just testing the external repo master branch against zephyr's master branch.
Note: the workflow for contributors will be more complicated if the external repo build breaks.

[galak]

My suggestion for a name would be either example-application or example-project

[galak]

dev-review (Dec 17 2020):

• Suggestion is to just have a "daily" ci in the new repo
• Suggestion is that this repo is setup for multiple "applications"

[gregshue]

@galak Right now the Zephyr directory tree structure has conf.prj files (arguably "applications") under tests and samples. Would it make sense to add a parallel projects tree for non-test, non-sample conf.prj et. al., to live?

[gregshue]

Is it appropriate for this repository to also be a regressive test case to verify that the west -m <> --mr <> and --mf <> options continue to work?

[nashif]

Just for illustration of the various use cases:

1. the upstream model:

2. a project (product tree) that is a zephyr module as well:

3. a project (product tree) that is not a zephyr module:

[mbolivar-nordic]

@nashif are the pictures where module A and module B appear in the zephyr manifest and the product manifest meant to show that those modules are imported, or that they are being overridden?

Also, just for reference, here is a table showing how different repositories that may be involved in the build can be set up and relate to the build system:

West project?	Zephyr module?	Notes or examples
N	N	Manually created application repository
N	Y	Set up via ZEPHYR_MODULES or ZEPHYR_EXTRA_MODULES
Y	N	Does NOT have zephyr/module.yml file or zephyr/CMakeLists.txt+zephyr/Kconfig files
Y	Y	DOES have zephyr/module.yml file or zephyr/CMakeLists.txt+zephyr/Kconfig files

As you can see, they really are orthogonal, and all four combinations are possible.

[nashif]

@nashif are the pictures where module A and module B appear in the zephyr manifest and the product manifest meant to show that those modules are imported, or that they are being overridden?

via import, so they are in the zephyr project west yaml file.

[nashif]

oh, I think I should use different names :-)

[nashif]

oh, I think I should use different names :-)

fixed now.

[mbolivar-nordic]

Your pictures still show west projects in a different color than modules, but any repository can be either or both.

[mbolivar-nordic]

I still also see "west project A" in the picture for "a project (product tree) that is a zephyr module as well". It's still a cause for confusion for me. That could mean that the project west.yml overrides that project, or that it imported it.

[nashif]

Your pictures still show west projects in a different color than modules, but any repository can be either or both.

yes, modules can be shaded to signify they are also projects.

[gregshue]

@nashif, Thank you for trying to describe this with a graphic. I assume you are describing the roll of each repository in each use case rather than the capabilities of each repository (since Zephyr always has a manifest file even if it is not the one referenced in the west workspace. Consider showing both what the repository contains and how it is used.

As I have worked with it, it is quite practical for every downstream module to also have a manifest file for each Zephyr fork it is verified against, and to use separate branches tracking different Zephyr versions. Perhaps one of the downstream modules in each use case should be noted as manifest module being used as just a module?

[nashif]

As I have worked with it, it is quite practical for every downstream module to also have a manifest file for each Zephyr fork it is verified against, and to use separate branches tracking different Zephyr versions. Perhaps one of the downstream modules in each use case should be noted as manifest module being used as just a module?

if I understand you correctly, this is case (2)

[gregshue]

Yes, and no.

When I am working on extending https://github.com/bacnet-stack/bacnet-stack to also be a Zephyr module, the bacnet-stack repository is Product XYZ in Case 2. When I am integrating the bacnet-stack module into my product, the bacnet-stack is Module C in either Case 2 and Case 3 (whether or not any of bacnet-stack's manifest files are imported).

Part of the need for this new repository is to have a Zephyr Build System testcase for Case 2. Can we verify Module C in Case 2 is properly handled by the Zephyr Build System with only this repository and the Zephyr repository?

[gregshue]

I think we will want to document this in terms of the source code topologies supported by west, and be a bit more explicit about module.yml files, DTS files, multiple projects, and Zephyr Project hosted repos vs external ones. How about this?

[carlescufi]

Is it appropriate for this repository to also be a regressive test case to verify that the west -m <> --mr <> and --mf <> options continue to work?

Yes, CI for the new repo should do that

[gregshue]

zephyrproject-rtos/example-application#1 submitted.