Linkerd Add-Ons

[Pothulapati]

Linkerd Add-Ons

Goals

• To Have a consistent exerience to install stuff along with Linkerd, that add/extend functionality.
• To provide an out of box experience, with limited Knobs for popular use-cases.

Non-Goals

• To have a scalable/maintanable experience for the add-on.
• To allow full blown configurable experience for the add-on.

All of the above cases, can always be done by using the official helm-charts/operators of those components.

Example Add-ons

• Jaeger and OC-Collector as add-on for tracing to show the tracing functionality.
• k8s-prometheus adapter, relevant config crds as an add-on to show the Scaling based on latency use-case.
• Grafana can also be made as an add-on but a default one.
• OPA can also be configured as an add-on for policies of resources.
• SMI Metrics API Server as an add-on.

Why not use Helm Charts as dependencies?

This can remove the hassle of having them in our code-base but introduces the following problems

• Proxy Injection into the components would be hard, as the dependency installation would happen along with Control plane.
• Gives us very limited experience to tweak configuration, and makes it very hard to configure common data across charts.

Procedure

• There will be add-ons in charts/add-ons/tracing, where each chart has its own templating, values and structure.

• All charts will be mentoned as dependencies, to linkerd.

• Add-ons will be optional using [conditional field](https://v2.helm.sh/docs/developing_charts/ #tags-and-condition-fields-in-requirements-yaml) global.Add-ons.<add-on>.Enabled.

• The same global.Add-ons.<add-on>.Enabled can be used to toggle stuff in the linkerd parent chart based on the component is present or not. Ex --control-plane-trace flag in control-plane components can be toggled based on global.Add-ons.tracing.Enabled

• Each Sub-Chart dosen't have to depend on partials again, as Linkerd depends on that, and helm would make that automatically available for other siblings charts i.e add-ons in this case.

• Each Sub-Chart can be made configurable by using global.Add-ons.<add-on>.property rather than using it's own Values.yaml, to make configuration also available using linkerd install.

• Values can be passed between linkerd and sub-charts by:

  • Linkerd -> Add-On :

    • Used to send common properties across all sub-charts like .Proxy, .Namespace, etc.
    • This can be done by using .Values.global.<name>

  • Add-On -> Linkerd :

    • Used to configure add-ons meta-data in Linkerd components. Ex: Getting the Trace Collector address out from add-on to Linkerd to configure the control-plane components and proxies on where to send the trace data.
    • In Helm2, there isn't a direct way of doing this but this shouldn't be a problem as we can statistcally bind to common values using Values.global.Add-Ons.<name>.property.

• Upgrades would work automatically, as the install cli flags are stored in the config map automatically.

Pre-requisite Changes Required

• Move Common values required by multiple sub-charts like .Values.Proxy.*, etc to .Values.global.Proxy.*, etc. This would need changes in linkerd install, and injector code as they re-use the same templates.

I'm working on the above mentioned changes at https://github.com/linkerd/linkerd2/compare/master...Pothulapati:add-ons?expand=1
This contains the pre-requisite helm changes. Once I get the linkerd install, and inject changes, will open a draft PR. Looking for feedback till then.

@ihcsim @grampelberg @alpeb

[alpeb]

@Pothulapati The Helm approach sounds great to me 👍

I guess chart addons would live in their own separate github repos, and we'd add them to our Helm repo at a place like https://helm.linkerd.io/addons. So we'd refer directly to that repo in requirements.yaml and wouldn't need to actually copy them into /charts/add-ons. For development we can just point to a local directory where we have checked out the addon.

But thinking about the CLI side of things, we might actually need access to the files of those subcharts, just like we currently consume the linkerd2 chart in order to build the manifests. So we could either pull them and include them when building the release, or maybe better, pull them dynamically during linkerd install|upgrade if needed (or through some other preliminary subcommand to prepare the addons).

You said you had some ideas about the CLI part. Do you mind opening an issue? I think it's worth thinking on both the Helm and CLI parts in parallel 🤔

[grampelberg]

I guess chart addons would live in their own separate github repos

We should be explicit about this decision. I've been assuming they'd all be in the same repo. That'll make contribution more difficult though. What would we need to get put together so that these addons can live in separate repos but still have a consistent look and feel? I'm thinking we'll want some kind of common CI toolbox at the bare minimum.

I'm of the opinion that we should be concrete about what we'd like to do for splitting Grafana and Prometheus out and go from there.

[Pothulapati]

@Pothulapati The Helm approach sounds great to me +1

I guess chart addons would live in their own separate github repos, and we'd add them to our Helm repo at a place like https://helm.linkerd.io/addons. So we'd refer directly to that repo in requirements.yaml and wouldn't need to actually copy them into /charts/add-ons. For development we can just point to a local directory where we have checked out the addon

But thinking about the CLI side of things, we might actually need access to the files of those subcharts, just like we currently consume the linkerd2 chart in order to build the manifests. So we could either pull them and include them when building the release, or maybe better, pull them dynamically during linkerd install|upgrade if needed (or through some other preliminary subcommand to prepare the addons).

IMO, they should just live under charts/add-ons/<addon-name>, because they are useless without having the main Linkerd chart, or the Linkerd installation. This definitely helps us to not have to maintain those charts, but they still have to add a reference in the requirements.yaml, pkg/charts/values.go and values.yaml. I'm not really sure about this because, helm would just work as the .repository can even be a URL but this would complicate the Linkerd install tool, as it has to then download them.

At least we should initially focus on the local file-path add-ons and later try to move them out-of-tree if needed.

[Pothulapati]

So, Add-on Model is planned to be done using helm charts as the underlying tool.
The following per-requisites have to be considered considered because of that:

• Sub-charts will live under charts/add-ons as new Helm charts with similar format and templating. As helm supports URL as the repo path for add-ons, They can also later be moved out-of-tree totally.
• The values of the sub-charts will be overriden by using the name of the add-on field in the parent's i.e linkerd2's values.yaml. These values are considered the defaults and can be overriden using cli or a new values file. charts/add-ons/grafana/values.yaml can left out empty so that we can maintain the config centrally.
• The requirements.yaml will be updated to have the dependencies.
  Ex:

dependencies:
- name: partials
  version: 0.1.0
  repository: file://../partials
- name: grafana
  version: 0.1.0
  repository: file://../add-ons/grafana
  condition: grafana.enabled 

The condition field is used like a toggle, to set the default value. If the condition is false, the sub-charts are not included. This field will point to the enabled field of the corresponding add-on in values.yaml. It's important that all the add-ons follow the same model.

• If the installation is done through Helm, it handles upgrades, etc everything.

Now, we have to replicate the same model using the linkerd CLI. like we already do similar stuff for linkerd charts. The following aspects are to be considered:

• Add-On Config

How does the CLI know about the templates of the sub-charts? Should it also maintain them? or just get those details from requirements.yaml?

IMO, We should have a similar config for add-ons in values.go, rather than reading from requirements.yaml about the add-on.
Example:

type (
	// Values contains the top-level elements in the Helm charts
	Values struct {
		Stage                       string            `json:"stage"`
		ControllerImage             string            `json:"controllerImage"`
        .
        .
        .

		// Add-Ons For Linkerd
		Grafana *add_ons.Grafana `json:"grafana"`
	}

This follows the already exisitng pattern of having config in values.yaml, and relevant fields in values.go. Thus allowing us to get the defaults, perform over-writes, and storing those configs for upgrades.

• Overriding Values

The Values struct will have the default values.

By Helm, Users can always pass a new values.yaml and override the defaults. This overriden values file can contain the fields of the add-ons and will work the same.

For Linkerd Install, How should the add-on values be over-written? I was thinking, we should allow users to perform linkerd install --grafana values.yaml. We will use this yaml and overwrite the default values.

Is there a better approach rather than passing values? We could do linkerd install --grafana --grafana.image,etc but this would add a lot of installOptions.WDYT?

• Handling Upgrades

The add-ons Values should be stored somehewhere. So that we can retrieve and update them, for handing upgrades. This can be done by adding another field in linkerd-config cm.
Something Like

Data
====
global:
----
...

install:
----
{"cliVersion":"edge-19.12.3","flags":[]}

proxy:
----
{"proxyImage":{"imageName":"gcr.io/linkerd-io/proxy","pullPolicy":"IfNotPresent"},"proxyInitImage":{"imageName":"gcr.io/linkerd-io/proxy-init","pullPolicy":"IfNotPresent"},"controlPort":{"port":4190},"ignoreInboundPorts":[],"ignoreOutboundPorts":[],"inboundPort":{"port":4143},"adminPort":{"port":4191},"outboundPort":{"port":4140},"resource":{"requestCpu":"","requestMemory":"","limitCpu":"","limitMemory":""},"proxyUid":"2102","logLevel":{"level":"warn,linkerd2_proxy=info"},"disableExternalProfiles":true,"proxyVersion":"edge-19.12.3","proxyInitImageVersion":"v1.2.0"}

add-ons:
---
{"name": "grafana", "image":""}{}

For Upgrades, this config-map will be parsed and all the add-ons that are present will be overriden with this config.

• Handling HA for add-ons

Add-Ons can also be highly-available. This can just work by specificying relavant values for add-ons in values-ha.yaml. The same will work by using --ha flag through cli.

@grampelberg @alpeb @zaharidichev

[Pothulapati]

After talking to @grampelberg,
For overriding values rather than having a config file for each add-on. They should be able to pass a single config file for full configuration i.e both linkerd2 and add-ons.
Something like linkerd install --config config.yaml. This will contain all the configuration following the same helm values format.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.