Helm charts repository index.yaml retention policy #10539

carrodher · 2022-06-02T08:49:33Z

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml.
This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

One of the alternatives considered was the use of compression at CloudFront, in that case, this solution doesn’t work since compression is not used by the Helm client (helm) itself (see helm/helm#8070) so it doesn’t solve the reported issue.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

⚠️ Please note this action is not removing/deleting any Helm chart, packaged tarballs (.tgz) won't be removed, this action is only affecting index.yaml used to list the Helm charts. Previous versions of the index.yaml can be used to install old Helm charts.

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

🔧 Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml.

Manually using helm repo add

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

When used as a dependency in Chart.yaml:

  - name: postgresql
    version: 8.1.0
-   repository: https://charts.bitnami.com/bitnami
+   repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
    condition: postgresql.enabled

The text was updated successfully, but these errors were encountered:

DmytroDrachov · 2022-06-02T11:08:22Z

Thank you very much

sisrael-dn · 2022-06-02T11:48:42Z

Hi,
https://charts.bitnami.com/bitnami seems to return 403 AccessDenied error at the moment.
Is this related to this issue?
My guess is that if you work with S3 bucket as CloudFront origin, the new index.yaml got the wrong ACL now (not publicly readable)

They've deleted the versions we use from the latest index. More details: bitnami/charts#10539

Bitnami has removed packages older than 6 months from their helm repo , breaking the build. We're in the middle of a release, enable workaround to unblock us. Eventually we want to get rid of the bitnami chart for memcached anyway, see issue 1315 See: bitnami/charts#10530 bitnami/charts#10539 #1315

Bitnami has removed packages older than 6 months from their helm repo , breaking the build. We're in the middle of a release, enable workaround to unblock us. Eventually we want to get rid of the bitnami chart for memcached anyway, see issue 1315 See: bitnami/charts#10530 bitnami/charts#10539 #1315 Signed-off-by: György Krajcsovits <[email protected]>

potiuk · 2022-06-02T13:31:02Z

This is TERRIBLY wrong. For example, we released Helm Chart to our users https://airflow.apache.org/docs/helm-chart/stable/index.html with bitnami's Postgres 10.5.3 and all the charts our users are using are useless now if they use embedded Postgres.

How is it possible you do such a breaking change in such a way ? This must be some kind of a joke.

This totally undermines any trust in bitnami as a source of charts - You basically rendered all historical charts released by likely huge number of projects useless.

We started to experience "Internal Error" when installing Helm chart and apperently bitnami "solved" the problem by removing from their index software older than 6 months(!). This makes our CI fail but It is much worse. This renders all our charts useless for people to install This is terribly wrong, and I raised this in the issue here: bitnami/charts#10539 (comment)

Bitnami has removed packages older than 6 months from their helm repo , breaking the build. We're in the middle of a release, enable workaround to unblock us. Eventually we want to get rid of the bitnami chart for memcached anyway, see issue 1315 See: bitnami/charts#10530 bitnami/charts#10539 #1315 Signed-off-by: György Krajcsovits <[email protected]>

We started to experience "Internal Error" when installing Helm chart and apperently bitnami "solved" the problem by removing from their index software older than 6 months(!). This makes our CI fail but It is much worse. This renders all our charts useless for people to install This is terribly wrong, and I raised this in the issue here: bitnami/charts#10539 (comment)

bschurig · 2022-06-02T14:03:09Z

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:
Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122
Producing a reduced 3.5MB index.yaml.

Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml, for example:
$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

It would be fantastic if someone could get this added to the readme.

see: bitnami/charts#10539

GMZwinge · 2022-06-02T14:31:37Z

Instead of removing all versions older than 6 months, would it be possible to keep the last major/minor versions older than 6 months, eg: mongodb-9.2.6.

They've deleted the versions we use from the latest index. More details: bitnami/charts#10539

* k8s: Work around bitnami removing old charts see: bitnami/charts#10539 * disable line-length Co-authored-by: toan <[email protected]>

fmonpelat · 2023-04-26T20:09:59Z

hi, i'm getting not found with this link: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami is there another one? I need an old version and reading, it will not be in OCI...

agomezmoron · 2023-04-27T06:28:28Z

Hi @fmonpelat, did you run?

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

Remember you have to use that to add the new repo. The URL https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami will give you always a 404 error code because when you add the repo, what it's really using is the index.yaml located at https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami/index.yaml.

Please check that and let us know if you keep having that issue.

Cheers

sushmithan2 · 2023-07-25T07:50:17Z

Not able to find mongo chart version 10.20.0 in bitnami repo

…s in the chart (#18941) * Add ability to specify an existing Secret (#18139) Closes #18139 * Allow using secrets with external postgres * Upgrade CronJob to batch/v1 * Allow using redis.auth.existingSecret * Helmignore mastodon-*.tgz for easy local development * Upgrade helm dependencies * Upgrade postgresql to 11 * Allow putting SMTP password into a secret * Add optional login to SMTP secret This to allow setting LOGIN either in values.yaml or in the secret. * Switch to bitnami charts full archive This prevents older versions from disappearing, see bitnami/charts#10539 for full context. Co-authored-by: Ted Tramonte <[email protected]>

github-actions · 2023-08-10T01:27:31Z

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions · 2023-08-16T01:20:06Z

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

Workaround for bitnami/charts#10539

Moglum · 2024-06-20T08:43:47Z

The index file at https://repo.broadcom.com/bitnami-files/index.yaml has over 15MB again, and we're getting Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://repo.broadcom.com/bitnami-files/index.yaml": EOF
Is the size expected or is there some problem with the pruning of the index?
Thank you

carrodher changed the title ~~Helm charts retaining policy~~ Helm charts retention policy Jun 2, 2022

carrodher pinned this issue Jun 2, 2022

carrodher changed the title ~~Helm charts retention policy~~ Helm charts repository index.yaml retention policy Jun 2, 2022

This was referenced Jun 2, 2022

helm repo add bitnami gives not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR #8433

Closed

[bitnami/redis] Old chart version deleted? #10542

Closed

stefanprodan mentioned this issue Jun 2, 2022

Enable automated upgrades for Bitnami charts fluxcd/flux2-kustomize-helm-example#66

Merged

thisislawatts mentioned this issue Jun 2, 2022

Old charts not available #10545

Closed

akshaymankar added a commit to wireapp/wire-server that referenced this issue Jun 2, 2022

Use old bitnami helm repo from git for redis and postgresql

42e8a04

They've deleted the versions we use from the latest index. More details: bitnami/charts#10539

akshaymankar mentioned this issue Jun 2, 2022

Use old bitnami helm repo from git for redis and postgresql wireapp/wire-server#2448

Merged

4 tasks

krajorama mentioned this issue Jun 2, 2022

Workaround for bitnami releasing mimir with old memcached grafana/helm-charts#1438

Merged

potiuk mentioned this issue Jun 2, 2022

Revert "Use github for postgres chart index (#24089)" apache/airflow#24109

Closed

potiuk mentioned this issue Jun 2, 2022

Temporarily fix bitnami index problem apache/airflow#24112

Merged

potiuk mentioned this issue Jun 2, 2022

Workflows assets & system tests migration (AIP-47) apache/airflow#24105

Merged

potiuk mentioned this issue Jun 2, 2022

Frequent failures of helm chart tests apache/airflow#24037

Closed

2 tasks

tarrow added a commit to wmde/wbaas-deploy that referenced this issue Jun 2, 2022

k8s: Work around bitnami removing old charts

a75c392

see: bitnami/charts#10539

tarrow mentioned this issue Jun 2, 2022

k8s: Work around bitnami removing old charts wmde/wbaas-deploy#364

Merged

akshaymankar added a commit to wireapp/wire-server that referenced this issue Jun 2, 2022

Use old bitnami helm repo from git for redis and postgresql (#2448)

2df962f

They've deleted the versions we use from the latest index. More details: bitnami/charts#10539

toban added a commit to wmde/wbaas-deploy that referenced this issue Jun 2, 2022

k8s: Work around bitnami removing old charts (#364)

0de1d30

* k8s: Work around bitnami removing old charts see: bitnami/charts#10539 * disable line-length Co-authored-by: toan <[email protected]>

calohmn mentioned this issue Jun 2, 2022

Ditto / HawkBit: dependent charts cannot be resolved eclipse/packages#372

Closed

This was referenced Apr 30, 2023

Update examples to use newer postgres and redis versions in Bitnami helm charts garden-io/garden#4141

Closed

Fix CI "not found" errors relating to old bitnami charts garden-io/garden#4142

Closed

williamlardier mentioned this issue May 17, 2023

Bugfix/zenko 4583 reconfigure mongodb rollback time scality/Zenko#1870

Merged

carrodher mentioned this issue Jun 2, 2023

can not run mongodb 4.4.10 #11121

Closed

kibahop mentioned this issue Jun 12, 2023

Update dependencies absmach/devops#125

Closed

github-actions bot added triage Triage is needed and removed on-hold Issues or Pull Requests with this label will never be considered stale labels Jul 25, 2023

This was referenced Jul 26, 2023

[bitnami/harbor] harbor Chartmuseum missing #17851

Closed

[bitnami/mongodb] Not able to find mongo chart version of 10.20.0 #17864

Closed

github-actions bot added the stale 15 days without activity label Aug 10, 2023

github-actions bot added the solved label Aug 16, 2023

bitnami-bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 16, 2023

rlratcliffe mentioned this issue Sep 3, 2023

Resolve #101 - Support K8s 1.22 - 1.27 MS3Inc/tavros#102

Open

danielskowronski mentioned this issue Oct 31, 2023

Fix issues with dependencies when using external Zookeeper G-Research/charts#118

Open

billybrona added a commit to billybrona/symfony-demo that referenced this issue Nov 26, 2023

bitnami charts workaround

5c399f1

Workaround for bitnami/charts#10539

etoensfeldt mentioned this issue Dec 20, 2023

Update bitnami repository for the mongodb dependency OPC-Router/helm-charts#6

Merged

erlandf mentioned this issue May 6, 2024

fix: use correct bitnami repository jenkins-x/cd-indicators#20

Merged

carrodher mentioned this issue Jun 20, 2024

The index file has over 15MB again #27470

Closed

bitnami locked as resolved and limited conversation to collaborators Jun 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm charts repository index.yaml retention policy #10539

Helm charts repository index.yaml retention policy #10539

carrodher commented Jun 2, 2022 •

edited

Loading

DmytroDrachov commented Jun 2, 2022

sisrael-dn commented Jun 2, 2022

potiuk commented Jun 2, 2022

bschurig commented Jun 2, 2022

Current situation

Mitigation

Result

Workaround for previous versions

GMZwinge commented Jun 2, 2022

fmonpelat commented Apr 26, 2023 •

edited

Loading

agomezmoron commented Apr 27, 2023 •

edited by carrodher

Loading

sushmithan2 commented Jul 25, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 16, 2023

Moglum commented Jun 20, 2024

Helm charts repository index.yaml retention policy #10539

Helm charts repository index.yaml retention policy #10539

Comments

carrodher commented Jun 2, 2022 • edited Loading

Current situation

Mitigation

Result

🔧 Workaround for previous versions

DmytroDrachov commented Jun 2, 2022

sisrael-dn commented Jun 2, 2022

potiuk commented Jun 2, 2022

bschurig commented Jun 2, 2022

Current situation

Mitigation

Result

Workaround for previous versions

GMZwinge commented Jun 2, 2022

fmonpelat commented Apr 26, 2023 • edited Loading

agomezmoron commented Apr 27, 2023 • edited by carrodher Loading

sushmithan2 commented Jul 25, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 16, 2023

Moglum commented Jun 20, 2024

carrodher commented Jun 2, 2022 •

edited

Loading

fmonpelat commented Apr 26, 2023 •

edited

Loading

agomezmoron commented Apr 27, 2023 •

edited by carrodher

Loading