FIX entrypoint.sh preventing regular and legitimate use-cases #495
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit fixes the entrypoint.sh script in order to comply to the user provided configurations. If they provided AIRFLOW__ environment variables, the entrypoint.sh *must not* override them. It also adds a new PostgreSQL configuration variable POSTGRES_EXTRAS in order to enable TLS encryption. See the Airflow documentation "PostgreSQL Connection" about this. It also adds two new Redis configuration variables REDIS_PROTO which was previously hardcoded to redis:// (in order to prepare the possible use of rediss://), and REDIS_DBNUM which was previously hardcoded to 1 (that prevents any use of a mutual broker between Airflow instances). It also adds the corresponding documentation about PostgreSQL and Redis variables.
vpavlin
referenced
this pull request
in opendatahub-io-contrib/docker-airflow
Apr 15, 2020
* Bump airflow version (#2) (#304) * Bump Airflow Version to 1.10.2 * Fix CI * Bump Airflow version to 1.10.3 (#355) * Bump airflow 1 10 3 (#3) * use the user home as airflow home * remove unused var env * set default config & remove duplicated line * Pin Debian version to prevent unintended OS upgrade (#401) * Bump to Airflow 1.10.4 and Python 3.7 * starting scheduler for SequentialExecutor (#359) * Bump to Airflow 1.10.6 * circleci - remove docker_layer_caching * remove circleci * Create ci.yml * Update README.md * Fix bug with kubernetes executor (#470) * Bump to Airflow 1.10.7 * FIX entrypoint.sh preventing regular and legitimate use-cases (#495) This commit fixes the entrypoint.sh script in order to comply to the user provided configurations. If they provided AIRFLOW__ environment variables, the entrypoint.sh *must not* override them. It also adds a new PostgreSQL configuration variable POSTGRES_EXTRAS in order to enable TLS encryption. See the Airflow documentation "PostgreSQL Connection" about this. It also adds two new Redis configuration variables REDIS_PROTO which was previously hardcoded to redis:// (in order to prepare the possible use of rediss://), and REDIS_DBNUM which was previously hardcoded to 1 (that prevents any use of a mutual broker between Airflow instances). It also adds the corresponding documentation about PostgreSQL and Redis variables. * Bump to Airflow 1.10.8 * Fix issue between flask-admin and werkzeug 1.0.0 * Bump to Airflow 1.10.9 * Debian base image - Move from 3.7-slim-stretch to 3.7-slim-buster * user Airflow in group root * user Airflow in group root * user Airflow in group root * testing entrypoint from barneys/docker-airflow * change sql host to pc-base-sql * bug fixes * added ./profile to propogate db connection string * Make load examples False Co-authored-by: Med <[email protected]> Co-authored-by: Jesus Abarca <[email protected]> Co-authored-by: Puckel_ <[email protected]> Co-authored-by: nsepetys <[email protected]> Co-authored-by: Kevin O'Riordan <[email protected]> Co-authored-by: Noël Bardelot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In my opinion, the behaviour of the project should be to help people start with an easy-to-use compose file, and then, allow advanced configuration by letting the user provide
AIRFLOW__variables to the containers.Typically, that would be done using a configmap in Kubernetes (one configmap allowing to configure several containers in the pod). And in fact, that's what is documented in the stable Airflow Helm chart:
https://github.com/helm/charts/tree/master/stable/airflow
But currently the containers entrypoint.sh script overrides three important variables even when the user provides them:
AIRFLOW__CORE__SQL_ALCHEMY_CONN,AIRFLOW__CELERY__BROKER_URLandAIRFLOW__CELERY__RESULT_BACKEND...--
This commit fixes the entrypoint.sh script in order to comply to the user provided
configurations. If they provided
AIRFLOW__environment variables, theentrypoint.shmust not override them.
It also adds a new PostgreSQL configuration variable
POSTGRES_EXTRASin order to enableTLS encryption. See the Airflow documentation "PostgreSQL Connection" about this.
It also adds two new Redis configuration variables
REDIS_PROTOwhich was previously hardcodedto
redis://(in order to prepare the possible use ofrediss://), andREDIS_DBNUMwhich waspreviously hardcoded to
1(that prevents any use of a mutual broker between Airflow instances).It also adds the corresponding documentation about PostgreSQL and Redis variables.