Update docs for v2.4: Roadmap, new Schema to CLI generation, Ansible plugin

docs/configuration-base.md

@@ -17,8 +17,13 @@ $tool
 * `cli` - object that contains CLI arguments
 * `options` - an object that offers additional control over how a tool executes
 
+## Environemnt Variables Defaulting
+Plugin environment variables are automatically mapped to the `bitops.config.yaml` keys. This means that you can use the same environment variable names as in the plugin config. For example, if you want to override the `ansible.cli.skip-tags` value, you can use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. In this case, `BITOPS` is the prefix, `ANSIBLE` is the plugin name, and `SKIP_TAGS` is the key name (note hypens are replaced with underscores).
+
+The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults. This way, ENV variables specified by user are taking highest precedence over config values and defaults. We recommend using them dynamically in CI/CD pipelines to control the deployment based on condition (PR run, branch, etc) or manually passing to the the BitOps docker container.
+
 ## Arbitrary Environment Variables
-During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. 
+During the docker run command, you can specify an ENV var and it will be accessible during all processing stages of BitOps. This is useful for passing in secrets or other custom values used in the workflows.
 
 ## Common Configuration
 There are some global configuration options that are shared among all tools and cloud providers during a BitOps run. These are set via environment variables

docs/development/local-plugin-creation.md

@@ -1,9 +1,9 @@
-# Local plugin development guide
+# Plugin creation guide
 
 So you wanna build a BitOps plugin, eh?  Follow along for how to do it locally!
 
 ## 1. Create a plugin repo
-```
+```sh
 cd /path/to/bitops-plugins
 mkdir my-plugin
 cd my-plugin
@@ -16,74 +16,85 @@ More information on what goes in a plugin [here](../plugins.md).
 For this example, we'll keep to a really simple plugin.
 
 To start with, create a file called `bitops.schema.yaml` and add the following content.
-```
+```yaml
 duplicate-environment:
     type: object
     properties:
+      # CLI properties will be composed into a string with
+      # arguments and exported as "${BITOPS_MY_PLUGIN_CLI}"
+      cli:
+        type: object
+        properties:
+          # positional argument, because no `parameter` is set
+          # ex: "run"
+          command:
+            type: string
+            default: run
+            required: true
+          # typical cli argument
+          # ex: "--key=value"
+          key:
+            parameter: key
+            type: string
+          # cli flag, added if value is set to `true`
+          # ex: "--bar"
+          bar:
+            parameter: bar
+            type: boolean
+            default: true
       options:
         type: object
         properties:
           foo:
             type: string
             export_env: DUPLICATE_ENVIRONMENT_FOO
             required: true
-            default: bar
+            default: foo_value
 ```
 
-Next, create a simple `deploy.sh` script.  This script does some checks and shows how to use some of the available environment variables then outputs a configuration value defined by the `bitops.schema.yaml`.
-```
+Next, create a simple `deploy.sh` script.  This script does some checks and shows how to use some of the system `BITOPS_` available environment variables then outputs configuration values defined by the `bitops.schema.yaml` in `cli` and `options` sections.
+```sh
 #!/bin/bash
 set -ex
 
 echo "Running Duplicate Environment Plugin deployment script..."
 
 # vars
-export DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS="$BITOPS_PLUGIN_DIR"
-export DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS="$BITOPS_OPSREPO_ENVIRONMENT_DIR"
-export BITOPS_SCHEMA_ENV_FILE="$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS/ENV_FILE"
-export SCRIPTS_DIR="$DUPLICATE_ENVIRONMENT_ROOT_SCRIPTS/scripts"
+export BITOPS_SCHEMA_ENV_FILE="$BITOPS_OPSREPO_ENVIRONMENT_DIR/ENV_FILE"
 
-if [ ! -d "$DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS" ]; then
-  echo "No duplicate-environment directory.  Skipping."
+if [ ! -d "$BITOPS_OPSREPO_ENVIRONMENT_DIR" ]; then
+  echo "No duplicate-environment directory. Skipping."
   exit 0
 fi
 
-printf "Deploying duplicate-environment..."
+echo "Deploying duplicate-environment..."
 
 if [ ! -f "$BITOPS_SCHEMA_ENV_FILE" ]; then 
   echo "No duplicate-environment ENV file found"
 else
   source "$BITOPS_SCHEMA_ENV_FILE"
 fi
 
-bash $SCRIPTS_DIR/validate_env.sh
-
-cd $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS
+cd $BITOPS_OPSREPO_ENVIRONMENT_DIR
 
-echo "Listing contents of duplicate-environment Root: $DUPLICATE_ENVIRONMENT_ROOT_OPERATIONS"
+echo "Listing contents of duplicate-environment Root: $BITOPS_OPSREPO_ENVIRONMENT_DIR"
 ls -al .
 
-echo "settings:"
+
+echo "Running the plugin CLI:"
+# Expected result: "plugin_command run --key=value --bar"
+echo plugin_command "${BITOPS_MY_PLUGIN_CLI}"
+
+echo "Options:"
 echo "DUPLICATE_ENVIRONMENT_FOO"
+# Expected result: "foo_value"
 echo "$DUPLICATE_ENVIRONMENT_FOO"
-```
-> **Note:** Much of the above is best practice boilerplate and is not strictly necessary.
 
-> **Important:** Be sure to `chmod +x deploy.sh`
-
-Create `scripts/validate.sh` as follows.  This script simply checks any necessary preconditions before continuing.
-```
-#!/bin/bash
-set -e 
-
-if [ -z "$BITOPS_ENVIRONMENT" ]; then
-  echo "environment variable (ENVIRONMENT) not set"
-  exit 1
-fi
 ```
+> **Note:** Much of the above is best practice boilerplate and is not strictly necessary.
 
 Finally, create a `plugin.config.yaml` to configure how BitOps uses the plugin.
-```
+```yaml
 plugin:
   deployment:
     language: bash
@@ -94,7 +105,7 @@ plugin:
 
 
 ## 2. Create an ops repo for testing
-```
+```sh
 cd /path/to/ops-repos
 mkdir duplicate-environment
 cd duplicate-environment
@@ -106,9 +117,13 @@ From there, you'll need to create an environment with a directory that matches y
 
 Populate the tool's `bitops.config.yaml` based on the schema defined above:
 `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops.config.yaml`
-```
+```yaml
 duplicate-environment:
-  properties:
+  cli:
+    command: run
+    key: value
+    bar: true
+  options:
     foo: baz
 ```
 
@@ -118,7 +133,7 @@ duplicate-environment:
 To test your plugin, you'll need BitOps to run with a BitOps-level BitOps config that has your plugin defined in the `deployments`.
 
 Create a `bitops.config.yaml` somewhere (say: `/path/to/ops-repo/plugin-no-install/duplicate-environment/bitops-level/bitops.config.yaml`), and add a `plugins` and `deployments` reference to your plugin:
-```
+```yaml
 bitops:
   fail_fast: true
   logging:      
@@ -135,7 +150,7 @@ bitops:
 
 ### 3.2. Run your test
 To run BitOps against a local plugin, you'll need to mount the plugin to the location BitOps expects plugins to be.
-```
+```sh
 docker run --rm --name bitops \
 -e BITOPS_ENVIRONMENT="duplicate-environment" \
 -v /path/to/bitops:/opt/bitops \
@@ -158,8 +173,9 @@ If your new plugin needs to run some install scripts (e.g. to install a CLI tool
 
 ### 4.1. Update the Plugin to Add an Install Script
 Add the `install` configuration to your plugin's `plugin.config.yaml`
-```
+```yaml
 plugin:
+  # this plugin has install instructions
   install: 
     language: bash
     install_script: install.sh
@@ -174,7 +190,7 @@ plugin:
 Add your install script:
 
 `install.sh`
-```
+```sh
 #!/bin/bash
 set -e
 
@@ -195,14 +211,14 @@ echo "Install your things here"
 ### 4.2. Build a BitOps image
 
 Create a new directory to hold your custom BitOps config:
-```
+```sh
 mkdir /path/to/bitops-custom
 cd /path/to/bitops-custom
 ```
 
 #### 4.2.1. Add the BitOps config
 First, add your BitOps level `bitops.config.yaml` and include a reference to your local file dependency (via `plugins`) and a reference in the `deployments` section:
-```
+```yaml
 bitops:
   fail_fast: true
   run_mode: default   # (Unused for now)
@@ -228,15 +244,15 @@ bitops:
 #### 4.2.2. Add your Dockerfile
 
 `Dockerfile`
-```
+```Dockerfile
 FROM bitovi/bitops:base
 ```
 
 #### 4.2.3. Copy plugin code to the BitOps directory
 In order for the build to have access to your local plugin files, they'll need to be in the same directory as the `Dockerfile`.  One quick way to do this is to set up a simple script to run prior to your docker build to clean and re-copy the plugin files:
 
 `copy-plugins.sh`
-```
+```sh
 #!/bin/bash
 
 mkdir -p /path/to/bitops-custom/plugins
@@ -250,13 +266,13 @@ cp -r /path/to/bitops-plugins/duplicate-environment /path/to/bitops-custom/plugi
 
 
 #### 4.2.4. Build the image
-```
+```sh
 ./copy-plugins.sh
 docker build bitops --tag bitovi/bitops:local-custom --progress=plain --no-cache .
 ```
 
 #### 4.2.5. Test your plugin
-```
+```sh
 docker run --rm --name bitops \
 -e BITOPS_ENVIRONMENT="duplicate-environment" \
 -v /path/to/bitops:/opt/bitops \
@@ -270,7 +286,7 @@ bitops:local-custom
 
 ## 5. Handling the Plugin Install Script (remote)
 As an alternative way to develop plugins locally is simply to host the plugin code/config remotely and specify the plugin via url instead of `file://` like:
-```
+```yaml
 bitops:
   fail_fast: true
   run_mode: default   # (Unused for now)
@@ -290,4 +306,4 @@ bitops:
       plugin: duplicate-environment
 ```
 
-Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script).
+Then, you can follow the steps in #5 above (sans the `copy-plugins.sh` script).

docs/examples/plugin-examples/plugin-install/duplicate-environment/bitops.config.yaml

@@ -1,2 +1,7 @@
 duplicate-environment:
-  foo: baz
+  cli:
+    command: run
+    key: value
+    bar: true
+  options:
+    foo: baz

docs/lifecycle.md

@@ -9,6 +9,7 @@ This is a useful way to extend the functionality of BitOps. A popular use case w
 
 ![lifecycle diagram](assets/images/lifecycle.png)
 
+### Main Execution Flow
 A single run of BitOps will:
 
 1. Copy the contents of `/opt/bitops_deployment` to a temporary working directory
@@ -27,7 +28,7 @@ A single run of BitOps will:
     * Run any `bitops.before-deploy.d/*.sh` scripts
     * Load `bitops.config.yaml` and set environment
     * Merge contents with [Default environment](default-environment.md) - [TODO](https://github.com/bitovi/bitops/issues/18)
-    * Run `ansible-playbook $playbook` for each `*.yaml` or `*.yml` file in `$env/ansible/` 
+    * Run `ansible-playbook playbook.yaml` in `$env/ansible/` 
     * Run any `bitops.after-deploy.d/*.sh` scripts
 4. If a `helm/` directory exists within the selected environment:
     * Run the following for `$env/helm/$ENVIRONMENT_HELM_SUBDIRECTORY/` or for all charts in `$env/helm/`
@@ -49,7 +50,9 @@ A single run of BitOps will:
     * Create or delete cfn stack. Wait for completion
     * Run any `bitops.after-deploy.d/*.sh` scripts
 
-### Environment Variables
-Plugins can export the environment when a value is specified in an ops_repo level `bitops.config.yaml`. These environment variables are prefixed with `BITOPS_` and their plugin name. 
+### Imported Environment Variables
+The plugin config values and defaults are overriden by user environment variables passed to BitOps by prefixing them with `BITOPS_`. For example, `BITOPS_ANSIBLE_SKIP_TAGS=tag1,tag2` will set the plugin's config `ansible.cli.skip-tags` value to `tag1,tag2`.
+See [Environemnt Variables Defaulting](configuration-base.md#environemnt-variables-defaulting) for more information.
 
-So for example, if the terraform plugin exported the environment variable `BUTTER_FLAG`, it would be accessible in the lifecycle hooks by referencing `BITOPS_TERRAFORM_BUTTER_FLAG`.
+### Exported Environment Variables
+BitOps exports the environment variables to the plugin when a ENV var name is specified in `bitops.schema.yaml` via `export_env`. This is useful for passing values to lifecycle hooks, custom scripts, or directly to the plugin executable.

docs/migration.md

@@ -1,5 +1,10 @@
 Here is the list of upgrade notes for major breaking changes that you need to be aware of when migrating between the BitOps versions.
 
+## v2.4
+* `ANSIBLE_ROOT` ENV var in Ansible plugin was removed. Use `BITOPS_OPSREPO_ENVIRONMENT_DIR` instead, which is consistent across all plugins.
+* Plugin environment variables are now automatically mapped to the plugin schema. This means that you can now use the same environment variable names as in the plugin schema. For example, if you want to override the `ansible.cli.skip-tags` value, you can now use the `BITOPS_ANSIBLE_SKIP_TAGS` environment variable. This is true for all the plugins. The precedence order is: `ENV` vars > `bitops.config.yaml` values > `bitops.config.schema.yaml` defaults.
+Keep the new convention in mind when upgrading to `v2.4` as it may clash with your existing environment variables.
+
 ## v2.2
 * Terraform plugin `stack-action` was moved from `options` to `cli` section in `bitops.config.yaml`.
 You need to update your configuration from old:

docs/plugins.md

@@ -109,4 +109,4 @@ Creating a plugin is easy, you only need 4 files:
 
 > For more information, you can look at our [example plugin](https://github.com/bitops-plugins/example-plugin) repo that prints your name and favorite color!
 
-> For more information on developing with plugins locally, see [development/local-plugin-creation](./development/local-plugin-creation.md).
+> Here is [Plugin Creation](./development/local-plugin-creation.md) guide on developing plugin locally.

docs/roadmap.md

@@ -22,6 +22,13 @@ Create an Issue, open a Discussion, join our Community and we'll be happy to wor
 Check the [`main`](https://github.com/bitovi/bitops) repository branch to see the ongoing development.
 
 ## Release History
+### Done in v2.4.0
+  - **Plugins:** ENV variable mapping based on plugin schema
+  - **Plugins:** Plugin CLI command generation based on schema (beta)
+  - **Website:** Revamp with new video and list of BitOps values
+  - **Community:** Switch from Slack to Discord
+  - **Community:** Add non-stop user survey
+
 ### Done in v2.3.0
   - **Security:** Secrets masking
   - **Core:** Local plugin install via `file://`