Add a new standalone st2-pack-install CLI command for installing a pack #4256
Conversation
pack_management.download module. This way code is decoupled better and can be re-used in other places.
we are unable to find pack in the index.
environment. This script doesn't depend on RabbitMQ and MongoDB connection and is to be used and Docker and similar environments where content (packs) are pre-baked into container / vm image.
functions in st2common.util.pack_management module.
|
Can make it to install several packs |
|
And yeah, you mentioned different naming:
|
|
@armab Yeah, I was thinking about it, but I'm not a fan of multi pack install in a single command (never was) so I intentionally didn't add support for that. The reason for that is that error reporting and handling gets much harder in such scenario since command now has many different failure scenarios. For example, it can fail half way through - some packs are installed successfully and some are not. In general in distributed systems and scenarios like that, I believe "one by one" approach is better since it makes error handling easier. |
|
I'm in favor of the "multi-install" approach that @armab suggested. This makes |
|
It'll make We want it to be as similar as possible in behavior, syntax, input with the original command, just don't interact with the DB/MQ. So from user's side original: |
|
I'll wait a bit and also let others chime on this one. If anything, I would prefer to remove installing multiple packs at once support from |
|
@nmaludy The big difference between those tools and StackStorm is that all those tools support and use transactions for those operations which makes error handling much easier and straight forward (either all succeed or all fail, no middle ground). That's not the case for StackStorm "st2 pack install" command. It doesn't support or use transactions so installing each pack is done in exactly the same manner if user ran the same command each time with a different pack. The problem is that handling and dealing with such errors is much harder and more complex in multi pack install scenario without transactions - aka how to proceed / rectify if it fails hall way through. And not to mention that current UX for multi pack install in See the example below, or this one - https://gist.github.com/Kami/958d45c07e52934ce38fb462b12a8fb5: 2018-07-20 13:39:06,826 WARNING - Auth API server is not available, skipping authentication.
For the "libcloud, xmlx=9.9.9, csv" packs, the following content will be registered:
rules | 0
sensors | 0
aliases | 0
actions | 32
triggers | 0
Installation may take a while for packs with many items.
[ succeeded ] download pack
[ succeeded ] make a prerun
[ succeeded ] install pack dependencies
[ succeeded ] register pack
+----------+-------------------------------------------------+---------+------------------+
| name | description | version | author |
+----------+-------------------------------------------------+---------+------------------+
| csv | st2 content pack containing CSV integrations | 0.4.2 | StackStorm, Inc. |
| libcloud | st2 content pack containing libcloud | 0.4.3 | StackStorm, Inc. |
| | integrations | | |
+----------+-------------------------------------------------+---------+------------------+So if anything, I'd rather have a fully working "install single pack per invocation" mode than broken (or at the very least kinda working which is not enough and gives user false sense of success) "install multiple packs in a single command invocation" mode. It boils down to that "st2 pack install" should never support multi pack installation (or do it correctly) and it was probably added and rushed at some point without too much thinking. And I don't want to repeat same bad behavior in a new command since there is no need to (and as mentioned, doing it correctly is not fully trivial and we would need to decide on the behavior we want). |
…m:StackStorm/st2 into pack_install_and_setup_virtualenv_command
|
I discussed this with @armab at length on Slack and we finally came up with a compromise. I will add support for installing multiple packs to this new command, but I won't really document it and installing single pack at once will be the preferred approach. Either @armab and I will also file an issue for improvements in the existing |
correctly if user passes in a version (e.g. st2-pack-instal bitcon=0.1.0).
|
@armab I wasn't able to replicate the pack group issue locally. It works fine and uses (virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ./st2common/bin/st2-pack-install bitcoin=0.1.0
2018-07-30 09:03:31,111 INFO [-] Installing pack "bitcoin=0.1.0"
2018-07-30 09:03:32,491 INFO [-] Successfuly installed pack "bitcoin=0.1.0"
2018-07-30 09:03:32,491 INFO [-] Setting up virtualenv for pack "bitcoin"
2018-07-30 09:03:35,689 INFO [-] Successfuly set up virtualenv for pack "bitcoin=0.1.0"
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ls -la /opt/stackstorm/packs/
total 32
drwxr-xr-x 8 vagrant vagrant 4096 Jul 30 09:03 .
drwxr-xr-x 5 vagrant vagrant 4096 Jun 22 10:00 ..
drwxrwxr-x 4 vagrant st2packs 4096 Jul 30 09:03 bitcoin |
…m:StackStorm/st2 into pack_install_and_setup_virtualenv_command
|
@armab Pushed a fix for I can't replicate your other two issues though.
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ./st2common/bin/st2-pack-install bitcoin=0.1.0
2018-07-30 09:03:31,111 INFO [-] Installing pack "bitcoin=0.1.0"
2018-07-30 09:03:32,491 INFO [-] Successfuly installed pack "bitcoin=0.1.0"
2018-07-30 09:03:32,491 INFO [-] Setting up virtualenv for pack "bitcoin"
2018-07-30 09:03:35,689 INFO [-] Successfully set up virtualenv for pack "bitcoin=0.1.0"
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ls -la /opt/stackstorm/packs/
total 32
drwxr-xr-x 8 vagrant vagrant 4096 Jul 30 09:03 .
drwxr-xr-x 5 vagrant vagrant 4096 Jun 22 10:00 ..
drwxrwxr-x 4 vagrant st2packs 4096 Jul 30 09:03 bitcoin
(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ./st2common/bin/st2-pack-install file:///opt/stackstorm/packs/bitcoin
2018-07-30 09:11:19,309 INFO [-] Installing pack "file:///opt/stackstorm/packs/bitcoin"
2018-07-30 09:11:19,373 INFO [-] Successfuly installed pack "bitcoin"
2018-07-30 09:11:19,373 INFO [-] Setting up virtualenv for pack "bitcoin"
2018-07-30 09:11:21,424 INFO [-] Successful set up virtualenv for pack "bitcoin"Both of those things should work out of the box since they use exactly the same code path as `st2 pack install. |
…and_setup_virtualenv_command
can be re-used by other code. Also re-use existing do_register_cli_opts instead of copy and pasting it over.
using copy and pasted version.
Those commands are analogous of packs.download and packs.setup_virtualenv StackStorm actions. Running st2-pack-install is the same as running st2-pack-download first and st2-pack-setup-virtualenv second.
|
As mentioned / discussed above, I added two new commands:
st2-pack-download --help
usage: st2-pack-download [-h] [--config-dir DIR] [--config-file PATH]
[--debug] [--force] [--nodebug] [--noforce]
[--noprofile] [--nouse-debugger] [--noverbose]
[--noverify-ssl] [--profile] [--use-debugger]
[--verbose] [--verify-ssl] [--version]
[pack [pack ...]]
st2-pack-setup-virtualenv --help
usage: st2-pack-setup-virtualenv [-h] [--config-dir DIR] [--config-file PATH]
[--debug] [--nodebug] [--noprofile]
[--nopython3] [--noupdate] [--nouse-debugger]
[--noverbose] [--profile] [--python3]
[--update] [--use-debugger] [--verbose]
[--version]
[pack [pack ...]]Those two commands are analogous of Running This gives users more flexibility and I don't think command pollution is an issue (commands are prefixed with I also prefer this approach over adding CLI flags to |
|
NOTE: End to end tests will keep failing until I merge this - StackStorm/st2tests#139. I plan to merge st2tests PR once I get this PR approved and once it's merged into master. |
|
Try installing the packages produced from CircleCI. I guess that brings some diff. from latest package in this PR: |
|
Can confirm that pack-install script now works well with custom versions, file://, etc 👍 The group ownership is still an issue. |
|
@armab I assume that's because We don't explicitly change or set permissions for pack virtualenv directory, it just inherits whatever user and group it runs under. I can make the change and add some code which sets the group based on the
In short, there really is no need for those commands to require sudo access - they only need it if we mess with the groups. |
of the pack virtualenv directory to the value defined in the config
(content.pack_group, defaults to st2packs).
This way the end result is consistent also when running
st2-pack-setup-virtualenv and st2ctl reload --register-setup-virtualenvs
command as a user which is not part of st2packs group.
Keep in mind that this step requires sudo access, but if sudo is not
available, or group ownership change step fails, the command itself
doesn't fail.
This means that sudo access for st2-pack-{download,setup-virtualenv,install}
commands is optional.
|
@armab I pushed a commit (5513b7d) which I believe is a good compromise and results in the same outcome no matter how you run the command. Previously, virtualenv directory owner group would only be set correctly when using action runner action ( As mentioned above, I'm not a fan of those commands requiring sudo access. That's why the "change directory owner" step doesn't fail if the directory group ownership change fails due to one reason or another (defined group doesn't exist, no sudo access, etc.). Keep in mind that this "ignore errors on change directory owner" behavior already existed before my changes, but I assume it was not done intentionally, but now that's done intentionally / as part of the design. So in short, this fixes inconsistencies between the commands (good catch, this issue existed for the virtualenv directory when using |
|
In a normal environment, when the StackStorm is shipped via packages you couldn't run So anyways, it was already a requirement to run |
This pull request adds a new independent and standalone
st2-pack-installCLI command for installing a pack and setting up pack virtual environment.End result is the same as running
st2 pack install <foo>minus the content registration step.This command is fully standalone and only requires Python, pip,
st2commonPyPi package and git binary to be installed on the system where it's used. It doesn't need / require database (MongoDB) and message bus access (RabbitMQ).This command is designed to be used in environments where all the resources (packs) are baked into the base VM / container image which is then deployed.
Keep in mind that the actual content still needs to be registered with StackStorm at some later point when a cluster is deployed and when MongoDB and RabbitMQ are up (
st2ctl reload --register-allwhich is also an idempotent operation).NOTE: This change required some long needed refactoring - moved a bunch of code outside of the action which makes it re-usable elsewhere and also easier to test.
Usage:
Resolves #3912.