Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Experimental and opt-in Python 3 support for Python runner actions #4148

Closed
wants to merge 62 commits into from

Conversation

Kami
Copy link
Member

@Kami Kami commented May 28, 2018

This pull request adds experimental and opt-in support for using Python 3 for pack actions.

It continues work from other PRs - #3922, #4016, #4075.

Our code changes in the past were a bit too optimistic and it turned out a lot more work is still needed to get that feature working.

The scope and target audience for this feature didn't change - it's still intended for advanced users and it's opt in - user needs to pass in --python3 flag to st2 pack install <pack name> CLI command and Python 3 binary will be used when creating virtual environment for that pack and for all the subsequent pack action executions.

As mentioned before, this feature is mostly meant as an experimental stop gap for advanced users until the whole platform supports Python 3 (aka StackStorm itself can run on Python 3).

We don't have any specific ETA for Python 3 support for the whole platform, but we are slowly and incrementally working towards it - every time I touch some code, I try to also make it work under Python 3 and and make sure tests for that code also run under Python 3 on Travis.

Included Changes

TBW

Example Usage

1. Not using Python 3 (default behavior)

$ st2 run packs.setup_virtualenv packs=examples             
2018-05-28 13:57:14,306  WARNING - Auth API server is not available, skipping authentication.
....
id: 5b0c0abb0640fd0cd6d00546
status: succeeded
parameters: 
  packs:
  - examples
result: 
  exit_code: 0
  result: 'Successfuly set up virtualenv for the following packs: examples'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "examples" (/opt/stackstorm/packs/examples)
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Removing virtualenv in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "examples" in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/examples" using Python binary "/data/stanley/virtualenv/bin/python2.7"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/data/stanley/virtualenv/bin/virtualenv -p /data/stanley/virtualenv/bin/python2.7 --system-site-packages --no-download /opt/stackstorm/virtualenvs/examples" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/examples/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/examples/requirements.txt with command /opt/stackstorm/virtualenvs/examples/bin/pip install -U -r /opt/stackstorm/packs/examples/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "examples" successfully created in "/opt/stackstorm/virtualenvs/examples"
    '
  stdout: ''
$ st2 run examples.python_runner_print_python_version.
id: 5b0c0aca0640fd0cd6d00549
status: succeeded
parameters: None
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]
"

2. Opting in and using Python 3 for a particular park

$ st2 run packs.setup_virtualenv packs=examples python3=true
id: 5b0c0af00640fd0cd6d0054c
status: succeeded
parameters: 
  packs:
  - examples
  python3: true
result: 
  exit_code: 0
  result: 'Successfuly set up virtualenv for the following packs: examples'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "examples" (/opt/stackstorm/packs/examples)
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Removing virtualenv in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "examples" in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/examples" using Python binary "/data/stanley/virtualenv/bin/python2.7"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/data/stanley/virtualenv/bin/virtualenv -p /usr/bin/python3 --system-site-packages --no-download /opt/stackstorm/virtualenvs/examples" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base Python 3requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement pyyaml>=3.12,<4.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install pyyaml>=3.12,<4.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/examples/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/examples/requirements.txt with command /opt/stackstorm/virtualenvs/examples/bin/pip install -U -r /opt/stackstorm/packs/examples/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "examples" successfully created in "/opt/stackstorm/virtualenvs/examples"
    '
  stdout: ''
$ st2 run examples.python_runner_print_python_version.
id: 5b0c0afc0640fd0cd6d0054f
status: succeeded
parameters: None
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.4.3 (default, Sep 14 2016, 12:36:27) 
[GCC 4.8.4]
"

TODO

  • Unit tests
  • Integration / end to end tests (st2tests)

stanley and others added 30 commits April 6, 2018 10:06
Add an entry for the st2-track-result command line utility into the setup script.
Update release date for the v2.7.0 release (v2.7)
for packs.

This way we ensure version of pip which is on the system is used instead
of latest stable version of pip and other packages being download from
PyPi on virtualenv creation.
a virtualenv for tests.

This way we ensure consistent build environment.
This way we ensure all the submodules are checked out.
Kami and others added 25 commits May 10, 2018 10:58
and allow it to be used outside the nose tests context.
This way it fixes regression introduced in 2.7.0 and sensors which rely
on subprocess.poll() (select.poll()) still work in the same manner as
they did before v2.7.0.
explicitly install pyyaml dependency into it instead of re-using one
from st2 virtual environment site-packages since pyyaml ships different
code for Python 2 and Python 3.
Python 3 is used for a pack virtual environment.
This way the code is more readable and easier to unit test.
@Kami Kami added the python3 label May 28, 2018
@Kami Kami added this to the 2.8.0 milestone May 28, 2018
@Kami
Copy link
Member Author

Kami commented May 28, 2018

EDIT - Closing, I some how merged that branch, accidentally based it on top of v2.6 and not master.

@Kami Kami closed this May 28, 2018
@cognifloyd cognifloyd deleted the python3_compat_changes_actions branch February 12, 2024 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants