st2 run --tail [action] error on Unicode output

[jbrownsc]

Version

root@c3c4ea471142:/opt/stackstorm# st2 --version
st2 2.7.1, on Python 2.7

Description
st2 action run --tail errors out when unicode output exists in stdout/stderr of task

Steps to reproduce

root@c3c4ea471142:/opt/stackstorm# st2 run core.local cmd="printf '\xE2\x98\xA0'"
.
id: 5af355953ce5fe535b96843a
status: succeeded
parameters:
  cmd: printf '�'
result:
  failed: false
  return_code: 0
  stderr: ''
  stdout: "☠"
  succeeded: true
root@c3c4ea471142:/opt/stackstorm# st2 run --tail core.local cmd="printf '\xE2\x98\xA0'"
Tailing execution "5af3559c3ce5fe535b96843d"
Execution 5af3559c3ce5fe535b96843d has started.

ERROR: 'ascii' codec can't encode character u'\u2620' in position 0: ordinal not in range(128)

root@c3c4ea471142:/opt/stackstorm#

[Kami]

Thanks for reporting this.

Can you please include the output of st2 --debug execution tail <id>? Since I can't reproduce the issue locally using v2.7.1:

st2 run core.local cmd="printf '\xE2\x98\xA0'" --tail
2018-05-10 17:00:04,744  WARNING - Auth API server is not available, skipping authentication.
Tailing execution "5af47a940640fd35b085f7a4"
Execution 5af47a940640fd35b085f7a4 has started.

☠
Execution 5af47a940640fd35b085f7a4 has completed (status=succeeded).

id: 5af47a940640fd35b085f7a4
status: succeeded
parameters: 
  cmd: printf '�'
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "☠"
  succeeded: true

I wonder if it's related to a locale setting on your machine or similar? Output of locale command would also help.

[Kami]

It looks like it's indeed an issue related to your environment (locale being used - most likely place where you are running StackStorm CLI is using C locale).

StackStorm simply uses and respects locale which is set as a default on the system (as the apps should do). On the distros we support out of the box, the default should be en_US.UTF-8.

It works just fine for me out of the box, but if I explicitly specify a non-utf8 locale, it breaks (as expected).

LC_ALL=C st2 run core.local cmd="printf '\xE2\x98\xA0'" --tail
2018-05-10 17:06:21,017  WARNING - Auth API server is not available, skipping authentication.
Tailing execution "5af47c0d0640fd35b085f7b6"
Execution 5af47c0d0640fd35b085f7b6 has started.

ERROR: 'ascii' codec can't encode character u'\u2620' in position 0: ordinal not in range(128)

Which distro are you using? And how did you install the StackStorm?

It doesn't seem to be related to StackStorm, but getting more information about the environment you are using wouldn't hurt either (distro, version, locale, how was stackstorm installed, etc.).

[jbrownsc]

This is the st2-docker stackstorm image (stackstorm/stackstorm:2.7.1)

root@c3c4ea471142:/opt/stackstorm# cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
root@c3c4ea471142:/opt/stackstorm# uname -a
Linux c3c4ea471142 4.9.87-linuxkit-aufs #1 SMP Fri Mar 16 18:16:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

root@c3c4ea471142:/opt/stackstorm# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

It appears that the st2-docker container does not account for locale. Filed an issue on st2-docker to resolve this StackStorm/st2-docker#129

[jbrownsc]

Would we want to handle this exception and provide a useful error like "Please check your system locale (None, None)" ?

root@c3c4ea471142:/opt/stackstorm# python
Python 2.7.6 (default, Nov 23 2017, 15:49:48)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
(None, None)
>>>
root@c3c4ea471142:/opt/stackstorm# LC_ALL=en_US.UTF-8 python
Python 2.7.6 (default, Nov 23 2017, 15:49:48)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('en_US', 'UTF-8')
>>>

[Kami]

I'm not sure that's the right thing to do since there are many places where that exception could be thrown (too many places to handle that exception).

In theory we could also check the locale and throw on service start up if it's not UTF-8, but I'm not convinced yet that's a good idea either.

Maybe we could include default and used system locale in st2 --version output as a compromise? And warn the user if it's not UTF-8 that some functionality might not work.

And thanks for filing this issue on our docker repo 👍

[arm4b]

Yes, I agree that forcing/quiting on non-expected locale is too agressive.

Showing a warning somewhere makes more sense (on any st2 command to stderr, in st2 logs or just st2 --version).

[Kami]

Closing since it's an issue in st2-docker images. Continuation in StackStorm/st2-docker#129.

Thanks again for reporting this.

I looked at adding locale information to st2 --version / --st2 debug, but locale.getdefaultlocale() doesn't seem to be working as I expect it to so I'm not sure if it's worth adding a locale of code just for reading and displaying used locale.