Ability to filter out AUDIT logs

[nmaludy]

SUMMARY

Currently, the log files /var/log/st2/st2*.log contain all of the AUDIT log information as well as the INFO and DEBUG information. These AUDIT logs are duplicated in the /var/log/st2/st2*.audit.log files.

In our case the /var/log/st2/st2scheduler.log and /var/log/st2/st2scheduler.log are both 5GB in size (10GB total) in less than 24h of runtime.

We would like the ability to disable sending AUDIT logs to the /var/log/st2/st2*.log files since the information is duplicated anyways.

ISSUE TYPE

• Feature Idea

STACKSTORM VERSION

$ st2 --version
st2 2.10.1, on Python 2.7.5

OS / ENVIRONMENT / INSTALL METHOD

$ uname -a
Linux nor1devssd01.dev.encore.tech 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

Install method = puppet-st2

STEPS TO REPRODUCE

$ grep "AUDIT action" st2scheduler.log | wc -l
1447

EXPECTED RESULTS

$ grep "AUDIT action" st2scheduler.log | wc -l
0

ACTUAL RESULTS

I think the "chattyness" of st2scheduler could be turned down as well. Not sure the entire action and results need to be printed every time a scheduling change occurs? (maybe it does?)

[nmaludy]

Looking at how Python logging works, it looks like you set a "log level" and then any logs that have a HIGHER number than that value are written out. Here's the default table values:

https://docs.python.org/2/library/logging.html#logging-levels

Level	Numeric Value
CRITICAL	50
ERROR	40
WARNING	30
INFO	20
DEBUG	10
NOT_SET	0

AUDIT seems to be a custom level that's defined here: https://github.com/StackStorm/st2/blob/master/st2common/st2common/log.py#L49-L50

It is set to CRITICAL + 10, so no matter what we change the level to on the logging handlers, AUDIT logs will always be written.

[Kami]

I agree - I encountered this issue / limitation in the past.

I will check if we can set it to a lower value and if that works (IIRC, I already tried that in the past and there were some issues so we might need a different "filter out" approach).

[Kami]

I believe a fix in #4538 should also the solve problem with st2scheduler noisiness / chatiness.

I had a look at the log messages in st2scheduler and other non-audit messages are small in size, so they shouldn't be causing much issues.