Make scheduling timeout constant to be a configurable option

[guzzijones]

Fixes #4887

This stopped my execution queue item from being requested more than once for long running requests.
Is there a config setting for the timing of this?

I am not sure if this is needed since there is retry code in the code base.

[CLAassistant]

All committers have signed the CLA.

[punkrokk]

@guzzijones Can you expand your explanation and why you commented out this line? I will ping @m4dcoder again for feedback, I think I saw a note that he was able to reproduce it. I just had it happen many times in one of my environments.

[guzzijones]

It prevents the action from redeploying on exit. Large json take a while to update.

[guzzijones]

More info?

[guzzijones]

@blag mentioned we need a unit test for this PR. before I spend a bunch of time on that can someone please at least bless the approach here.
my PR basically negates this entire function.
It probably is better to figure out why we end up here in the first place if the action is still not finished writing it's changes to the database.
I have the issue listed above if someone wants to verify we nuke this function.

[m4dcoder]

@punkrokk I don't know where you read I was able to reproduce this. This is the first I look at this issue #4887 and the proposed solution here. I have not had a chance to review this to make the connection here between the issue and how this solution fixes it.

[m4dcoder]

The scheduler's garbage collection actively looks for action execution that is locked for long time and hasn't been released. This may be caused by a scheduler being terminated abnormally before releasing locks. The time before a lock is manually released by GC is set at the constant EXECUTION_SCHEDUELING_TIMEOUT_THRESHOLD_MS. This is currently not configurable. Since this issue is specific to the end user (large input which causes delay in scheduling), the solution should be to make EXECUTION_SCHEDUELING_TIMEOUT_THRESHOLD_MS configurable and then the end user adjust the value according to needs.

[m4dcoder]

The alternative solution is to add service discovery for all the st2 components and instead of blindly releasing the lock, first look to see if the scheduler that is processing the action execution is still healthy and alive. This takes more work though.

[guzzijones]

Thanks @m4dcoder. I will redo this pr at some point soon hopefully. That makes a lot more sense.

[guzzijones]

Yes , this is also breaking the unit test for this method. Working on this today. I will probably have to close this request and point it to a new one as I made this change in the gui through github.com

[blag]

You don't have to close this a create a new PR, you can just:

# Fetch changes from all branches from GitHub
git fetch --all
# Checkout the master branch
git checkout master
# Pull in all changes to your local master branch
git pull
# Switch back to your patch-5 branch
git checkout patch-5
# Rebase back on top of the master branch
git rebase master

And then you can just continue development in this branch as normal.

[guzzijones]

I simply cherry picked this from a working local branch.
I am not sure why it thinks the option doesn't exist. I will have to double check.
I see this error in the unit tests:
raise NoSuchOptError(opt_name, group) NoSuchOptError: no such option in group scheduler: execution_scheduling_timeout_threshold_m

[m4dcoder]

I don't understand the change here. Why not just remove the constant EXECUTION_SCHEDUELING_TIMEOUT_THRESHOLD_MS and then replace it with a config option at https://github.com/StackStorm/st2/blob/master/st2actions/st2actions/scheduler/handler.py#L103? The config option if min should be float/decimal to allow for under 1 minute. The calculation self._execution_scheduling_timeout_threshold_ms should be just cfg.CONF.scheduler.execution_scheduling_timeout_threshold_m * 60 * 1000.

[guzzijones]

Is there a fixture I need to put my new scheduler config in? Everything runs locally fine with I start up stackstorm using the launch.sh start command.

[guzzijones]

I see it

[guzzijones]

I undid the change to st2.conf.sample. I guess I am not supposed to touch that file. Someone will have to fill me in on how to handle that. It is failing the CI checks.

[guzzijones]

I will need some help on why the unit test for st2.conf.sample is failing.

[m4dcoder]

Run make configgen to update st2.conf.sample and then push the changes as commit.

[guzzijones]

Run make configgen to update st2.conf.sample and then push the changes as commit.

still failing

[arm4b]

st2.conf.sample looks good now.

@guzzijones If you're curious, the 4774918 was close. If you look at CI failure, - it highlighted the following diff: https://travis-ci.org/github/StackStorm/st2/jobs/675013639#L1571-L1576 where new st2.conf setting was expected to appear a few lines before its position. Computers ¯_(ツ)_/¯.

[arm4b]

@guzzijones Thanks for the contribution and everyone involved @m4dcoder @blag @punkrokk for reviews 👍

[guzzijones]

Oh wow that is specific. Thanks for the quick fixes. And thanks for all the patience. A lot of hours went into this on our end so we are glad it is merged.