Proposal: Support MongoDB 3.6

[LindsayHill]

Proposal: Support MongoDB 3.6 in the next version of ST2, and then make it the default in the next version after that.

Background: MongoDB 3.6 was released around 6 months ago, and we are now up to version 3.6.4, so stability should be good.

MongoDB 3.2 is supported until September 2018, while no end date has been set for MongoDB 3.4 or 3.6.

ST2 currently works with MongoDB 3.2 or 3.4. The recommendation (and default) is to use v3.4.

ST2 2.7.1 does not work consistently with MongoDB 3.6. See #4097.

Tasks:

• Figure out some workaround or upstream patch for mongoengine issue Increased x1.5-x2.0 memory usage after recent pip upgrade [staging-unstable] #3595
• Test ST2 master with MongoDB 3.6, and make any changes required to support both 3.4 & 3.6.
• Ship those changes in ST2 v2.8
• Update docs to reflect that initial v3.6 support has been added
• In the next version of ST2 after 2.8, update our CI to use 3.6
• Update install scripts to default to installing 3.6.
• Update docs to reflect that 3.6 is preferred version

[nmaludy]

+1

Also created tasks for ansible-st2 (StackStorm/ansible-st2#193) and puppet-st2 (StackStorm/puppet-st2#217)

[Kami]

I'm +1 for that.

In fact, I already wanted to do it in the past, but one of the pre-requisites for that is upgrading mongoengine and pymongo dependency.

Sadly, we are blocked on mongoengine upgrade due to performance regression in 0.13.0 - #3595, MongoEngine/mongoengine@466935e#commitcomment-23225459.

[Kami]

I had a look at the latest version of mongoengine and the same issue still exists.

Memory usage with mongoengine version 0.12.0 (regression was introduced in 0.13.0):

(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i 'st2' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
90.0547MB 4596 vagrant python st2api/bin/st2api

Memory usage with the latest version (0.15.3, post 0.13.0):

(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i 'st2' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
117.469MB 4695 vagrant python st2api/bin/st2api

This memory usage regression can be consistently reproduced.

Sadly at this point only option we are left with is forking mongoengine repo in StackStorm org and reverting the change which causes the regression.

@armab and others, what do you think?

I really wish we wouldn't need to fork it, but we have been stuck with an old version of mongoengine for more than a year now and I can't think of a better solution :/

[arm4b]

Before forking, I think we should at least open a valid Issue in mongoengine with a clear description and reproducible example.

That would be a better, comparing to commenting in a code commit/history. Formalizing it that way can hopefully lead to a better attention/awareness, which is not different to how we expect contributions to land in our repo, for example.

[nmaludy]

@Kami the other thing to consider is this bug / performance issue caused by mongoengine #4030

Not sure if there is maybe an alternative solution out there with better performance characteristics?

[Kami]

@armab Agreed.

Opened an issue so we can better track it - MongoEngine/mongoengine#1832. I also included a very simple script which reproduces the issue.

It only affects Python 2.7.x so it's likely some edge case / bug in CPython in 2.7.x, but that doesn't matter that much since a lot of people still use Python 2.7.

[Kami]

@nmaludy Sadly I didn't have time to look into that issue yet so I can't comment on it.

Having said that, I don't think totally moving away from mongoengine is something we can / should do at this point.

One think we can do though is to optimize queries and similar (and potentially skip mongoengine document abstraction) where it makes sense and where it impacts the performance.

[Kami]

This PR will allow us to support MongoDB 3.6 - #4291.

I tested it locally and things seem to be working fine.

I propose the following approach which we have already used in the past when we added support for 3.4 in addition to 3.2:

1. Update Travis so we also run Python 2.7 unit and integration tests under 3.6 in addition to 3.4. This will slow things down since it means one more builder, but there is really no way around that. Good news is that I recently improved build times a lot so hopefully overhead shouldn't be too big.
2. Upgrade st2cicd server to latest version of st2 2.9dev and MongoDB 3.6
3. After a while, if everything looks OK announce official support in the next release and make necessary docs changes.
4. Update documentation and installer scripts to install and use 3.6 by default (but still support 3.4 for a reasonable amount of time giving people enough time to upgrade)
5. Eventually remove 3.4 from Travis build and only run tests under 3.6 (this way we get rid of one builder and speed the whole build up)

@armab @LindsayHill @lakshmi-kannan and others - what do you think?

[LindsayHill]

Yes, that approach looks good to me. Maybe do something like this for timelines:
2.9 - announce MongoDB 3.6 support
3.0 - update installer scripts, install 3.6 by default, announce deprecation of 3.4 support (NB: announce only, not actually removing support yet)
3.2 - Formally remove 3.4 support - mainly just by removing 3.4 from Travis. Will probably still work, but we won't test it or claim support.

Also: I see MongoDB has released 4.0. Still too early for us though, don't want to test with that for at least a few patch cycles.

[Kami]

#4291 has been merged and our unit tests now also run under MongoDB 3.6.

The bad news is that unit tests under 3.6 are substantially slower than under 3.4. (30-40% or so).

Nothing else has changed which makes it seem like the issue is indeed related to MongoDB 3.6 (and perhaps this issue - https://jira.mongodb.org/browse/SERVER-34604).

So even though our code now works with 3.6 we shouldn't really start looking into officially supporting it until this tests performance regression issue is resolved.

[arm4b]

To summarize it, we now have 2 issues:

• memory usage x2 with the official mongoengine lib (that we had to fork)
• DB performs up to ~50% slower with the 3.4 -> 3.6 migration

Looks like a bad trend 😞

[Kami]

It looks like we have an issue - https://jira.mongodb.org/browse/SERVER-34604?focusedCommentId=2015826&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2015826

We need to decide how to proceed when a new 3.6.x patch version with some of those fixes is out. Current code should already support and work with v3.6, but it looks like there are performance regressions which they acknowledged and don't plan to fix at the moment.

I also plan to test against v3.6 against with tests, etc. once a new 3.6.x patch is out to see if the performance regression is smaller than before.

[nmaludy]

I don't see those fixes included in 4.0.x either: https://docs.mongodb.com/manual/release-notes/4.0/

[LindsayHill]

I've just been thinking about MongoDB 4.0.x. Rather than moving to 3.6.x now, should we target 4.0.x?

In order to support Ubuntu 18.04, we need MongoDB 4.0.x. (https://docs.mongodb.com/manual/administration/production-notes/#prod-notes-supported-platforms)

Maybe we just skip straight ahead to 4.0.x, assuming no major performance regressions there?

[Kami]

Closing this since v3.6 and v4.0 has been supported for a while now.

Documentation change in StackStorm/st2docs#949.