Async-Await to support User Variable Session Managers

[kirsle]

This is a placeholder ticket and a new place to discuss how to implement a user variable session manager interface (to allow botmakers to automatically persist user variables in Mongo or MySQL or so), like the Python, Go and Java versions have.

The upcoming EcmaScript Async Await may finally make it possible to support user variable session managers, which can support async calls to a database by "pausing" execution within RiveScript while it awaits a database lookup before continuing.

Out of curiosity just now I was looking at the documentation for node-mongodb-native (to see whether it supported synchronous actions) and saw they were using this co library in their example snippets, which allowed for writing generators/coroutines that can 'yield' to async functions and wait for them to resolve before continuing execution 'synchronously'; the linked MongoDB example uses it to serialize database operations without chaining promises together.

It seems that with Babel we may be able to use async/await now in a way that can be backported to support ES5 (by translating async functions into generators automatically). The code changes necessary to RiveScript would probably be small and not require the massive refactoring that I expected it would to support user session managers:

• Make the underlying reply functions be async functions so they can use the await keyword.
• Build a session manager template interface that provides async functions
• Use await throughout the code when interacting with the session manager interfaces

The other thing I'll have to investigate is how well CoffeeScript supports async functions. This pull request came up after googling for a few minutes which seems to suggest that the CoffeeScript compiler will detect functions that call await and make them async implicitly, but I'll have to see how that works in reality. Otherwise, this may be a good time to drop CoffeeScript and port the code to ES2016+ so that we're not held back anymore by the limitations of third party transpilers.

[dcsan]

for those running rivescript server side, async await is working now in node7
https://blog.risingstack.com/async-await-node-js-7-nightly/

[julien-c]

Yay for porting to ES6/7 or even better, Typescript :)

[dcsan]

we've actually stopped using typescript a bit recently for new apps, now that node has async/await support... you can also get someway with instanceof for basic type checking.

typescript is still preferable, but it just adds time/overhead when bringing on new people.

[kirsle]

Updates:

I looked into CoffeeScript's async/await support and wrote the following example script that works:

#!/usr/bin/env coffee

resolveAfter2Seconds = (x) ->
  return new Promise (resolve, reject) ->
    setTimeout(->
      resolve(x)
    , 2000)

main = ->
  console.log "program start"

  res = await resolveAfter2Seconds("hello")
  console.log "res1: %s", res

  res2 = await resolveAfter2Seconds("bye")
  console.log "res2: %s", res2

  console.log "program end"

main()

The following were required to get this script to work:

• CoffeeScript v2.0.0+
  • The NPM package coffee-script (which rivescript uses) is the old name and is always a 1.x.x version
  • Instead, install coffeescript (no hyphen) for the 2.0+ releases. Found here
• Nodejs 7+
  • Simply installing the new coffeescript would cause it to generate JavaScript syntax that Node doesn't support (async function) unless you're on v7+ (Fedora 26 ships v6.11.2 at time of writing)
  • I installed nvm to get Node 8 on my local machine to test this.

---

To resolve this ticket, the following sorts of changes are needed to RiveScript (thankfully these should be very small):

• In package.json replace the coffee-script dependency with coffeescript so you'll get the 2.0+ version with the await support.
• Sprinkle the await keyword throughout RiveScript in places where async object macros may be found, and when you want them to resolve immediately.
  • Example: to get async macros in *Condition to work:
    • Around here you might go ahead and await them, like left = await @processCallTags(left, scope, true)
    • Or get closer to the root and make processCallTags() itself await promises when needed (perhaps when the param async=false because that signals to processCallTags() that somewhere at the tip of the call stack, the user made a synchronous call and RiveScript should await globally throughout that request so it can return a string at the very end)
• Update documentation for contributors to inform them that Node 7+ is required, and how to get it.
• Make sure the appropriate polyfills are being used so that we can build ES5 compatible JavaScript to support all the same legacy targets we currently do.