backcalls: let's add them

[michaelficarra]

Copied proposal from #2662 (comment) below

---

Taken pretty much from the coffee-of-my-dreams list, the best way to explain my preferred backcall syntax is through example:

• The continuation is captured and tacked on as the final arg by default

  <- fn a
  ...
  b

  compiles to

  fn(a, function(){
    ...;
    return b;
  });

• The left side of the arrow specifies the argument list for the continuation. When only one is given, parens can be omitted.

  a <- fn b
  ...
  c

  compiles to

  fn(b, function(a){
    ...;
    return c;
  });

• And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

  (a, b) <- (_) -> fn c, _, d
  ...
  e

  compiles to

  (function(_){ return fn(c, _, d); })(function(a, b){
    ...;
    return e;
  })

  or, if we detect this case, the simpler:

  fn(c, function(a, b){
    ...;
    return e;
  }, d);

this must be preserved in the continuation by rewriting references as we currently do in bound functions. I don't think we should worry about arguments rewriting, but that's your call.

Here's a real-world example @paulmillr gave in #1942:

compile = (files, callback) ->
  async.sortBy files, @sort, (error, sorted) =>
    return @logError error if error?
    async.map sorted, @map, (error, mapped) =>
      return @logError error if error?
      async.reduce mapped, null, @reduce, (error, reduced) =>
        return @logError error if error?
        @write reduced, (error) =>
          return @logError error if error?
          do @log
          callback? @getClassName()

and here it is using backcalls:

compile = (files, callback) ->
  (error, sorted) <- async.sortBy files, @sort
  return @logError error if error?
  (error, mapped) <- async.map sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce mapped, null, @reduce
  return @logError error if error?
  error <- @write reduced
  return @logError error if error?
  do @log
  callback? @getClassName()

edit: dropped the callback position indicator, opting for "hugs" style

[00dani]

As observed back on #2662, backcall syntax works well with RequireJS:

($, someLib) <- require ["jquery", "someLib"]
codeThatUses $, someLib

Another potential use is amb:

x <- amb [1,2,3]
y <- amb [4,5,6]
amb() if x + y < 10
x * y

[michaelficarra]

Also, for any Haskell fans, yes, this is just do notation under a different name. But don't let the procedural programmers know, or they will reject it as "elitist".

[summivox]

And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker.

From the previous post I proposed ^ (caret) as non-final callback placeholder. Justification: visual hint of control flow.

(a, b) <- fn c, ^, d
# use (a, b)
e

[satyr]

JFYI, it originates from #1032.

[michaelficarra]

Also, we could always drop the concept of a placeholder and force people to force the callback as the final argument. There's @00Davo's approach from #2662 of defining higher-order functions such as flip:

flip = (f) -> (x, y, rest...) -> f y, x, rest...
<- (flip setTimeout) 250
do takeAction

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction

[paulmillr]

i’d vote for dropping placeholder +1 it brings more complexity, still

[epidemian]

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction

Using the right variable name you can have a person giving you a hug there:

<- (u_u) ->

---

I'd also vote against having the placeholder syntax; unless that concept of placeholder could be extended and used elsewhere too. In general i'm not much of a fan of this proposal, but i must admit that the use cases @00Davo mentions are quite really 😃

[shesek]

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

[ghost]

I'll throw in my hat in favor of the placeholder syntax. Just reading the code, this:

(a, b) <- fn c, ^, d
# ...
e

is infinitely more clear than this:

compose = (f) -> (c, d, callback) -> f c, callback, d
<- (compose fn) c, d
# ...
e

The latter also requires a separate "helper" function for each non-"standard" argument style that needs to be adapted for, harming interoperability with existing code.

[vendethiel]

A compose function is unneeded here. I'm in favor of the hug approach

[ghost]

@Nami-Doc Ah, missed that. Yep, that looks good to me.

[epidemian]

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

Why not? In one of the use-cases mentioned in this thread, defining AMD modules, returning a value from the backcall would be really important:

($, _) <- define 'thing', ['jquery', 'underscore']
# Very typical use-case for AMD modules: returning a constructor function.
class Thing
  foo: -> # ...

[ghost]

What if intermediate callbacks need to have a value returned? Would backcalls not be usable in this situation?

[vendethiel]

Why not?

Because we can't no-op them if chained (for example).

<- epi
<- dem i for i in [0..2]
an

I'd probably vote +1 tho.

[ghost]

@Nami-Doc I don't understand. Couldn't you tack a return, undefined, or null to the end of that snippet if you don't want it doing an implicit return?

[vendethiel]

but that would no-op the inner backcall, not the one with the loop ;-).

[michaelficarra]

@Nami-Doc: That's why you would never write it like that.

<- epi
for i in [0..2]
  <- dem i 
  an
return

or my preferred

<- epi
for i in [0..2]
  dem i, -> an
return

I hate bad code strawmen. You can write bad code in any language, we get it.

[ghost]

@Nami-Doc Ah, so

<- dem i for i in [0..2]

translates to

for i in [0..2]
  <- dem i
    # ...

and not

<- dem (i for in in [0..2])
# ...

?

[vendethiel]

Considering
dem i for i in [0..2]
is
(dem i) for i in [0..2]
probably - coco treats it as an invalid callee (array)

[ghost]

Thinking about this a bit more, conditionals and loops complicate things quite a bit. Consider:

if a
  b = <- fn c
  d = b
else
  d = e()

console.log d

b =
  if a
    d <- fn c
  else
    d <- fn c

console.log b

console.log
  for item in array
    element <- transform item

result = null

console.log
  until result?
    result <- generate

Lots of messy edge cases to handle.

[vendethiel]

Why are they complicated in your examples?

[michaelficarra]

@mintplant: They don't complicate anything at all. The captured continuation is the rest of the block. Maybe you're thinking it's the rest of the function/program?

[ghost]

Ah, I see what you mean. I suppose I misunderstood the scope of this syntax addition. So, these only help with non-branching code paths, then?

[epidemian]

dem i, -> an

WTF happened to my name? xD

Anyway, between all those messages i got lost. The consensus was that it does make sense to return values from backcalls, wasn't it?

[michaelficarra]

@mintplant: They work perfectly fine with branching code paths. It is a very simple transformation, you're overcomplicating it.

@epidemian: Yeah, it makes sense to auto-return from backcalls. That's why satyr/coco does it.

[vendethiel]

WTF happened to my name? xD

Sorry, no idea why I came up with that :p . And yeah, that'd make sense.

[ghost]

@michaelficarra Right, I get that now. I never said they didn't work with branching code paths, just that they don't provide any special functionality to help account for them, which I had hoped they would, but now see is outside the scope of this change. Sorry for the misunderstanding.

[00dani]

The "hug-style" <- (u_u) -> is amazing.

As for the higher-order function approaches, this seems problematic:

compose = (f) -> (c, d, callback) -> f c, callback, d

Because it looks nothing like a standard definition of compose and has totally different semantics.

I think very simple higher-order manipulations, like flip, are reasonable to use, though, unless they have a big performance impact at runtime (although I doubt they'd really have a big impact?).

[michaelficarra]

Hugs it is! I will edit the original proposal to omit the callback position indicator.

edit: done. Copied below:

---

And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

(a, b) <- (_) -> fn c, _, d
...
e

compiles to

(function(_){ return fn(c, _, d); })(function(a, b){
  ...;
  return e;
})

or, if we detect this case, the simpler:

fn(c, function(a, b){
  ...;
  return e;
}, d);

[vendethiel]

This would make @tenderlove happy

[danschumann]

Also, what about other types of deferred? Are we expecting the callback to be the last argument in the function?

# consider
$.post('/url').sucess(callback1).error(callback2)

# OPTIONAL ( ERROR OR SUCCESS )
post_deferred = $.post('/url')
(success_args...) <~ post_deferred.success
(error_args...) <~ post_deferred.error
if success_args
  return 'cool'
else
   return 'doh'

Hey look at that, a pretty decent reason to use ~, optional callbacks. What if we have multiple callbacks like this? How do we break them up? We could use <~~ to start a chain of optional callbacks and <~ for each subsequent one.

• 1 for options

[danschumann]

Also, what about parallel?

I don't see the need for async anymore, the whole point is to make coffeescript good enough to do it on it's own, right?

(err{users}, users) <== users_collection.fetch()
(err{pages}, pages) <== pages_collection.fetch()
# now a single arrow to execute the parallel functions
(err{documents}, documents) <= documents_collection.fetch()
# Err could be {users: 'user error', pages: 'pages error', documents: 'documents error'}

There is no err{users} right now, because there is no parallel / separate declaration of a single value. == two arrows, means you're doing two things(at once), in parallel.

In the past my suggestions have been shutdown, probably due to the difficulty that would be in implementing them, so it could end up being that a new compiled js language emerges, inspired by the beauty that is coffeescript, while fulfilling my wildest dreams.

[00dani]

@danschumann

maintaining the same arrow syntax makes sense

We can't have <= mean a bound backcall, because <= already means "less than or equal to". This is why Coco switched to ~> and <~ for bound functions, since that introduces no such ambiguity.

Are we expecting the callback to be the last argument in the function?

Yes. In earlier iterations support for putting a "placeholder" elsewhere (to relocate the callback) was suggested, but eventually it was decided that simply wrapping the expression with a function would suffice. Look for the "hug operator".

Hey look at that, a pretty decent reason to use <~, optional callbacks. What if we have multiple callbacks like this?

By far the most common pattern for callbacks is to use a single callback with an (err, res) signature: a nodeback. Backcalls are designed to work with a sequential chain of async calls following that pattern. There's really no particular support for a call that requires multiple callbacks, because such calls don't correspond to the concept behind a backcall, that being that <- is nothing more than a magical async =:

x =  someSyncCall  arg, arg
y <- someAsyncCall arg, arg
f x, y

Most calls that require more than one callback just don't make sense if you view <- as magic =. There's one common case that takes multiple callbacks and also makes perfect sense when used with backcalls, however: promises, which can take a normal callback and an error-case callback. Fortunately, however, you can still use them with backcalls easily enough, since the backcalls simply provide the success codepath:

p = (pr) -> (f) -> pr.then f
backcallCode = ->
  x <-p somePromiseCall arg, arg
  y <-p someOtherPromiseCall arg
  x + y
directPromisesCode = ->
  somePromiseCall(arg, arg).then (x) ->
    someOtherPromiseCall(arg).then (y) ->
      x + y

Error is simply propagated past the backcall chain, invoking none of the success-path calls. This essentially corresponds to the way the promise monad's error-handling basis, the Either monad, works.

I don't see the need for async anymore, the whole point is to make coffeescript good enough to do it on it's own, right?

Actually, no. If we were trying to make CoffeeScript's asynchronous support complex/powerful enough to model all async patterns by itself, we would probably have jumped straight to IcedCoffeeScript. The problem is that nearly all asynchronous patterns other than simple serial will compile to JavaScript code that's a lot less pleasant: Running calls in parallel requires some kind of reference-counter to be declared and tracked, for example, and at the deep end we end up with the monstrosity that is CPS-transformed JavaScript.

Backcalls are a simple enough syntactic transformation that the compiled JavaScript is not particularly horrible. All they do is flatten callback pyramids. They do not and will not model all async patterns, because that need is much better served by a library such as caolan/async. Such libraries would be used in conjunction with backcalls:

(err, [one, two]) <- async.parallel [oneF, twoF]
codeUsingAsyncResults one and two

[xixixao]

-1 Let's not. Promises or generators are the way out of callback hell, otherwise I (did) would use IcedCoffeeScript.

[00dani]

@xixixao Absolutely they are, but backcalls a) work quite well with promises in conjunction with a helper like the p I defined earlier and b) also work for things other than the callback hell promises solve. They're superior to using IcedCoffeeScript specifically because the syntactic transformation is so trivial: Your compiled JS isn't going to become CPS-conversion hell when you use backcalls.

Generators are definitely a more flexible way to make promise code look synchronous, but they're still not available in every browser, nor do they exist in CoffeeScript yet either. Meanwhile, generators are not going to help you at all with cases like:

export = (x) -> module.exports = x
grunt <- export
grunt.loadNpmTasks 'whatever'
grunt.allYourGruntStuffHere withNoIndent

Or:

$, _ <- define ['jquery', 'underscore']
requireJS.module goesHere withNoIndent

Or even:

result = do ->
  x <- amb [1..10]
  y <- amb [1..30]
  fail() if x * y < 10
  x*2 + y*2

[kibin]

Is this still valid? Does somebody investigate/develop them?

[zhaizhai]

A while ago I prototyped a basic implementation:

https://github.com/zhaizhai/coffee-script/tree/backcall

In particular, look at

https://github.com/zhaizhai/coffee-script/blob/backcall/test/backcall.coffee

In my opinion it ended up being weird to not be able to write if statements and for loops normally with "backcalls" inside, so you would really need to compile the appropriate CPS transformations on those constructs to make things feel natural. At that point maybe you should just use IcedCoffeeScript.

If you're interested the code linked above should still work, but presumably it's a ways behind the main branch by now. It seems like people are not that interested in this feature anymore. It's an interesting discussion topic and attempts to address a real problem, but nobody (including me) actually tries seriously to get it done 😛 .

[kibin]

I see, thanks!

[jashkenas]

In my opinion it ended up being weird to not be able to write if statements and for loops normally with "backcalls" inside, so you would really need to compile the appropriate CPS transformations on those constructs to make things feel natural. At that point maybe you should just use IcedCoffeeScript.

That's really useful feedback on the feature. Thanks for sharing it!

[vendethiel]

I don't get how you can't use if and stuff in a backcall ? You're saying the backcall doesn't work like await. We're aware of that, that's not what we're proposing here.

EDIT : We're not proposing an alternative to iced's async, but something much, much more general.

[jamesonquinn]

I think the thing with using if statements is talking about a situation like the following (using =< for the backcall operator, as discussed above):

  if is_async get_x_using
    x =< get_x_using y
  else
    x = get_x_using y
  ....

It's true that this makes things hairy for the compiler. Of course, the whole if/else block above could be refactored into a backcallable function; but zhaizhai is saying that, upon playing with it, that seems artificial (if the programmer does it) or ugly (if the compiler does).

I'm not sure what the right answer is, but it's probably worth leaving this one open for a little longer to see if somebody comes up with a good solution for this issue. For instance, I suspect you could make a "hugs"-like idiom for a workaround....

[vendethiel]

But -- as I said -- we don't want <- (or =<) to be async-like. Just a callback that fits better.

[danschumann]

Looking back on this again it just looks confusing. Won't this be sort of moot when generators and yield become the norm?

[vendethiel]

co rather than generators

[mmotorny]

I skimmed through the thread but didn't find a convincing example why monadic notation is useful, so I decided to compile my own. Please review: https://docs.google.com/document/d/1pmvd6Gd-Scj06dB6uLunU52eNpMCjAqM9t6HobOJ758/edit#

[QxQ]

Just a thought an alternate syntax for this, maybe this would be easier for people to understand?

What if we did "->..." meaning "make a function, but the indentation for it is on the same line", so...

callme param, (arg) ->...
codeHere()

is equivalent to

callme param, (arg) ->
    codeHere()

Also, for the problem with the function parameter in a funny place, it could be solved like this:

setTimeout ->..., 100
codeHere()

is equivalent to

setTimeout ->
    codeHere()
,100

[taoeffect]

I just saw that ES7 will have async and await keywords. So what does that imply for coffeescript?

Should I open another issue for async and await support in CS or is this issue sufficient? cc @jashkenas

[jashkenas]

Feel free to open another issue that specifically talks about what adding the ES(7) keywords would look like — and how we might compile them.

I don't think it's something that's possible without large-scale runtime support, no?

[aurium]

+1 to @QxQ syntax ->...

[Baudin999]

I appologize for the long comment, please forgive me; but; I really think that a backcall operator obfuscates the code, imagine this:

result1 <- foo a
result2 <- foo b

Now, does this traslate into:

function foo(a, function (result1) { /* do something */ }
function bar(b, function (result2) { /* do something */ }

or:

function foo(a, function(result1) { 
    function bar(b, function(result2) { /* do something */ })
})

with the proposed syntax you do not know whether or not the second function should be a part of the inner-function of the first or if it should run in parallel.

I would rather wrap my code in a yield statement like so:

result1 = yield foo a
result2 = yield foo b

this would resolve into the first option. For the second option you would get:

await result1 = yield foo a
result2 = yield foo b

This results into much cleaner code. Now resolving a promise becomes almost trivial:

promise = do -> foo a
await result = yield promise.done
# do something with the result
console.log result

Now for the problem of the order of the parameters and the callback being somewhere in the middle, combined with mutiple parameters within the callback:

r1, r2 = do -> foo a, yield, b
# you could rewrite the first example to:
result1 = foo a, yield
# but what would be the point?

Here you sacrifice readability for distance to code.

[00dani]

@Baudin999 Backcalls always desugar into the second of your proposed syntaxes, i.e.,

a <- thingOne
b <- thingTwo
moreCode
// *always* means
thingOne(function(a) {
  thingTwo(function(b) {
    moreCode;
  });
)};

This is consistent with the behaviour of backcalls in Coco and LiveScript, Haskell's do notation, and the intuitive sense that a backcall is a "magical async = operator". If you want parallel operations, you'll still need a "normal" async helper library. For example:

{items, otherStuff} <- async.parallel items: getItems, otherStuff: getOtherStuff
# or with promises and a 'p' helper as discussed earlier
results <-p Q.all [firstThing(), secondThing(), thirdThing()]

Allowing for distinct yield and await keywords would be more powerful but also vastly complicates the compiled JavaScript - take a look at IcedCoffeeScript, which provides those exact features, for instance.

[maxtaco]

FWIW, I'm contemplating switching IcedCoffeeScript over to a generator-based transpilation, which is of course much simpler. Here's a hand-compiled example of how it can work.

[GeoffreyBooth]

Skimming through this, it seems that the consensus was to not implement this feature. As such, I’m closing this ticket. I think at this point if someone wants to re-propose backcalls, please start a new issue or pull request. Any new proposal should be aimed at the 2 branch, and take into account its support for async/await.