Real-Time features

[jonathan-casarrubias]

Description/Steps to reproduce

I'm opening this thread to discuss and define a scope for real-time functionalities possibly provided by LB-Next.

Currently there are implemented functionalities like event streams, but honestly are not helpful at all when building real life applications, therefore I have built several real-time approaches on top of loopback, which I believe 2 were the most important.

1.- Implemented PubSub functionality.
2.- Implemented Firebase alike interface.

I want to start the conversation because I know from @ritch expects to provide better RT Interface(s), I will talk about my experiences by sharing the pros and cons I consider from these approaches.

PubSub (Hooking from REST CUD -create, update, delete- Calls)

Pros

• Small learning curve
• Easy to subscribe to CUD methods
• Does't require a complex client sdk (for instance can subscribe using plain socket.io)
• Lighter in terms of processes
• Easier to implement
• Easier to scale

Cons

• If hooking from REST calls, triggers won't fire when using LoopBack's NodeJS API
• Requires HTTP Calls to trigger WS Event, unless we provide the exact same REST Api but using WS Events instead
• Is too coupled to the REST API

NOTE: Sails.js out of the box offers PubSub events to be subscribed from any REST Endpoint

Firebase alike API

Pros

• Similar to ORM in the back-end but for realtime communications from the front-end
• Does't depend on any REST interface at all, is more model oriented (like real-time ORM)
• Richest interface of communication, allows different behaviour and not just subscribing to latest event change
• Allows to build more complex real-time applications by allowing every subscriber to receive filtered data (Every client receives their custom requested data, is not generic to everyone)

Cons

• Higher learning curve
• Requires to provide complex client sdks (Will be difficult to provide different clients for different languages)
• Managing subscriptions is harder and more difficult to scale since every request from different clients might be different for the exact same event (Example in paginated lists)
• Can be heavier in terms of processes
• Is more difficult to scale while attending unique client requests for same event

That is all I can think for now, but is only to start the conversation... There might be more cons and pros from these 2 approaches I already offer while using the modules I maintain.

Looking forward to see what your plans are on RT Features.

Cheers
Jon

[ritch]

@jonathan-casarrubias thanks for bringing this up. Realtime is an important use case for node. As you mentioned, our support for it in 3.x isn't really that powerful. I think that is a result of loopback core trying to do to many things itself instead of making it easy for someone like you to go build something like FireLoop on top of it.

Given your experience building FireLoop on LoopBack 3, what sort of things would you like to see in the loopback-next core. Right now we are spending most of our energy with this use case:

// controller example
class MyController {
  @GET('/echo')
  public echo(@inject('msg') msg) {
    return msg;
  }
}

A generic realtime version of this is something like this...

// simple realtime example
// server
server.on('connection', connection => {
  connection.send('ping')
});

// client
connection.on('message', => {
  connection.send('pong');
});

What I'm not so sure about is what the simple realtime example would look like in the context of LoopBack.next...

Its worth thinking about the differences between the two examples:

controller example is

• responding to a client message
• declaring the function and telling lb when/how to execute that function
• injects its dependencies, doesn't go out and create a server

simple realtime example is

• sending a message to a client right away
• imperative, it builds out everything and calls the right functions
• gets and constructs its dependencies (server)

[jonathan-casarrubias]

@ritch thanks for clarifying, I was unsure about the scope and plans. But from what I read here and in other issues like the ORM one (#117), I think you guys learned not trying to build everything your selves.

Also, in FireLoop there is a lot of client side work that I don't think you guys want to deal with, so if I'm correct and we are in the same page, these are the down level features I would expect from you to provide, so we can build more complex interfaces on top like fireloop.

• 1. Scalability.- I'm assuming you are implementing pure websockets and not a library like socket.io, if this is the case... How will you scale between servers? socket.io uses adapters to notify other server instances about certain events, this way all clients from different servers are notified. Is not just a vertical communication server <-> client, but also horizontal server <-> server.

• 2. Flexibility.- Basically I need to be able to send events from either, custom remote methods, remote hooks and operation hooks and components.

• Custom Remote Methods.- This allows users to send any custom data to clients, this is the most basic use case.

• Remote Hooks.- This allows me to provide a PubSub functionality, people subscribe to remote methods and I publish to subscribed clients.

• Operation Hooks/LB Components.- The FireLoop API is URI agnostic, it doesn't care about remote paths, but it relays on model operations, so to keep the FireLoop Api requires the basic real-time example to be accesible from Operation Hooks and from components.

If you provide access to the basic real-time example within the 3 contexts defined above is a good start point.

• 3. Event customization.- The FireLoop API is really complex, is too client specific and having a single generic "message" event would require to send huge amount of unexpected information to all the clients, every client receives custom data according custom LoopBack filters, so allowing to create custom real-time events others than just "message" would be the expectation, similar to socket.io.

These are the main 3 points I can think for now, If you manage to provide these it would be helpful.

Of course in LB2 and 3 I managed to create a realtime server on my own and distribute that within the different framework's contexts to be able to send the information at the right point, so if you deal with these I can deal with the complex flow and complex client implementations.

I'll keep adding more information as I can think on

Regards
Jon

[Brian-McBride]

There is an interesting server/tech called deepstream.io that is a real-time server that in opinionated to the database it connects to.

Their license is going to preclude it from working with Loopback I suspect - however if you look at their core server source it is node.js based.

With some elbow grease, someone could hook the two together, perhaps through a dataconnector and listeners to the deepstream events. There would be some work unifying auth and the roles engine so that they align in an easy way too.

[kjdelisle]

I like this idea, but I think it's probably best implemented as a component rather than a part of the core offerings of loopback-next. Our intent will be to release beta soon(ish), and attempt to keep the component APIs stable so that developers can get a head start on creating their own components.

@raymondfeng @bajtos @virkt25 thoughts?

[bajtos]

+1 for implementing this as a component/extension. However, we should ensure that our core is flexible and extensible enough to make this extension easy to build.

[jskrzypek]

Fwiw, it might be worth looking at feathers. They also have an opinionated model of realtime eventing backing standard CUD operations, like deepstream.io, but they're entirely OSS, I believe.

They have some overlap in functionality with the goals of loopback, providing a pretty basic, if extensible ORM with various database connectors, but like loopback-next they have also recently decoupled from express, and it's entirely possible to run feathers purely as a realtime/socket server.

I would certainly love to see and would also be interested in contributing to an extension/connector that could bridge the gap between them.

[kattsushi]

@jskrzypek i would like to collaborate in this feature from beginning with an extension that provides feat to some controller or method in loopback-next maybe with Socket.io or WebSocket, im trying to figure out yet how works, but already i started working on it here https://github.com/kattsushi/loopback4-extension-socketio based in loopback4-extension-grpc @jonathan-casarrubias has working in this one and he has a broader vision about the extensions, and it seemed like a good starting point

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[collaorodrigo7]

Have you guys started some work on this? I am not an expert, but I would truly like to help

[jannyHou]

Hi @collaorodrigo7 thank you for being interested in adding real-time features in LoopBack 4!
Recently the team is focusing on several fundamental epics, like cloud native, new boot module, new relation, http server refactor, etc... We haven't work on the real-time functionalities and appreciate your help.

Maybe we can start with describing our targets:
Based on the discussion above, any specific feature(s) you would like to implement?

Here are some references may help you:

LB4 provides a default rest server to process HTTP requests, see concept server, it also talks about how to customize the sequences and actions to modify the default behaviour.

User can also create their own server, see creating server

To get familiar with contributing extensions in LoopBack4, we have an example repo that demos various approaches to extend the framework:
https://github.com/strongloop/loopback-next/tree/master/examples/log-extension
And the doc creating components

[collaorodrigo7]

Thanks for the references @jannyHou.
I was reading at the posts and the options mentioned before on the discussion as feathers and deepstream
All pretty interesting
While looking at them I run into gun.eco (they call themselves self-hosted firebase),which called my attention because it is/has (according to their docs):

• Realtime updates
• Distributed (peer to peer by design)
• Offline first
• Graph database

Now, I am not sure if that will be something you guys want or think that will be good to have on lb4 or will be in the scope of the project. I am not sure but I don't think we have connectors for graph databases, do we?

Answering your question, I would like to implement the real-time updates keeping in mind offline capabilities.

Please consider it is my first time contributing to an open source project, so I have no previous experience, but I will try my best.

[matejthetree]

any updates on socket implementation in lb4?

[virkt25]

@matejthetree It's not part of the current scope of work we're doing for General Availability (GA) of the framework. Most likely it will be after that unless you would be interested in helping out by implementing it? The team would be more than happy to help you in any way we can along the way.

[matejthetree]

I was thinking of creating just my own socket server and router(like rpc example). How different is this from creating an extension for socket.io

If it is not a huge scope, I would be happy to do it.

[basitsattar]

Hi, any update about this?

[raymondfeng]

See #1884

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.