switch from protobufjs to google-protobuf

[renkei]

protobuf.js version: 6.8.8

This is rather an experience report than an issue. But maybe, this is helpful for some of you.

Since this project seems not to be maintained anymore and makes more and more trouble (see the increasing count of issues here) when used with Node.js 12 or newer, I've decided to replace protobufjs with the original protobuf runtime maintained by Google, it's the npm package google-protobuf.

Short result:

• It works great and I don't want to go back anymore.

Long result:

• The public interface of a generated class is different to the one generated by protobufjs, but it follows more or less the design of the classes you get for other programming languages with protoc. Anyway, you have to port your code.
• 64bit integers are not implemented with long.js. Instead, Google allows the extension [jstype = JS_STRING] for related message fields in proto files. The 64bit number is then available as string value, from now on, you can easily read the string with long.js or Bigint for further processing
• With google-protobuf, every message class is derived from jspb.Message, means that every class has the methods getExtension() and setExtension() for accessing the Google Protobuf Extension API. If you define in your proto file a message with a field called "extension" then this would result in a class with methods setExtension() and getExtension(). Because these methods are already defined by jspb.Message, protoc automatically adds a $ to your methods, in the end you get setExtension$() and getExtension$() for your field.
• protoc is currently not able to create TypeScript definition files, this is usually a no-go if you work in a TypeScript environment, but fortunately there is a workaround. grpc-web is able to create the required TypeScript definition files and can be used as protoc plugin. See chapter TypeScript for more details.
• This still seems to be in progress, for example, in future, protoc shall be able to create TypeScript code out-of-the-box, but at least it works mostly fine even today and also works with Node.js 12 or newer.
• Unfortunately, it could happen that you need some post-processing after creating the TypeScript definition files. For instance, grpc-web is not aware of the automatically renamed methods getExtension$() and setExtension$(), means that you have to add the $ manually to the d.ts file

In the end, after some work, it works without any further issues. I can use protoc now for all programming language targets like C++, C# and even JavaScript and TypeScript (with the grpc-web plugin), not that bad.

It's a little bit sad because protobuf.js was a great community project. But for now, I would try to avoid it and maybe my remarks are helpful to find an alternative way.

[FanAs]

Hello @renkei, what about perfomance? Is it fast enough?

[renkei]

I didn't recognized any perfomance issues so far. But, to be honest, I didn't executed any benchmarks or something similar. Since JavaScript is an officially supported language by Google's proto compiler I would expect that Google has done some optimizations an will continuously optimize whereever possible.

However, if someone has some benchmark results or other hints, I'm always interested in.

[anselanza]

I am currently building a distributed system where most components use the standard protoc compiler with TypeScript definitions generated via a plugin (https://www.npmjs.com/package/ts-protoc-gen), but I still have a use for protobufjs because it provides some functionality that protoc compiler does not; most importantly it can load protobuf files at runtime, and use these to decode (or encode) messages.

Unless I'm missing something, this is not really possible with the "normal" google-protobuf compiler?

[benjamin-rood]

@renkei What about that there's no equivalent (that I can see) for an encode method or initialising a message object from a JS object? How do you like having to call myMsgObj.set(x) for every single field in your proto message definitions that you need to use? It's horrible and brittle as hell, makes refactoring a nightmare.

The google-protobuf implementation is always going to be up to date, but as they say on their own readme, "The API is not yet well documented". 3+ years later it still isn't, and it's a pig to use, as well.

[chadi-kazan]

Hi I am trying to do the opposite. I am noticing google-protobuf can be slow in some scenarios so i am experimenting with protobufjs. Are the benchmarks listed in this repo still valid as of the latest version of google-protobuf? is it worth it?

[derhuerst]

I'd also be interested in a performance comparison with (the compiled version of) protocol-buffers. It doesn't seem to support proto3, but otherwise works hassle-free.

[alexcardell]

We're actually switching in the other direction because google-protobuf for JS doesn't follow the canonical proto3 -> JSON mapping, and protobufjs does. However I do find that having the central interfaces of jspb.Message etc is incredibly useful, and I wish protobufjs had something similar.

[isherman]

@alexcardell are you sure that's the case? There are number of open issues (e.g. #1304) here about the lack of support for canonical proto3 <-> JSON mapping. Would love to know if you're experiencing otherwise.

[alexcardell]

I suppose I meant that within our use cases (as well as the fact the JSON usage is an intermediate step) the mapping seems fine, unlike google-protobuf which appends List to anything repeated.

[foreverzmy]

hello， i set [jstype = JS_STRING],but not working in js, the d.ts is working，so I want to know are you has this problem?

[zim32]

google-protobuf is total mess with a lot of opened issues and wrong javascript code generation. Do not recommend to use

[nhayfield]

One thing that is annoying is that you can convert the protobufs to json with toObject which works well with Redux. However a big gotcha is there is no conversion fromObject method. So you have to build a whole separate middle layer to convert it back. Or else pass the whole protobuf around instead of passing around smaller bits of state.

[vitalets]

I've also faced inconvenience with google-protobuf, especially missing fromObject feature.

Finally I've stayed on google-protobuf but with special wrapper function. It extends GRPC client instance with following:

• all methods return Promises
• all methods can accept plain object as request (when possible)
• all methods resolve with plain object response

Maybe you find it useful: https://gist.github.com/vitalets/c52c3efa40fc1154ccc6639903758565

[lorensr]

For those looking for JSON serialization with protobufjs, check out:

https://www.npmjs.com/package/proto3-json-serializer

[foreverzmy]

您好，您的来信我已经收到，请等待回复！