Invalid wire type and index out of range errors #602
Comments
|
Well, hard to tell. I assume that the buffer becomes corrupted because it is reversed or something like that, in particular: //slice does some bad things to original data
// ^ I think it's "reverse" because it moves around your data in place
type = pl1.slice(0,4).reverse().readUIntBE(0,4); // 0,4 doesn't seem like anything readUintBE accepts to me
size = pl2.slice(4,8).reverse().readUIntBE(0,4);
message = pl3.slice(8,pl4.length).reverse(); // why do you reverse the protobuf blob?could probably just be If I understand your code correctly, then you are reading fixed32 LEs prior to the message anyway , which you could also do like that: function getPayloadData(data){
protobuf.load("../proto/PegasusGame.proto",function(err, root){
var PowerHistory = root.lookup("PegasusGame.PowerHistory");
var reader = protobuf.Reader.create(data);
var type = reader.fixed32() >>> 0;
var size = reader.fixed32() >>> 0;
if (type === 19) {
var myPowerHistory = PowerHistory.decode(reader/*, size */); // if size is the length of the protobuf message
console.log(myPowerHistory.list);
}
});
}You should of course cache the |
|
I appreciate you taking the time to answer this. I have modified my code based on what you did and it seems to not throw the wire error now, and it seems to be at least decoding the message against the .proto file now. I shall verify the output after I get done with some post new year's stuff. I'll explain what I did with type/size/message in my code. Here is an example buffer from a TCP payload that I want: <Buffer 13 00 00 00 5e 01 00 00 0a 0a 32 08 08 07 10 00 18 1a 20 00 0a 08 22 06 08 02 10 19 18 03 0a 09 22 07 08 02 10 a2 03 18 20 0a 09 22 07 08 02 10 8d 02 ... > The first 4 bytes are 0x13000000 which in decimal is 318767104 , if I reverse it, then it becomes 0x13 = 19dec. However, you didn't do any reversal at all and the reader seemed to pick up the 19 just fine. I assumed everything came in backwards so that is why I reversed everything. Also the reason that I made so many buffers after slicing was because of this: "Note that modifying the new Buffer slice will modify the memory in the original Buffer because the allocated memory of the two objects overlap." Perhaps I have a fundamental misunderstanding in how javascript buffers work? I don't understand how reverse would corrupt anything. Are there any good resources for understanding this better? Edit: I have validated outputs and it seems to be decoding the messages correctly, but occasionally, when I receive very specific packets, it will still throw invalid wire type like below. It only happens on a certain type of action that I expect to read from the packet. |
Sure! Regarding reversed bytes: Little endian byte order (LE) encodes the least significant byte first while big endian (BE) encodes the most significant byte first, but this is only relevant for fixed sized integers and IEEE754 floats, NOT for entire protobuf messages. (see: Endianness) Regarding buffers: When sliced, buffers create a shallow copy with shared memory. Hence, if you modify the data of one buffer, you also modify the data of the other if the data is in a region visible to both slices. This is most likely not the cause of the invalid wire type-issue, though. To understand what's wrong with the buffer, you'd need to reverse engineer the wire format. Stripping out the first 8 bytes, this is what we have: There are a couple of examples on how this can be done starting here: #55 |
|
We're having the same issue. We updated from 5.0.1 which was decoding fine, to version 6.3.1 which has the behavior described here, certain protobuf messages cannot be decoded and throw wire type errors |
|
I've created an example for your issue: Hope it helps! |
|
That you for being so thorough. So then there are no wire types 6 or 7? If we find the point in the buffer where the error occurs, what should be done or can anything be done at all? Where are the points that a buffer can be corrupted, if that is the problem? I'll print a buffer tonight and go over it tomorrow morning on my commute. |
Well, we'll know one of two things: Either the buffer is corrupt or there is a bug with protobuf.js. In both cases there is something we can do about it.
The usual suspect here is string conversion (see). |
|
So I have taken the time to pan through a couple of buffers to look for any wire types that matched the error given or otherwise. Here are the long results All messages appeared to be length delimited. So perhaps it is the data that is wrong. However, if I continue reverse engineering the buffer, I run into wire type 7 before wire type 4, which is what protobuf.js decided the error was. For the second one, I did not find an error in the data when verifying the encoding, so I have no idea why an invalid wire type is thrown. What do you think? |
|
I'd rather guess |
|
I agree that 0a is not a varint and the tag is 1 and wire type is 2, and b7 01 is the varint length because wire-type 2 is length delimited. b7 01 is 183, and 183 bytes follow, which was a mistake on my part in the previous post. I'll come back again after I looked at the individual messages. |
This buffer segment seemed to be the culprit for throwing wire type = 4, from looking at it. I don't know what to do with this information other than create my own protobufjs, which is probably impractical. I would just throw out the message and ignore it if it threw a wire type because the rest of the buffer looked fine. |
|
It says int32, but the data is encoded otherwise. |
|
What happens when you change that to |
|
With this commit, protobuf.js does not rely on reflection information anymore to decide whether to generate a decoder for packed or non-packed repeated fields. It now just checks the wire format and acts accordingly. While I am not 100% sure that this caused your issue, it is likely. Give 6.4.3 a try and let me know! |
|
That fixed everything. Not seeing a single error anymore! :D Thank you for working through this with me and producing this awesome module! |
protobuf.js version: 6.3.1
I am a novice, so bear with me on the below explanation. Please do not mind the excessive buffer declarations.
I am reading a TCP stream with cap from npm to obtain a TCP payload by listening to a particular port. This is the structure of the packet:
struct PegasusPacket {
uint32_t type;
uint32_t size;
uint8_t *message;
};
The first 4 bytes are the message type
The next 4 bytes are the message size
The remaining size bytes are the protocol buffer encoded message
The packets themselves don't seem to have any issue because I can always extract the type and size faithfully, but when I try to decode the message from my .proto file, things start to go wrong. I have included some buffer examples and stack trace results below. The buffers are the expected message after being reversed.
I also had an issue with index out of range, but I was unable to generate that error for this issue. I think it resolved when I reverse()'d the message buffer..
The text was updated successfully, but these errors were encountered: