WARN invalid UTF-8 at transliterate (transliterate.c:790) errno: Resource temporarily unavailable

[johnhamelink]

Hi there,

I'm working on an Elixir NIF for libpostal (mainly just to learn how to build NIFs to be honest). When I retrieve the binary string data from the Erlang VM and copy it into a signed char, I pass it through to libpostal to parse/expand the address input. It seems to work perfectly around 20% of the time, and all the other times I instead get the following response:

WARN  invalid UTF-8
   at transliterate (transliterate.c:790) errno: Resource temporarily unavailable

I would've assumed that the problem was in my code (it probably still is) but the errno: Resource temporarily unavailable as well as the fact that /sometimes/ it does work has thrown me off...

Would you be able to provide any insight?

You can check the code out here: https://github.com/johnhamelink/postie

[albarrentine]

Hi John - from my hazy recollection of Erlang, strings are represented as linked lists and then there's a more efficient type called a binary which is a pointer to a character array and its size similar to strings in C++, Python, etc. and that's the type you're using (bravo). I'll assume that the original string is already UTF-8 encoded (if not, that's what libpostal expects so should check/ensure its encoding on the way in).

The problem, I would guess, is that the Erlang string is not NUL-terminated ('\0' at the end) and that's how C expect strings to be represented. Most string operations in C, including those used in libpostal, will start at the pointer address and continue to read bytes from memory until a zero is encountered. When it occasionally works, that means there happened to be a zero somewhere beyond the boundaries of your string and the intervening "garbage memory" happened to be valid UTF-8. The simplest case would be if there was a zero in memory at strlen + 1, in which case it would behave like a normal C string.

So you'll want to create a NUL-terminated C string from the Erlang binary before passing it to libpostal. Haven't tested this, but something like changing https://github.com/johnhamelink/postie/blob/master/src/postie.c#L78 to char *address = strndup(in_binary.data, in_binary.size); should do the trick. Note that strndup is a caller-frees function so you'll also need to call free(address) somewhere after the libpostal call to free up the allocated memory.

[johnhamelink]

@thatdatabaseguy Thank you for such a clear and definitive explanation! Adding in that line did the trick, and because of no extra random data making its way into the libpostal call, the responses have become much less erratic as well, which also makes my unit tests work better.

I will keep working on it, and then perhaps I can submit a PR to add postie to your list of unofficial libs?

[albarrentine]

No problem, and yes, happy to accept pull requests!