I recently had the mixed fortune of needing to send encrypted payloads back and forth between a service running in NodeJS and a service built in Golang. It was not a straightforward thing to do, which made me appreciate just how particular and difficult these libraries are to use, doubly so to communicate across services written in different languages. Hence, I decided to write a blog post about the intricacies of the process.
First Issue: Transport Encoding
Enciphered payloads are just strings of bits. These bit strings are not ASCII or UTF8, though. You can’t output them to your terminal, for instance, because they can completely destroy the terminal due to byte sequences that the terminal interprets as instructions. Nor can you include them as HTTP request bodies, for the same reason. They can include characters that are interpreted by HTTP parsers in weird ways, most likely causing message truncation.
So, you need a transport encoding for sending ciphertext around. Base64 works great for this… with a catch.
Golang has multiple Base64 encoding methods. Specifically, StdEncoding and URLEncoding. URLEncoding does what it says on the tin — it creates Base64 output that is safe for use in a URL. Standard Base64 includes characters like the “=” sign, which requires escaping in order to work as part of a URL. (It’s worth noting that URLEncoding Base64 is also a lot longer than StdEncoding.)
I’d written a crypto wrapper library for my Golang service that was using URLEncoding internally. However, NodeJS doesn’t have native support for URLEncoded JSON. There are some NPMs for this, but if you’re like me and you hate adding a million dependencies, you just want this to work out of the box. So, I had to switch my existing Go library to use StdEncoding. Not a big deal, but it’s a Gotcha™ that tripped me up because I just didn’t expect that NodeJS wouldn’t support this. Ho hum.
Second Issue: Terminology
If you read the NodeJS crypto library documentation, it’s a) sparse, b) full of unexplained jargon and initials, and c) uses different words from the Golang library. Because of course they wouldn’t be the same.
So, what NodeJS calls an “IV,” which is a shortening of the term Initialization Vector, Golang refers to as a nonce. Nonce <sarcasm>*obviously*</sarcasm> is a mash-word for “number used once.” And, <sarcasm>as all of us crypto experts know</sarcasm>, you should never use the same Initialization Vector more than once. Don’t we all feel better, wiser, and superior to the mere mortals who don’t know all this already.
In truth, there is a subtle difference between the two terms. An initialisation vector means “choose some random bytes, used to lock in security of the encryption algorithm.” A none on the other hand, refers to “choose a random number with the correct number of bytes, used to lock in the security of the encryption algorithm” Initialization vectors are usually appended to the message, meaning your ciphertext is slightly longer than your message. A nonce (in theory) can be derived from context. In practice however, it seems the two are used interchangeably.
For AES 256 GCM, your nonce/IV/initialization vector ought to be 12 bytes long (i.e. 96 bits). This has to do with the size of the blocks in the block cipher. An IV needs to be sized correctly for the length of the block size in the block cipher and for the specific counter method used. For GCM with AES256, 12-bytes is standard.
This is important to get right for security and also because if you try to use the wrong sized nonce, you’ll get obscure errors like:
Error: Unsupported state or unable to authenticate data
Super helpful, that. Especially because you can get this error a bunch of different ways. In truth, ambiguous errors when attempting to decrypt a ciphertext is actually an important part of the security of the algorithm. If you can get the algorithm to output different errors by tweaking the ciphertext, you can use chosen ciphertext attacks to infer details about the plaintext. In some cases, you can even decrypt the ciphertext this way.
Unrelated, if you’re using a different cipher algorithm (i.e. aes256 rather than aes-256-gcm), you might get this error instead:
Error: Invalid IV length
My solution to this problem was to generate 12 bytes of randomness to use as the nonce. I chose to prepend these bytes to the start of my encoded blob.
Lastly, secure encryption requires a mechanism for message integrity. This is done by one of many algorithms which calculate a MAC or Message Authentication Code. I’m not going to attempt to discuss the details of this here, but there is a terminology hurdle with this notion. What some algorithms and papers refer to as a MAC others will refer to as an Auth Tag, which is short for Authentication Tag which is just one of an endless string of unexplained synonyms in the crypto world. Now you know.
Third Issue: Some disassembly required
Here’s where things get really gnarly. If you look at the Golang API for sealing an AES 256 GCM-enciphered plaintext message, you’ll note that it deals only with a nonce, a plaintext, and the mysterious “additionalData”:
What’s important about this is that there is something missing.
If we look at NodeJS’s implementation of the same thing, we’ve got an extra piece of required information — the AuthTag.
So that’s an interesting little mystery. Where is this AuthTag component in the Golang library?
Also, note that the deciphering will not work in NodeJS without the AuthTag. You’ll get the extremely useful, highly unique error message:
Error: Unsupported state or unable to authenticate data
Ugh.
To resolve this mystery, I had to dig into the actual source code of the Golang library for crypto. If we crawl into the source a bit, we can see that they’re helping us out by hiding the auth tag from us entirely:
Here you can see that Golang is using/anticipating it’s own structure for the ciphertext and the AuthTag. This isn’t documented in the Golang docs, of course. However, once you know that the ciphertext produced by Golang’s GCM sealing code, you can easily write your own code to splice out these bits. My code looks like this:
The assumption here is that the bundle is a Base64 encoded string of bits with the first 12 bytes being the nonce, the last 16 bytes being the AuthTag, and all the bytes in the middle being the ciphertext. Golang auto-appends the AuthTag to the end and I wrote some assembly code to prepend this with the nonce. With each of these pieces extracted from the Buffer, performing the decryption is finally possible.
(You might note the weird ordering of parameters to this function. It is strange to have callbacks followed by additional params. This was done because this particular callback is used in a chain of callbacks and I curried some of the arguments. Without doing so, it would have meant a lot more forwarding of arguments along the chain, which results in messy code. This way, the function that decrypts the AES key only needs to call onKeySuccess with a single argument, because the other three were already bound to the onKeySuccess function. Yay JavaScript.)
For the Golang code that does Encryption, check out the example code here. Mine is quite similar.
Conclusion
Writing cross-service, secure communication is hard. Different libraries choose to implement the ciphertext and AuthTag packaging differently. Golang is particularly problematic because it does not tell you it has hidden away some of the details. Because it’s not part of the public API, it means that relying on this particular implementation is unsafe. They might choose to put it the other way around without telling you because you were never supposed to reach inside the blackbox is the first place. Alas. Alack.
Links
Basics of what these different pieces of the puzzle actually do:
A fully-baked NodeJS encryption example:
http://lollyrock.com/articles/nodejs-encryption/