Communicating via AES 256 GCM between NodeJS and Go(lang)

I recently had the mixed fortune of needing to send encrypted payloads back and forth between a service running in NodeJS and a service built in Golang. It was not a straightforward thing to do, which made me appreciate just how particular and difficult these libraries are to use, doubly so to communicate across services written in different languages. Hence, I decided to write a blog post about the intricacies of the process.

First Issue: Transport Encoding

Enciphered payloads are just strings of bits. These bit strings are not ASCII or UTF8, though. You can’t output them to your terminal, for instance, because they can completely destroy the terminal due to byte sequences that the terminal interprets as instructions. Nor can you include them as HTTP request bodies, for the same reason. They can include characters that are interpreted by HTTP parsers in weird ways, most likely causing message truncation.

So, you need a transport encoding for sending ciphertext around. Base64 works great for this… with a catch.

Golang has multiple Base64 encoding methods. Specifically, StdEncoding and URLEncoding. URLEncoding does what it says on the tin — it creates Base64 output that is safe for use in a URL. Standard Base64 includes characters like the “=” sign, which requires escaping in order to work as part of a URL. (It’s worth noting that URLEncoding Base64 is also a lot longer than StdEncoding.)

I’d written a crypto wrapper library for my Golang service that was using URLEncoding internally. However, NodeJS doesn’t have native support for URLEncoded JSON. There are some NPMs for this, but if you’re like me and you hate adding a million dependencies, you just want this to work out of the box. So, I had to switch my existing Go library to use StdEncoding. Not a big deal, but it’s a Gotcha™ that tripped me up because I just didn’t expect that NodeJS wouldn’t support this. Ho hum.

Second Issue: Terminology

If you read the NodeJS crypto library documentation, it’s a) sparse, b) full of unexplained jargon and initials, and c) uses different words from the Golang library. Because of course they wouldn’t be the same.

So, what NodeJS calls an “IV,” which is a shortening of the term Initialization Vector, Golang refers to as a nonce. Nonce <sarcasm>*obviously*</sarcasm> is a mash-word for “number used once.” And, <sarcasm>as all of us crypto experts know</sarcasm>, you should never use the same Initialization Vector more than once. Don’t we all feel better, wiser, and superior to the mere mortals who don’t know all this already.

In truth, there is a subtle difference between the two terms. An initialisation vector means “choose some random bytes, used to lock in security of the encryption algorithm.” A none on the other hand, refers to “choose a random number with the correct number of bytes, used to lock in the security of the encryption algorithm” Initialization vectors are usually appended to the message, meaning your ciphertext is slightly longer than your message. A nonce (in theory) can be derived from context. In practice however, it seems the two are used interchangeably.

For AES 256 GCM, your nonce/IV/initialization vector ought to be 12 bytes long (i.e. 96 bits). This has to do with the size of the blocks in the block cipher. An IV needs to be sized correctly for the length of the block size in the block cipher and for the specific counter method used. For GCM with AES256, 12-bytes is standard.

This is important to get right for security and also because if you try to use the wrong sized nonce, you’ll get obscure errors like:

Error: Unsupported state or unable to authenticate data

Super helpful, that. Especially because you can get this error a bunch of different ways. In truth, ambiguous errors when attempting to decrypt a ciphertext is actually an important part of the security of the algorithm. If you can get the algorithm to output different errors by tweaking the ciphertext, you can use chosen ciphertext attacks to infer details about the plaintext. In some cases, you can even decrypt the ciphertext this way.

Unrelated, if you’re using a different cipher algorithm (i.e. aes256 rather than aes-256-gcm), you might get this error instead:

Error: Invalid IV length

My solution to this problem was to generate 12 bytes of randomness to use as the nonce. I chose to prepend these bytes to the start of my encoded blob.

Lastly, secure encryption requires a mechanism for message integrity. This is done by one of many algorithms which calculate a MAC or Message Authentication Code. I’m not going to attempt to discuss the details of this here, but there is a terminology hurdle with this notion. What some algorithms and papers refer to as a MAC others will refer to as an Auth Tag, which is short for Authentication Tag which is just one of an endless string of unexplained synonyms in the crypto world. Now you know.

Third Issue: Some disassembly required

Here’s where things get really gnarly. If you look at the Golang API for sealing an AES 256 GCM-enciphered plaintext message, you’ll note that it deals only with a nonce, a plaintext, and the mysterious “additionalData”:

What’s important about this is that there is something missing.

If we look at NodeJS’s implementation of the same thing, we’ve got an extra piece of required information — the AuthTag.

So that’s an interesting little mystery. Where is this AuthTag component in the Golang library?

Also, note that the deciphering will not work in NodeJS without the AuthTag. You’ll get the extremely useful, highly unique error message:

Error: Unsupported state or unable to authenticate data


To resolve this mystery, I had to dig into the actual source code of the Golang library for crypto. If we crawl into the source a bit, we can see that they’re helping us out by hiding the auth tag from us entirely:


Here you can see that Golang is using/anticipating it’s own structure for the ciphertext and the AuthTag. This isn’t documented in the Golang docs, of course. However, once you know that the ciphertext produced by Golang’s GCM sealing code, you can easily write your own code to splice out these bits. My code looks like this:

The assumption here is that the bundle is a Base64 encoded string of bits with the first 12 bytes being the nonce, the last 16 bytes being the AuthTag, and all the bytes in the middle being the ciphertext. Golang auto-appends the AuthTag to the end and I wrote some assembly code to prepend this with the nonce. With each of these pieces extracted from the Buffer, performing the decryption is finally possible.

(You might note the weird ordering of parameters to this function. It is strange to have callbacks followed by additional params. This was done because this particular callback is used in a chain of callbacks and I curried some of the arguments. Without doing so, it would have meant a lot more forwarding of arguments along the chain, which results in messy code. This way, the function that decrypts the AES key only needs to call onKeySuccess with a single argument, because the other three were already bound to the onKeySuccess function. Yay JavaScript.)

For the Golang code that does Encryption, check out the example code here. Mine is quite similar.


Writing cross-service, secure communication is hard. Different libraries choose to implement the ciphertext and AuthTag packaging differently. Golang is particularly problematic because it does not tell you it has hidden away some of the details. Because it’s not part of the public API, it means that relying on this particular implementation is unsafe. They might choose to put it the other way around without telling you because you were never supposed to reach inside the blackbox is the first place. Alas. Alack.


Basics of what these different pieces of the puzzle actually do:

A fully-baked NodeJS encryption example:


Least Disruptive Subrange Coding Interview Problem

Here’s the code from the first episode. This is a basic coding interview question that you might be asked in an interview at a tech company. You can see the video associated with this code here:

The problem here can be stated pretty simply. Imagine that you need a function that can take two inputs: 1) a list of integers and 2) another list of integers. The first list could be thousands of integers long. It can contain positive and negative numbers. It’s not sorted in any way. Any integer could be at any position. The second list is similar except that it’s equal in length to the first list or shorter.

What we want to do with these lists is find where in the first list we could substitute the second list, integer for integer, that would create the least amount of change in each integer from the original list. For this problem, we consider change to be measured in number line distance (i.e. absolute value). So, you can’t use some negative distance to offset some positive distance. If you substitute -2 for 2, that’s a change of 4.

An example would be something like this:

original =    [1, 2, 3, 4, 5]
replacement = [3, 5, 3] 

In this example, the “disruption” created by each possible substitution looks like this:

0th position swapping
 0  1  2  3  4
[1, 2, 3, 4, 5]
[3, 5, 3] 
 2, 3, 0 -- total disruption of 5

1st position swapping
 0  1  2  3  4
[1, 2, 3, 4, 5]
   [3, 5, 3] 
    1, 2, 1 -- total disruption of 4

2nd position swapping
 0  1  2  3  4
[1, 2, 3, 4, 5]
      [3, 5, 3] 
       0, 1, 2 -- total disruption of 3

You can see from this, that the best replacement choice here would be the 2nd index, which would create a subrange disruption of just 3, compared to all the other options.

So how might you solve this? Well, here’s a bit of JavaScript that aims to tackle the problem. This algorithm runs in O(n*m) time, where n refers to the length of the first input and m is the length of the second input. Interestingly, the longer the second input is, the shorter the run of the algorithm will be. So, the worst case is something like the length of replacement being half the length of original. In that case, the algorithm will do something along the lines of m*m work. It’s a constant memory problem in that it uses no additional arrays or objects to store state. I guess you could achieve this with a hash table if you just felt like wasting RAM. 😀

While this algorithm (as far as I know) is correct, it does leave many details unserved. For instance, I don’t address the potential to integer underflow by subtracting from the minimum integer value in JavaScript. Likewise, I could integer overflow by adding to distance until it bubbles over the maximum integer value in JavaScript. My test cases are also fairly limited and don’t test for cases like empty list inputs. Also, because JavaScript, I should be checking for null inputs and handling that case reasonably. I’m also not checking for the case where replacement could be longer than original. I’m sure there are like a dozen other defects here.

The point is not to create a bullet-proof library function. Rather, I was aiming to simulate a real coding interview focusing on what you’d have time to accomplish. Let me know what you think. Especially let me know if I got it all wrong!