Encryption with AES-256-GCM using (LibreSSL) libcrypto

Question

Given an appropriate key and iv, this C program should encrypt stdin, outputting to stdout.

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
EVP_EncryptInit(ctx, EVP_aes_256_gcm(), key, iv);

const size_t block_size = 128;
unsigned char inbuf[block_size];
unsigned char outbuf[block_size + EVP_MAX_BLOCK_LENGTH];
int inlen, outlen;
for (;;)
{
        inlen = fread(inbuf, 1, block_size, stdin);
        if (inlen <= 0)
                break;

        EVP_EncryptUpdate(ctx, outbuf, &outlen, inbuf, inlen)
        fwrite(outbuf, 1, outlen, stdout);
}
EVP_EncryptFinal_ex(ctx, outbuf, &outlen)
fwrite(outbuf, 1, outlen, stdout);

(Error checking removed for brevity.)

I am verifying the output of this code by running

openssl aes-256-gcm -in ciphertext.txt -K <key> -iv <iv> -d

This successfully and reliably decrypts the ciphertext, but writes bad decrypt afterwards, for instance

$ openssl aes-256-gcm ...
Hello World.
bad decrypt

What could be going wrong to cause it to say this?

Answer 1

I was completely failing to get the GCM authentication tag after encryption and then providing it during decryption.

The "bad decrypt" message was misleading — the decryption was going fine, but the tag providing authentication was not provided.

The tag can be gotten at after the call to EVP_EncryptFinal_ex with

unsigned char *tag = malloc(TAGSIZE);
EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, TAGSIZE, tag);

TAGSIZE is the size in bytes of the tag, and may be a number of different values. This is discussed on Wikipedia among other places.

Answer 2

The issue here is, that you do not implement proper padding as PKCS#7 requires. openssl expects correct padding to be present in the decoded message. Rules are:

• There has to be always at least 1 byte of padding
• It has to be padded to block size of the cipher

Now with AES-256 you have a block size of 128 bit (16 byte). This means, you have to add 1-16 padding bytes. The padding bytes to add are always the size of the padding, so if you have to add 9 bytes of padding, you would add 9 0x09 bytes.

If you properly add this padding to your original plaintext, then openssl should stop complaining on decrypt.