GZip Magic Number Missing After AES Encryption

Question

I am compressing a file using GZip and then encrypting using AES.

When I step through the code, I see that the magic number (1f 8b) is present in the intermediate compressed unencrypted data and then the file is encrypted. When I go to decrypt the file, the intermediate decrypted compressed data does not contain the magic number and the GZipStream fails when decompressing it.

Here is the main code:

    private static void CompressThenEncrypt(string inputFileName, string outputFileName, ICryptoTransform encryptor)
    {
        using (var inputFileStream = new FileStream(inputFileName, FileMode.Open, FileAccess.Read))
        {
            using (var outputFileStream = new FileStream(outputFileName, FileMode.Create, FileAccess.Write))
            using (var cryptoStream = new CryptoStream(outputFileStream, encryptor, CryptoStreamMode.Write))
            {
                using (var gZipStream = new GZipStream(cryptoStream, CompressionMode.Compress))
                {
                    inputFileStream.CopyTo(gZipStream);
                }
            }
        }
    }

    private static void DecryptThenDecompress(string inputFileName, string outputFileName, ICryptoTransform decryptor)
    {
        using (var inputFileStream = new FileStream(inputFileName, FileMode.Open, FileAccess.Read))
        {
            using (var cryptoStream = new CryptoStream(inputFileStream, decryptor, CryptoStreamMode.Read))
            using (var gZipStream = new GZipStream(cryptoStream, CompressionMode.Decompress))
            using (var outputFileStream = new FileStream(outputFileName, FileMode.Create, FileAccess.Write))
            {
                gZipStream.CopyTo(outputFileStream);
            }
        }
    }

Source file is 19000 bytes. When compressed it becomes 603 bytes (with magic number), then encrypted it becomes 608 bytes (due to padding). When decrypted it becomes 603 bytes (no magic number) and I simply can not get beyond this point.

Here is the calling code:

        using (var aes = new AesCryptoServiceProvider())
        {
            ICryptoTransform encryptor = aes.CreateEncryptor();
            ICryptoTransform decryptor = aes.CreateDecryptor();

            CompressThenEncrypt(OriginalFileName, CompressThenEncryptFileName, encryptor);

            DecryptThenDecompress(CompressThenEncryptFileName, DecryptThenDecompressFileName, decryptor);
        }

Edit: More Info

Compressed Data (603 bytes): 1F-8B-08-00-00-00-00-00-04-00-ED-CA-B1-0D-00-20-08-00-C1-DE-C4-21-51-64-FF-11-DC-80-84-FE-9A-6F-FE-E2-DC-7C-15-6D-F7-EA-3F-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-D4-44-7D-3F-A6-A4-8A-30-E6-02-00

Encrypted Data (608 bytes): C3-02-64-A9-15-15-52-A9-9F-2A-62-EF-89-B2-9A-E0-72-9A-BA-E2-A2-C8-35-BA-3C-CF-D7-07-8B-DC-D4-63-65-AF-A8-62-22-E5-CC-C6-7D-6F-9B-64-57-9F-C0-94-75-E7-6C-A9-DB-B3-57-29-01-95-F5-B9-E5-3D-FD-AB-EC-E3-AC-AF-EB-A6-C0-81-B3-47-4A-EB-F6-DD-03-DF-92-0A-82-D9-DD-4E-46-DF-55-2E-EB-34-AC-98-1A-7E-A9-25-94-2D-E8-32-4B-F9-2A-61-64-CB-09-9C-D5-8C-A2-0A-C1-22-90-98-93-26-A6-9F-69-F8-EE-6E-95-96-56-28-71-3B-94-1E-5F-50-DB-15-DF-C6-46-3F-04-57-5E-0B-47-44-BB-13-9F-14-08-FB-87-E6-97-65-1B-CA-50-52-7F-10-5D-AF-CC-2F-5E-D0-39-A6-C4-70-3B-90-5F-63-EA-F0-59-46-9E-99-2D-31-34-66-5D-72-E5-85-D5-00-1E-E7-B2-1C-3B-E0-E8-F8-35-BF-90-24-00-DC-47-09-66-92-2F-43-92-48-CB-42-4C-3C-86-CC-67-33-62-A1-1E-76-D1-D6-AD-5F-50-DC-D9-C7-31-F2-33-FE-77-CB-4C-EE-2A-AE-54-63-46-48-B4-FA-6F-0A-E0-1B-F2-F6-C2-D0-E0-24-A1-79-B8-29-FB-04-F3-D5-4E-CC-64-E8-FA-67-55-7B-E6-CD-FD-D2-13-3D-F0-C4-10-A9-5C-BB-34-66-54-A4-6E-B7-AC-54-7A-21-5F-C3-01-7B-97-AD-71-C5-2A-E9-39-B1-27-C7-F8-AC-BF-76-EA-D6-C8-05-22-54-4B-71-73-F2-FD-8E-6E-D7-D2-F2-F7-83-B3-9F-75-3C-CA-BA-BD-F3-C2-E6-16-37-9E-C1-88-C4-69-F5-95-E8-A5-81-C9-FC-22-73-1D-09-AB-A8-6D-A3-BA-CB-0F-27-4E-C7-8C-7A-6C-9D-9B-9D-1F-45-F2-7C-B7-7B-F6-DF-24-50-71-A7-BA-F7-F2-8C-AC-19-A3-86-77-4A-EC-5A-06-12-1F-00-AD-5D-EE-E0-61-D0-80-B7-2C-72-2E-77-6A-32-24-0C-64-78-63-37-A9-08-A2-90-9D-21-5E-E4-5E-E6-1D-66-7D-F5-E1-FC-3C-F1-DE-76-68-D7-1D-46-70-A5-32-31-2B-F5-02-6A-E4-95-CD-8E-B8-76-8D-6E-0B-98-E9-4A-DD-8D-C3-6F-D8-0C-BE-C7-32-3F-99-26-94-26-41-80-E4-E6-E7-D3-26-E9-2F-C0-5D-7B-98-24-BD-6E-9B-E8-9D-F6-DF-51-90-FE-EE-86-DE-9E-31-9F-1C-BA-1A-C4-5B-DD-5A-84-43-02-B9-99-01-6A-95-7C-FF-86-28-C2-4C-EF-4D-D6-36-BD-08-0F-30-25-E7-FF-D9-BC-DA-A7-87-65-1A-1E-83-55-D2-60-38-EC-51-97-FA-FD-11-70-83-70-66-39-78-47-93-7C-B7-FB-48-96-2F-C6-1E-6E-7D-29-38-F6-AB-06-45-C5-F7-50-D6-C7-44-5C-AB-96-A0-60-7D-0E-63-4E-B2-EA

Decrypted Data (603 Bytes): 66-90-D0-0F-8B-67-60-9B-AC-39-FC-45-04-3F-9D-C5-08-00-C1-DE-C4-21-51-64-FF-11-DC-80-84-FE-9A-6F-FE-E2-DC-7C-15-6D-F7-EA-3F-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-14-45-51-D4-44-7D-3F-A6-A4-8A-30-E6-02-00

As you can see, all data is the same in the compressed format except for the first 16 bytes:

Before Encryption: 1F-8B-08-00-00-00-00-00-04-00-ED-CA-B1-0D-00-20

After Decryption: 66-90-D0-0F-8B-67-60-9B-AC-39-FC-45-04-3F-9D-C5

The magic number is gone and I do not know why.

The file that I am encrypting is a text file with the line abcdefabcdefabcdefabcdefabcdefabcdef repeated 5000 times.

So if it still has 603 bytes, and the magic number is missing, what are the new two bytes? And are the other 601 bytes correct? — Thilo, Jan 03 '16 at 07:50
Does this happen with a particular input file? I tested your code on some file and it worked fine. — Darin Dimitrov, Jan 03 '16 at 08:00
@Thilo: Everything but the first 16 bytes are the same. See the updated answer for the complete info on what those bytes are pre encryption and post decryption. — Derek W, Jan 03 '16 at 19:36
@DarinDimitrov: The file that I am encrypting is just a text file with the line `abcdefabcdefabcdefabcdefabcdefabcdef` repeated 5000 times. I have tweaked the contents in a few attempts, but get the same missing magic number error every time and the `GZipStream` class has every right to complain because the magic number is simply not there. — Derek W, Jan 03 '16 at 19:39
Is the calling code you show _exactly_ the usage you use? Don't you have multiple `using (new Aes...())` or something? — CodeCaster, Jan 03 '16 at 19:43
@CodeCaster: Thanks for your comment which ultimately lead me down the path to find the issue! If I can save just one other developer from this issue, then that will make my day! :) — Derek W, Jan 03 '16 at 20:10

score 2 · Accepted Answer · edited May 23 '17 at 11:50

Having just the first 16 bytes (default block size of AES) being incorrect is the signature of another problem which is the fact that the AesCryptoServiceProvider decryptor object cannot be reused as it stores state information from the last decryption which causes the such bizarre results in subsequent decryptions.

The full original calling code included the fatal reuse of the decryptor object.

Full original calling code:

using (var aes = new AesCryptoServiceProvider())
{
      ICryptoTransform encryptor = aes.CreateEncryptor();
      ICryptoTransform decryptor = aes.CreateDecryptor(); // <-- Decryptor fails on second usage

      // Compress/ Encrypt
      CompressThenEncrypt(OriginalFileName, CompressThenEncryptFileName, encryptor);
      EncryptThenCompress(OriginalFileName, EncryptThenCompressFileName, encryptor);

      // Decrypt/ Decompress
      DecompressThenDecrypt(EncryptThenCompressFileName, DecompressThenDecryptFileName, decryptor);
      DecryptThenDecompress(CompressThenEncryptFileName, DecryptThenDecompressFileName, decryptor);
}

This was a personal exercise I developed to showcase the importance of compressing then encrypting data. I had two scenarios "Compress Then Encrypt" and "Encrypt Then Compress" and I also wrote out the code to perform the reverse operations.

The issue occurred when I reused the ICryptoTransform decryptor object. Despite the fact that the CanReuseTransform property is true - it is false advertisement. According to this answer it appears that there is a bug where the input buffer is not cleared correctly after decryption. The linked answer describes a few workarounds and simply creating a separate decryptor object works.

Revised working calling code:

using (var aes = new AesCryptoServiceProvider())
{
      ICryptoTransform encryptor = aes.CreateEncryptor();
      ICryptoTransform decryptor = aes.CreateDecryptor();
      ICryptoTransform decryptor2 = aes.CreateDecryptor();

      // Compress/ Encrypt
      CompressThenEncrypt(OriginalFileName, CompressThenEncryptFileName, encryptor);
      EncryptThenCompress(OriginalFileName, EncryptThenCompressFileName, encryptor);

      // Decrypt/ Decompress
      DecompressThenDecrypt(EncryptThenCompressFileName, DecompressThenDecryptFileName, decryptor);
      DecryptThenDecompress(CompressThenEncryptFileName, DecryptThenDecompressFileName, decryptor2);
}

GZip Magic Number Missing After AES Encryption

1 Answers1