Calling advapi.dll using Pinvoke: CryptDecrypt and CryptEncrypt unexpected behaviour

Question

This is a follow up from this question.

I am calling the WinAPI Cryptography functions from VB.net to ensure compatibility with a legacy product that shares the resulting data.

The code functioned fine for many years however I have recently run into a problem when running on Windows Server 2012. After taking a look at various examples and the answer to my previous question I gave the code an overhaul, including changing what was previously a string variable to a Byte array, which I believe is more correct functionality. How it previously ever worked with string I have no idea, but after fixing the original Server 2012 issue it was failing due to NTE_BAD_LENGTH of the string.

Now, after converting the string to a byte array, The code does not appear to encrypt in the same way it did previously, nor will it decrypt what is generated by CryptEncrypt.

The main bulk of the code follows:

(Calls to CryptCreateHash, CryptHashData, CryptDeriveKey precede this)

Dim _encoding As System.Text.ASCIIEncoding = New System.Text.ASCIIEncoding()

Dim data As String = "Testing123" ''define test String
Debug.WriteLine("Test Data: " & data)

''convert data to Byte()
Dim _bigBuffer As Byte() = New Byte(127) {} ''Buffer to be encrypted
Dim _dataBuffer As Byte() = _encoding.GetBytes(data) ''Get the text to be encrypted into a byte()
Buffer.BlockCopy(_dataBuffer, 0, _bigBuffer, 0, _dataBuffer.Length) ''Copy the data into the big buffer
Dim _dataBufferLength As UInteger = Convert.ToUInt32(_dataBuffer.Length) ''Get the length

Debug.Write("PlainText Data in buffer: ")
For Each _b As Byte In _dataBuffer
    Debug.Write(_b & ", ")
Next

''Encrypt data.
If CryptoApi.CryptEncrypt(_hKey, IntPtr.Zero, True, 0, _bigBuffer, _dataBufferLength, _bigBuffer.Length) = False Then
    Dim _error As Integer = Err.LastDllError
    Throw New Exception("Error during CryptEncrypt. Error Code: " & _error.ToString)
End If

''print out the encrypted string
Dim _encryptedDataBuffer As Byte() = New Byte(_dataBufferLength - 1) {} ''Buffer, size is size of the output encrypted string
Buffer.BlockCopy(_bigBuffer, 0, _encryptedDataBuffer, 0, _dataBufferLength) ''copy from output buffer to new buffer
Dim _encryptedStringFromBuffer As String = _encoding.GetString(_encryptedDataBuffer) ''Decode buffer back to string

Debug.WriteLine("")
Debug.WriteLine("Encrypted: " & _encryptedStringFromBuffer) ''Write encrypted string from buffer

Debug.Write("Encrypted Data in buffer: ")
For Each _b As Byte In _encryptedDataBuffer
    Debug.Write(_b & ", ")
Next

''Copy encrypted strings buffer, ready to decrypted
 Dim _encryptedBufferCopy As Byte() = New Byte(127) {} ''new buffer to hold encrypted data
 Buffer.BlockCopy(_bigBuffer, 0, _encryptedBufferCopy, 0, _bigBuffer.Length) ''copy the output of encryption
 Dim _inputLengthCopy As UInteger = _dataBufferLength ''Copy the length of encrypted data

 ''Decrypt data.
 If CryptoApi.CryptDecrypt(_hKey, IntPtr.Zero, True, 0, _encryptedBufferCopy, _encryptedBufferCopy.Length) = False Then
     Dim _error As Integer = Err.LastDllError
     Throw New Exception("Error during CryptDecrypt. Error Code: " & _error.ToString)
 End If

 Dim _outBuffer As Byte() = New Byte(_dataBufferLength - 1) {}
 Buffer.BlockCopy(_bigBuffer, 0, _outBuffer, 0, _dataBufferLength)
 Dim _stringFromBuffer As String = _encoding.GetString(_outBuffer)

 Debug.WriteLine("")
 Debug.Write("Decrypted Data in buffer: ")
 For Each _b As Byte In _outBuffer
     Debug.Write(_b & ", ")
 Next

 Debug.WriteLine("")
 Debug.WriteLine("Decrypted: " & _stringFromBuffer) 

My method prototypes are defined as:

Friend Declare Function CryptEncrypt Lib "advapi32.dll" _
                            (ByVal hKey As IntPtr, _
                             ByVal hHash As IntPtr, _
                             <MarshalAs(UnmanagedType.Bool)> ByVal final As Boolean, _
                             ByVal dwFlags As Integer, _
                             ByVal data As Byte(), _
                             ByRef pdwDataLen As Integer, _
                             ByVal dwBufLen As Integer) _
                          As <MarshalAs(UnmanagedType.Bool)> Boolean


Friend Declare Function CryptDecrypt Lib "advapi32.dll" _
                              (ByVal hKey As IntPtr, _
                              ByVal hHash As IntPtr, _
                              <MarshalAs(UnmanagedType.Bool)> ByVal final As Boolean, _
                              ByVal dwFlags As Integer, _
                              ByVal data As Byte(), _
                              ByRef pdwDataLen As Integer) _
                            As <MarshalAs(UnmanagedType.Bool)> Boolean

An Example output is as follows:
Test Data: Testing123
PlainText Data in buffer: 84, 101, 115, 116, 105, 110, 103, 49, 50, 51,
Encrypted: CW?? ???_
Encrypted Data in buffer: 12, 67, 87, 133, 158, 9, 139, 250, 255, 95,
Decrypted Data in buffer: 12, 67, 87, 133, 158, 9, 139, 250, 255, 95,
Decrypted: CW?? ???_

As you can see, post decryption the buffer stays exactly the same.

I have no idea why this is happening, my only theory is that it is something to do with which charset I am encoding the Byte() with, although I have tried several and it makes no difference. This may also apply to the password parameter (Byte() passed to CryptHashData).

Any help is greatly appreciated.

EDIT 1: Thanks for the help, I realised the buffer copy issue earlier today, I was attempting to isolate the problem in my test app but ended up over complicating my variables, yes.

Indeed, arrays in VB are instantiated with the highest index so 127 creates a length 128 array, confusing I know.

I believe my problem now lies with the encoding, as you discuss. Previously the data was handed into CryptEncrypt/CryptDecrypt as a .NET String instead of first being converted to a byte(), like I am doing now. However I do not know how the unmanaged code would have dealt with this string, presumably the default .NET encoding or maybe the Windows encoding?

I have tried UTF8 and Unicode however neither will mirror the behaviour of the original code and encypt/decrypt previously stored strings to their original value. This is also the reason I want to continue to use the unmanaged methods, to mirror exactly legacy behaviour (A VB6 app using the unmanaged methods is used to generate the encrypted data before it is decrypted in .NET).

My Code now encrypts and decrypts, but, as I said, the values output do not match the original data, despite using the same cryptographic techniques. Thank you for your continued help.

Answer 1

The reason your decrypted data appears the same as the encrypted data is because you're displaying the wrong buffer. You're decrypting to _encryptedBufferCopy but then copying _bigBuffer to _outBuffer, and displaying that as the decrypted data. But _bigBuffer contains the encrypted data.

You have too many temporary buffers. Also, it's a little disconcerting to see you allocate an array with dataBufferLength - 1. I'm trying to figure out why the -1. Or is that because of VB's insistence that the parameter is the highest index rather than the length?

You should also note that the last parameter to CryptDecrypt is ByRef. When the function returns, that is supposed to contain the number of bytes in the decrypted plaintext. Your call to CryptDecrypt is passing _encryptedBufferCopy.Length, which seems like it should be an error. In any case, the function won't be able to set the decrypted length.

One problem you're going to have is that you're using ASCII encoding. ASCII is a 7-bit encoding, which means that if you have any characters with codes over 127, calling encoding.GetBytes(textString) is going to give you ? characters for those with the high bit set.

Probably your best bet is to use Encoding.UTF8.

Once the data is encrypted, you can't reliably turn it into a string by calling encoding.GetString. That data buffer can contain any byte value, including what would be considered control characters that might not be displayed, might draw funny characters, or who knows what. So where you have:

''print out the encrypted string
Dim _encryptedDataBuffer As Byte() = New Byte(_dataBufferLength - 1) {} ''Buffer, size is size of the output encrypted string
Buffer.BlockCopy(_bigBuffer, 0, _encryptedDataBuffer, 0, _dataBufferLength) ''copy from output buffer to new buffer
Dim _encryptedStringFromBuffer As String = _encoding.GetString(_encryptedDataBuffer) ''Decode buffer back to string

That's almost certain to create a string that can't be displayed. The ? characters you're seeing here represent bytes with character values above 127, which your ASCII encoding won't interpret correctly.

All things considered, I'm wondering why you don't just use the managed APIs in the System.Security.Cryptography namespace for this. I think you'll find that it's much easier, and you don't have to worry about handles and 32/64 bit issues, etc.