Decrypting message with RSA-OAEP and Android Keystore: IllegalBlockSizeException

Question

I have the following code inside a try block that should generate a RSA public/private keypair use the public key to encrypt a message and decrypt again with the private key:

KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance(
         KeyProperties.KEY_ALGORITHM_RSA, "AndroidKeyStore");
keyPairGenerator.initialize(new KeyGenParameterSpec.Builder(
                 "key1",
                 KeyProperties.PURPOSE_DECRYPT | KeyProperties.PURPOSE_ENCRYPT)
                 .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
                 .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_OAEP)
                 .build());
KeyPair keyPair = keyPairGenerator.generateKeyPair();
byte[] src = "hello world".getBytes();

Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher.init(Cipher.ENCRYPT_MODE, keyPair.getPublic());
byte[] cipherData = cipher.doFinal(src);

Cipher cipher2 = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
cipher2.init(Cipher.DECRYPT_MODE, keyPair.getPrivate());
byte[] msg = cipher2.doFinal(cipherData);

Taken mostly from here and here.

The final line throws an exception of type javax.crypto.IllegalBlockSizeException with no message/further details. The three lines in logcat before the exception are

 E keymaster1_device: Finish send cmd failed
 E keymaster1_device: ret: 0
 E keymaster1_device: resp->status: -1000

in case that matters at all.

Does anyone have an idea what could be going wrong?

Using minSdkVersion 23

Edit: I just realised, if I use PKCS#1 v1.5 padding it works. That helps me for now, but I'd still like to try get it work with OAEP.

https://gist.github.com/Diederikjh/36ae22d5fde9d8f671a70b5d8cada90e — kelalaka, Feb 29 '20 at 22:07

Lena Bru · Accepted Answer · 2020-03-01T17:07:45.967

3

You need to put into the cipher the algorithm parameter spec when you encrypt

if (algorithmParameterSpec != null) {
            encrypter.init(Cipher.ENCRYPT_MODE, getKey(), algorithmParameterSpec)
        }

algorithmParameterSpec is

OAEPParameterSpec("SHA-256",
                    "MGF1",
                    MGF1ParameterSpec.SHA256,
                    PSource.PSpecified.DEFAULT)

edited Mar 01 '20 at 17:07

answered Mar 01 '20 at 09:34

Lena Bru

12,299
9
53
109

thanks for the update, that helped me: providing the specs as you suggested gives me a better error, `InvalidAlgorithmParameterException: Unsupported MGF1 digest: SHA-256. Only SHA-1 supported`, when I change `MGF1ParameterSpec.SHA256` to `MGF1ParameterSpec.SHA1` my code works. – matec Mar 01 '20 at 19:04
try setting the cipher transformation to ```KeyProperties.KEY_ALGORITHM_RSA + "/" + KeyProperties.BLOCK_MODE_ECB + "/" + KeyProperties.ENCRYPTION_PADDING_RSA_OAEP``` i.e : ```Cipher.getInstance(KeyProperties.KEY_ALGORITHM_RSA + "/" + KeyProperties.BLOCK_MODE_ECB + "/" + KeyProperties.ENCRYPTION_PADDING_RSA_OAEP)``` – Lena Bru Mar 01 '20 at 19:13
instead of going to Sha1, because sha1 is breakable – Lena Bru Mar 01 '20 at 19:14
Using `MGF1ParameterSpec.SHA256` on Android will not work, because the OS doesn't support it as @matec saw in the error. Changing the transformation won't make a difference. – mahler Sep 03 '20 at 15:57
well i have a working app with this line and it works just fine... what specifically doesnt work for you ? – Lena Bru Sep 03 '20 at 20:06
Decrypting (`Cipher.DECRYPT_MODE`) or unwrapping (`Cipher.UNWRAP_MODE`) an input using those parameters will result in the error message that was mentioned. Encryption (`Cipher.ENCRYPT_MODE`) works fine. – mahler Sep 04 '20 at 11:11

Decrypting message with RSA-OAEP and Android Keystore: IllegalBlockSizeException

1 Answers1