certutil -scinfo decrypt test fails with CRYPT_E_NO_DECRYPT_CERT

Running certutil -scinfo against a smart card certificate reports the private key as valid, the public key matching test as successful, yet the decryption self-test fails with CRYPT_E_NO_DECRYPT_CERT. This article explains why the two results are not contradictory, and traces the failure to a DER encoding property of the certificate serial number.

Symptom

certutil -scinfo prints:

AES256+RSAES_OAEP(RSA:CNG) test FAILED: Cannot find the certificate and
private key to use for decryption. 0x8009200c (-2146885620 CRYPT_E_NO_DECRYPT_CERT)
The apparent contradiction

The rest of the -scinfo output is healthy. The container is resolved, the public key matching test passes, and the private key operation on the card verifies:

Serial Number: ea946dfa5f0d949f
Issuer: CN=EIDTEST
Subject: CN=User

Performing  public key matching test...
Public key matching test succeeded
  Key Container = 70054947-e6c2-4eeb-985c-5d5800be613d
  Provider = Microsoft Smart Card Key Storage Provider
  KeySpec = 0 -- XCN_AT_NONE
Private key verifies
Microsoft Smart Card Key Storage Provider: KeySpec=0
AES256+RSAES_OAEP(RSA:CNG) test FAILED: Cannot find the certificate and
private key to use for decryption. 0x8009200c (-2146885620 CRYPT_E_NO_DECRYPT_CERT)
certutil -scinfo output: public key matching succeeds, decrypt test fails with CRYPT_E_NO_DECRYPT_CERT
certutil -scinfo output — the key works, only the decrypt self-test fails

The two results describe two different operations. The matching test only proves that the private key on the card corresponds to the public key of the certificate. The decrypt test builds a CMS enveloped message addressed to that certificate and then asks CryptoAPI to locate the matching recipient certificate in order to decrypt it. The failure happens during that lookup, not during any cryptographic operation.

API-level evidence

Capturing the CryptoAPI calls with API Monitor (Cryptographic Primitive filter, certutil.exe process) confirms where the failure sits:

API Monitor trace showing the second CryptDecryptMessage returning FALSE with CRYPT_E_NO_DECRYPT_CERT, with no BCrypt/NCrypt/SCard call in between
The second CryptDecryptMessage returns FALSE — no BCrypt*, NCrypt* or SCard* call is issued
  • The first CryptDecryptMessage call succeeds (it is the size-query pass, pcbDecrypted only).
  • The second CryptDecryptMessage call returns FALSE with -2146885620 (CRYPT_E_NO_DECRYPT_CERT).
  • Between the two calls there is no BCrypt*, NCrypt* or SCard* call at all.

The absence of any NCrypt/BCrypt/SCard activity is the decisive clue: the private key on the card is never invoked. CryptoAPI never reaches the card because it fails earlier, while trying to match the enveloped message's recipient identifier against the certificate in the store.

Root cause — the serial number

The recipient of a CMS enveloped message is identified by IssuerAndSerialNumber: the issuer Distinguished Name plus the certificate serial number. To decrypt, CryptoAPI re-reads that structure and calls CertFindCertificateInStore to locate the certificate whose issuer and serial match. The comparison is done on the raw DER-encoded serial value.

In DER, a serial number is an INTEGER, which is a signed, two's-complement value. When the most significant byte is ≥ 0x80 the high bit is set, so a correct encoder must prepend a 0x00 padding byte to keep the value positive.

Here the serial is:

Serial Number: ea946dfa5f0d949f
                 ^^
                 first byte = 0xEA  (0xEA >= 0x80  ->  high bit set)

When the two sides of the comparison disagree on whether the leading 0x00 padding byte is present — one side encodes 8 bytes (EA 94 …, read as a negative integer), the other 9 bytes (00 EA 94 …) — the raw serial values no longer match byte-for-byte. CertFindCertificateInStore finds no recipient certificate, and CryptDecryptMessage returns CRYPT_E_NO_DECRYPT_CERT. Serial numbers whose first byte is < 0x80 are never affected because no padding byte is involved.

Solution

Re-issue the certificate with a serial number whose first byte is < 0x80. On the issuing CA, either:

  • Regenerate the certificate until the serial's leading byte is below 0x80, or
  • Force the leading byte low in the serial-generation logic (for a random serial, mask the top bit of the first byte).

This is not a smart card, minidriver or KSP problem: the card, its container and its private key are working correctly, as the public key matching test and Private key verifies both confirm. The decrypt self-test in certutil -scinfo is the only operation that resolves the certificate through IssuerAndSerialNumber, which is why it is the only one that fails.

Another problem? Contact us to help us improve this article.