When I execute my testsuite of which below test is part of, sometimes it is PASSED, sometimes it FAILED. After How can the behavior be so erratic? I don't know the cryptopp library well enough to debug it correctly. What makes it even more strange is that I have the same functions for a hex-encoded key, copy/pasted, only the function names changed accordingly, and it works.

The error in the failure case is

unknown file: Failure
C++ exception with description "BER decode error" thrown in the test body.

Update - Solution found, but not understood

// Switched to these methods, and added missing `MessageEnd`
CryptoPP::Base64Encoder privKeySink(new CryptoPP::StringSink(keyPair.privateKey),insertLineBreaks);
privateKey.DEREncode(privKeySink);
privKeySink.MessageEnd(); // This was missing!
CryptoPP::Base64Encoder pubKeySink(new CryptoPP::StringSink(keyPair.publicKey),insertLineBreaks);
publicKey.DEREncode(pubKeySink);
pubKeySink.MessageEnd(); // This was missing!

// Why does this code not work?
// What should be added to make it work?
// For example the exact same code, only with `HexEncoder`
// instead of `Base64Encoder` is used for a hex key generation,
// and it works without issues. I don't understand the difference.
//publicKey.Save( CryptoPP::Base64Encoder(
//                new CryptoPP::StringSink(keyPair.publicKey), insertLineBreaks).Ref());
//privateKey.Save(CryptoPP::Base64Encoder(
//                new CryptoPP::StringSink(keyPair.privateKey), insertLineBreaks).Ref());

The next code snippet shows that the pipelining style works for HexEncode, and I wonder why this does not work for Base64Encoder. Do I maybe need to add another pipeline step that does the "MessageEnd" in case of Base64Encoder?

publicKey.Save( CryptoPP::HexEncoder(
                new CryptoPP::StringSink(keyPair.publicKey)).Ref());
privateKey.Save(CryptoPP::HexEncoder(
                new CryptoPP::StringSink(keyPair.privateKey)).Ref());

Update end

Original question

Testcase:

TEST(signature, rsa_new_pem)
{
    bool ok;
    KeyPairBase64 keys;
    keys = RsaGenerateBase64KeyPair(3072);
    std::string message("secret message");
// The code sometimes fails at next call.
    auto signature = RsaPemSignStringCreateHexSignature(keys.privateKey, message);
    bool verified = RsaPemVerifyStringWithHexSignature(keys.publicKey, message, signature);
    EXPECT_TRUE(verified);
}

This is how I generate the key pair. I even tried multiple methods of which I think they are interchangeable.

struct KeyPairBase64 {
    std::string publicKey;
    std::string privateKey;
};

inline KeyPairBase64 RsaGenerateBase64KeyPair(unsigned int aKeySize)
{
    KeyPairBase64 keyPair;
    CryptoPP::AutoSeededRandomPool rng;
    
    // generate keys
    CryptoPP::RSA::PrivateKey privateKey;
    privateKey.Initialize(rng,aKeySize);
    //privateKey.GenerateRandomWithKeySize(rng, aKeySize);
    CryptoPP::RSA::PublicKey publicKey(privateKey);
    
    bool insertLineBreaks = false;
    //  CryptoPP::Base64Encoder privKeySink(new CryptoPP::StringSink(keyPair.privateKey),insertLineBreaks);
    //  privateKey.DEREncode(privKeySink);
    //  CryptoPP::Base64Encoder pubKeySink(new CryptoPP::StringSink(keyPair.publicKey),insertLineBreaks);
    //  publicKey.DEREncode(pubKeySink);
    publicKey.Save( CryptoPP::Base64Encoder(
                    new CryptoPP::StringSink(keyPair.publicKey), insertLineBreaks).Ref());
    privateKey.Save(CryptoPP::Base64Encoder(
                    new CryptoPP::StringSink(keyPair.privateKey), insertLineBreaks).Ref());
    
    return keyPair;
}

Here is my custom convenience function that creates a signature

inline std::string RsaPemSignStringCreateHexSignature(
        const std::string &aPrivateKeyStrPem,
        const std::string &aMessage)
{
    // `cleanPemInputFile` is a simple custom function that strips
    // of pem file header and footer.
    std::string encodedPriv = cleanPemInputFile(aPrivateKeyStrPem);
    CryptoPP::RSA::PrivateKey privateKey;
    CryptoPP::StringSource ss(encodedPriv, true, new CryptoPP::Base64Decoder);
    privateKey.BERDecode(ss);

    // sign message
    std::string signature;
    Signer signer(privateKey);
    CryptoPP::AutoSeededRandomPool rng;

    CryptoPP::StringSource ss2(aMessage, true,
                            new CryptoPP::SignerFilter(rng, signer,
                              new CryptoPP::HexEncoder(
                                new CryptoPP::StringSink(signature))));

    return signature;
}