From C++ via pybind11 to Python project I've met with an issue and I don't know how to proceed.

Intro, "Holder classes are only supported for custom types"

There is an abstract C++ class Page with a pure virtual method Render that returns a heap buffer guarded by a smart pointer:

class Page {
    // ...
    std::shared_ptr< uint8_t > Render( /*...*/ ) const = 0;
    // ...
};

To solve the abstraction - I added a trampoline:

class PageTr : public Page {
    //...
    std::shared_ptr< uint8_t >  Render( /*...*/ ) const override {
        PYBIND11_OVERRIDE_PURE( std::shared_ptr< uint8_t >, Page, Render,
            /*...*/ );
    }
    //...
};

The c++17 compiler (from within pybind11.h) complained with: Holder classes are only supported for custom types.

After a deep study (based on discussion on binding std::shared_ptr<void>) I realized that I must define an opaque custom type like:

using UInt8_Ptr = std::shared_ptr< uint8_t >;
PYBIND11_MAKE_OPAQUE( UInt8_Ptr );

inside appropriate .h file, and, in a .cpp file:

py::class_< UInt8_Ptr >( m, "UInt8_Ptr" );

And all was fine. The binding is:

void BindPage( py::module_& m ) {
    py::class_< Page, PageTr, std::shared_ptr< Page >> cls( m, "Page" );
    cls.def( "Render", &Page::Render, "ToDo", return_value_policy::reference )
    // ...
}

The code compiles but was neither checked nor used yet.

Side Effect "TypeError: a bytes-like object is required, not 'UInt8_Ptr'"

But, after some time, at a totally independent place in project, an issue is raised.

I needed an access from Python to a C++ heap-allocated buffer, containing RGBA uint8_t image samples in an instance of this class:

class Bitmap {
public:
    //...
    BitmapSamplesPtr GetSamples() {
        return this->m_SamplesPtr;
    }

private:
    BitmapSamplesPtr    m_SamplesPtr;
};

where BitmapSamplesPtr is defined as:

using BitmapSamplesPtr = std::shared_ptr< uint8_t >;

It is bound as:

void Bind_Bitmap( py::module_& m ) {
    py::class_< Bitmap, std::shared_ptr< Bitmap >>  cls( m, "Bitmap" );

    cls.def( "GetSamples", &Bitmap::GetSamples, "ToDo",
        py::return_value_policy::reference );
    //...
}

and I managed to get, within a test suite, an instance of it on Python side (as later pageBmp). But when I tried to create a GIL.Image.frombuffer() in this way:

w = pageBmp.GetWidth()
h = pageBmp.GetHeight()
pilImg = Image.frombuffer('RGBA', (w, h), pageBmp.GetSamples(), 'raw')

the Python complained with:

  [...]
  File "[...]\tests\test_Doc.py", line 87, in OnPageDone
    pilImg = Image.frombuffer('RGBA', (w, h), pageBmp.GetSamples(), 'raw')
  File "D:\Python38\lib\site-packages\PIL\Image.py", line 2796, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "D:\Python38\lib\site-packages\PIL\Image.py", line 2742, in frombytes
    im.frombytes(data, decoder_name, args)
  File "D:\Python38\lib\site-packages\PIL\Image.py", line 807, in frombytes
    s = d.decode(data)
TypeError: a bytes-like object is required, not 'UInt8_Ptr'

Thinking...

Then I realized that BitmapSamplesPtr is treated as an alias of UInt8_Ptr (once it is being propagated as opaque through the whole project) - and it is OK.

But how to solve the PIL Image issue? Should not un-opaque it since abstract Page still needs it. I am considering several scenarios/solutions:

1. numpy, ndarray
2. py::buffer_protocol
3. py::bytes

but not sure on the propper way to go. It looks to me that, the most promising way (3.) is to replace UInt8_Ptr with py::bytes. Seems for that, I'll have to wrap the bindings with lambda to return py::bytes.

And here are my questions...

1. How to ensure that std::shared_ptr< uint8_t > exists as a Page::Render() return value? Or in other words, was the opaque type of it a correct decision?
2. If opaque UInt8_Ptr is a must, how to expose the uint8_t* from it on Python side? A getter method?
3. What are the bytes-like objects which will make PIL.Image happy?
4. If py::bytes is a solution, how the lambda function should look like?
5. Since PIL.Image is cappable to use a const uint8_t* with and without making a copy - what would be the optimal but enough secure way to go? (I would prefer a copy and decouple the ownership regardless it is not expected the bitmap-samples-buffer to be modified on Python side.)

Edit

I made an attempt that gave me first result. First, (according to pybind11 PDF doc 12.2.7 Memory view) I used lambda function to expose the Bitmap samples via memoryview as:

cls.def( "GetSamples",
    []( const Bitmap& bmp ) {
        const uint8_t*  bmpBuf  = bmp.GetSamples().get();
        uint32_t        height  = bmp.GetHeight();
        uint32_t        stride  = bmp.GetStride();

        return py::memoryview::from_memory( bmpBuf, height * stride );
    },
    "ToDo" );

But that produced another error:

TypeError: a bytes-like object is required, not `memoryview`

Then I investigated how to get bytes-like from 'memoryview' and it was unexpectedly simple - wrap with bytes() on Python side:

pilImg = Image.frombytes( 'RGBA', ( w, h ), bytes( pageBmp.GetSamples() ), 'raw' )

I switched to Image.frombytes instead of Image.frombuffer since the first makes a copy which ensures that the original buffer could be freely released once we return from the test code. Saved PIL.Image to file looks fine.

Maybe there is a better way, thus I still appreciate comments.