Access Violation When Trying To Read Out Object Created In Python Passed To Std::vector On C++ Side And Then Returned To Python

Working with VS 2019, Python 3.7 64bit on Windows 10 and pybind11 2.4.3 I have run into the following problem: When I create an object with a pybind11 py::class_ on the Python sid

Solution 1:

Note: I'm not by far a PyBind11 expert, I just read the question and tried to figure out what the cause might be. My guess is that the difference is that in the case where it doesn't work, the Python object is created just before add_item call (so is the C++ wrapped one) then just after the call it's being garbage collected (and together with it the C++ wrapped one), yielding Undefined Behavior (an invalid pointer). Conversely, in the case it works, the object is not being garbage collected as it's "saved" in item (its refcount is greater than 0), and therefore the C++ wrapped object is also present. A delete item just after value.add_item(item) should reproduce the faulty behavior.

According to [ReadTheDocs.PyBind11]: Functions - Keep alive:

In general, this policy is required when the C++ object is any kind of container and another object is being added to the container. keep_alive<Nurse, Patient> indicates that the argument with index Patient should be kept alive at least until the argument with index Nurse is freed by the garbage collector.

So, the solution is to make the UseClassHierarchyAsPythonModule.PyXdmItem object persistent until the container is destroyed (note that this might keep objects in memory longer than expected, there might be a cleaner way to achieve this), and that is by specifying in add_item:

...

.def("add_item", &XdmValue::addXdmItem, py::keep_alive<1, 2>());

Solution 2:

PyBind11 can auto-cast using an RTTI trick (that polymorphic_type_hook; it's the same as I do in cppyy: you make a fake base class, cast the given address to the fake base, then read the RTTI to get the actual name, then do a lookup of the Python-proxy and apply a base to derived offset as needed). If the Python code creates the object first, it is found later by address (this to guarantee object identity), so no cast happening.

For that auto-cast to work properly, you really need a virtual destructor to guarantee (per the C++ standard) proper placement of the RTTI in the dll. I don't see in your base class (XdmValue).

(Aside, specific to Windows, I also always export the RTTI root node from the main application to guarantee there is only one. But if so, that should be the Python interpreter doing, or the first module, so I don't think that applies here. Also, I'm assuming of course you enable RTTI when building.)