View source for C++ reflection (P2996) and moc

This is a WORK IN PROGRESS page to understand the implications of "Reflection for C++" for the future of Qt and moc.  


'''<big><u>NOTE: THIS HAS BEEN WRITTEN AGAINST A VERY EARLY REVISION OF P2996. MANY THINGS HAVE CHANGED SINCE, AND EVERYTHING ON THIS PAGE NEED TO BE RE-EVALUATED BASED ON WHAT EFFECTIVELY LANDED IN C++26!</u></big>'''

== TL;DR== 

*Reflection in C++26 might be insufficient for replacing moc.
*We may require token injection, function definition, string-based lookup; at least the first is C++29 material, don't know about the others.
* Extra fancy features like outputting extra JSON files (or embedding JSON in plugins) may be out of scope for this work.

== Why?==
moc extracts interesting metadata from QObject subclasses / gadgets / namespaces. This metadata is then used at runtime to implement many different QMetaObject facilities.

C++26 will (likely) ship with compile-time reflection. This means that the work that moc does today (as an external tool, with its custom lexer and parser etc.) could be done by the compiler itself. In the future, this may unlock lots of interesting possibilities (e.g. templated QObjects) and reduce technical debt (no need to maintain our own C++ parser for moc).

However, we're not sure how to get there just yet. Some questions that this work aims to answer:

*Can we replace moc with a pure C++ solution?
**Is there something missing from standard C++ that we need for moc?
*How many source code (API) breaks are to be expected?
**If there's breakage, can porting tools automate the transition?
*Are there going to be issues with buildsystems?

==Reflection for Standard C++: references==
Papers that are interesting for Qt:

*P2996 "Reflection for C++26" https://wg21.link/P2996
*P3394 "Annotations for Reflection" https://wg21.link/P3394
*P3096 "Function Parameter Reflection in Reflection for C++26" https://wg21.link/P3096
*P3491 "define_static_{string,object,array}" https://wg21.link/P3491
* P3294 "Code Injection with Token Sequences" https://wg21.link/P3294
*P1306 "Expansion Statements" https://wg21.link/P1306

==What needs to be extracted?==
moc extracts:

*the class name
*the parent class name (following QObject's inheritance chain)
*properties (Q_PROPERTY)
*invokables (methods marked with Q_INVOKABLE, Q_SIGNAL, Q_SLOT; may include constructors)
**for each invokable, also the parameter lists, i.e. types+names of the arguments; and whether they're defaulted (Qt considers these as separate overloads)
**the revision number  (Q_REVISION)
* Q_CLASSINFO metadata
*Q_ENUM and Q_FLAGS
*Q_INTERFACES

===Can C++26 reflection extract the same information?===

====Class name ====
Should there be no problems, `display_string_of` and friends give us this info.

====Parent class ====
Again, no problems. Can use `bases_of` and extract the first (or even identify the one that gives us access to QObject). This can be spliced to access that class' metaobject and chain the metaobject to the parent's.

=====Invokables=====
Marking them with `signals:` or `public slots:` is not going to work.

Instead, we have to use per-invokable markers. We already have Q_SIGNAL, Q_SLOT, Q_INVOKABLE, which are a bit verbose, get the job done. One can even concoct a porting tool...?

We'll need to have those macros expand to some annotation (see P3394, for instance `[[=Qt::signal{}]]` or similar); when reflecting we'll then be able to query for these annotations and extract the relevant function(s).

We also have to extract the (meta)types and the parameter names (needed e.g. for QML). This should be perfectly possible using P3096. The "limitation" of having matching parameter names across all declarations isn't so stringent (in theory at least).

=====Properties=====
At the moment declaring a property looks like this: <syntaxhighlight lang="cpp">
Q_PROPERTY(Type propName READ getProp WRITE setProp NOTIFY propChanged RESET resetProp [...])
</syntaxhighlight>This is basically a "bag of data" that moc tokenizes (the sequence is pretty much "free form"). The individual "attributes" of a property are extracted and set in the metaobject. Q_PROPERTY itself expands to nothing.

There's a bunch of related problems here: in a reflection world we need Q_PROPERTY to expand to ''something'' that describes the property. It could be as simple as a tagged string_view:<syntaxhighlight lang="cpp">
[[=Qt::property{}]] 
static constexpr std::string_view qt_property_COUNTER =
    "Type propName READ getProp WRITE setProp NOTIFY propChanged RESET resetProp [...]";
</syntaxhighlight>We can certainly tokenize the string (i.e. Q_PROPERTY's argument) at compile time, all it's needed is a consteval function (after all that's what std::format does for the format string).

'''However P2996's reflection (AFAICT) does not give us string-based reflection.''' In other words, given "getProp" as a string, we would need to "look it up" in the class context and get  something like a PMF out of it. That does not seem possible at the moment. :-(

We could engineer an alternative, more structured way to declare properties, for instance something like:<syntaxhighlight lang="cpp">
constexpr static QProperty<Class, int> prop = {
  .name = "prop",
  .getter = &Class::getProp,
  .setter = &Class::setProp,
  .notify = &Class::propChanged,
  .stored = true,
};
</syntaxhighlight>It's very heavy on the eyes compared to the current solution, but it might ''just'' work. <blockquote>There's a lot of subtle details here that make the exercise scary, for instance what is exactly "getter"? A pointer a to non-static member function which returns `int` and it's `const`? What if it has extra defaulted arguments, or it's not const? And what about `notify`, it could have arbitrary arguments and return values... There's lots of engineering here.  Of course, if we get string-based lookup in C++29 the exercise becomes quite moot. :-(</blockquote>

===== Q_CLASSINFO =====
These are simple (?) key/value pairs. 

=====Q_ENUM / Q_FLAG=====
moc extracts the enumerators (string/value) pairs, this is perfectly possible with P2996.

=====Q_INTERFACES=====
WIP; not sure at the moment. Is it just used for qt_metacast?

==Triggers for moc / buildsystem integration==
moc needs to be run on source files that define classes or namespaces and in which a "trigger keyword" is found: Q_OBJECT, Q_GADGET, Q_NAMESPACE, etc.

Usually, the buildsystem identifies these files, and prepares build rules so that moc is run on them.

The output of moc is some additional .cpp code, this code can either get compiled standalone (and linked into the final target) via additional build rules, or directly included by a .cpp file (e.g. #include "moc_foo.cpp") in which case no additional rules are needed.

===How to "trigger reflection"?===
The main idea on the table is that we're going to need a macro of some sorts to be placed in a .cpp file. That macro will expand to everything necessary -- defining all the Q_OBJECT-declared stuff by calling consteval functions that apply reflection, extract the necessary data, build tables and the like.

Something like this:
 Q_GENERATE_METAOBJECT(Class)
Having to do add this ''manually'' to every marked class is certainly annoying.
==What needs to be generated?==
In a nutshell: some static data, the implementation of various things declared by the Q_OBJECT macro, the implementation of the class' metaObject() function, the implementation of signals, ...

For instance, for QWidget (in Qt 6.10):<syntaxhighlight lang="cpp" line="1">
namespace {
struct qt_meta_tag_ZN7QWidgetE_t {};
} // unnamed namespace

template <> constexpr inline auto QWidget::qt_create_metaobjectdata<qt_meta_tag_ZN7QWidgetE_t>()
{ ... }

Q_CONSTINIT const QMetaObject QWidget::staticMetaObject = { ... }

void QWidget::qt_static_metacall(QObject *_o, QMetaObject::Call _c, int _id, void **_a) { ... }

const QMetaObject *QWidget::metaObject() const { ... }

void *QWidget::qt_metacast(const char *_clname) { ... }

int QWidget::qt_metacall(QMetaObject::Call _c, int _id, void **_a) { ... }

// SIGNAL 0
void QWidget::windowTitleChanged(const QString & _t1)
{
    QMetaObject::activate<void>(this, &staticMetaObject, 0, nullptr, _t1);
}

// SIGNAL 1
void QWidget::windowIconChanged(const QIcon & _t1)
{
    QMetaObject::activate<void>(this, &staticMetaObject, 1, nullptr, _t1);
}

// SIGNAL 2
void QWidget::windowIconTextChanged(const QString & _t1)
{
    QMetaObject::activate<void>(this, &staticMetaObject, 2, nullptr, _t1);
}

// SIGNAL 3
void QWidget::customContextMenuRequested(const QPoint & _t1)
{
    QMetaObject::activate<void>(this, &staticMetaObject, 3, nullptr, _t1);
}
</syntaxhighlight>

===Unique tag ===
I'm not sure what this is for.

===qt_create_metaobjectdata() ===
This is the basically the storage for the extracted data:<syntaxhighlight lang="cpp" line="1">
    namespace QMC = QtMocConstants;
    QtMocHelpers::StringRefStorage qt_stringData {
        "QWidget",
        "windowTitleChanged",
...
    };

    QtMocHelpers::UintData qt_methods {
        // Signal 'windowTitleChanged'
        QtMocHelpers::SignalData<void(const QString &)>(1, 2, QMC::AccessPublic, QMetaType::Void, {{
            { QMetaType::QString, 3 },
        }}),
        // Signal 'windowIconChanged'
        QtMocHelpers::SignalData<void(const QIcon &)>(4, 2, QMC::AccessPublic, QMetaType::Void, {{
            { QMetaType::QIcon, 5 },
        }}),...
    };
    QtMocHelpers::UintData qt_enums {
    };
    return QtMocHelpers::metaObjectData<QWidget, qt_meta_tag_ZN7QWidgetE_t>(QMC::MetaObjectFlag{}, qt_stringData,
            qt_methods, qt_properties, qt_enums);
</syntaxhighlight>

===staticMetaObject ===
This is basically initializing an object with a bunch of fields defined around. qt_staticMetaObjectStaticContent etc. are just parts of the data returned by qt_create_metaobjectdata.

Maybe it could be generated as part of the reflection-triggering macro? Not sure how to deal with the unique metatag. Otherwise could be generated via token injections, I wager.<syntaxhighlight lang="cpp">
Q_CONSTINIT const QMetaObject QWidget::staticMetaObject = { {
    QMetaObject::SuperData::link<QObject::staticMetaObject>(),
    qt_staticMetaObjectStaticContent<qt_meta_tag_ZN7QWidgetE_t>.stringdata,
    qt_staticMetaObjectStaticContent<qt_meta_tag_ZN7QWidgetE_t>.data,
    qt_static_metacall,
    nullptr,
    qt_staticMetaObjectRelocatingContent<qt_meta_tag_ZN7QWidgetE_t>.metaTypes,
    nullptr
} };

</syntaxhighlight>

===qt_static_metacall()===
This implements the various metaobject-related operations: invoke a method, read a property's value, write a property's value, etc.

This may be doable using P2996's reflection. We can provide a skeleton implementation that gets "filled in" via reflection / expansion statements.<syntaxhighlight lang="cpp">
void QWidget::qt_static_metacall(QObject *_o, QMetaObject::Call _c, int _id, void **_a)
{
    auto *_t = static_cast<QWidget *>(_o);
    if (_c == QMetaObject::InvokeMetaMethod) {
        switch (_id) {
        case 0: _t->windowTitleChanged((*reinterpret_cast< std::add_pointer_t<QString>>(_a[1]))); break;
        case 1: _t->windowIconChanged((*reinterpret_cast< std::add_pointer_t<QIcon>>(_a[1]))); break;
...
        default: ;
        }
    }
    if (_c == QMetaObject::IndexOfMethod) {
        if (QtMocHelpers::indexOfMethod<void (QWidget::*)(const QString & )>(_a, &QWidget::windowTitleChanged, 0))
            return;
        ...
    }
    if (_c == QMetaObject::ReadProperty) {
        void *_v = _a[0];
        switch (_id) {
        case 0: *reinterpret_cast<bool*>(_v) = _t->isModal(); break;
        case 1: *reinterpret_cast<Qt::WindowModality*>(_v) = _t->windowModality(); break;
        ...
        }
    }
    if (_c == QMetaObject::WriteProperty) {
        void *_v = _a[0];
        switch (_id) {
        case 1: _t->setWindowModality(*reinterpret_cast<Qt::WindowModality*>(_v)); break;
        case 2: _t->setEnabled(*reinterpret_cast<bool*>(_v)); break;
        ...
        }
    }
   ...
}

</syntaxhighlight>

===metaObject()===
<syntaxhighlight lang="cpp">
const QMetaObject *QWidget::metaObject() const
{
    return QObject::d_ptr->metaObject ? QObject::d_ptr->dynamicMetaObject() : &staticMetaObject;
}
</syntaxhighlight>This can be hardcoded; it doesn't need reflection at all. This is moc's code that emits the above:<syntaxhighlight lang="cpp">
    fprintf(out, "\nconst QMetaObject *%s::metaObject() const\n{\n"
                 "    return QObject::d_ptr->metaObject ? QObject::d_ptr->dynamicMetaObject() : &staticMetaObject;\n"
                 "}\n",
            cdef->qualified.constData());

</syntaxhighlight>

===qt_metacast()===
<syntaxhighlight lang="cpp">
void *QWidget::qt_metacast(const char *_clname)
{
    if (!_clname) return nullptr;
    if (!strcmp(_clname, qt_staticMetaObjectStaticContent<qt_meta_tag_ZN7QWidgetE_t>.strings))
        return static_cast<void*>(this);
    if (!strcmp(_clname, "QPaintDevice"))
        return static_cast< QPaintDevice*>(this);
    return QObject::qt_metacast(_clname);
}

</syntaxhighlight>This is a "cast by name". It compares the input to each base's name and performs a static_cast if successful. It also handles interfaces (Q_INTERFACES).

Should be doable with P2996.

However Q_INTERFACES poses the same problem as Q_PROPERTY -- it's just a raw bag of data. Luckily, it's just a list of type names (interfaces that the class implements).

===qt_metacall()===
This can also be possibly generated using P2996. <syntaxhighlight lang="cpp">
int QWidget::qt_metacall(QMetaObject::Call _c, int _id, void **_a)
{
    _id = QObject::qt_metacall(_c, _id, _a);
    if (_id < 0)
        return _id;
    if (_c == QMetaObject::InvokeMetaMethod) {
        if (_id < 29)
            qt_static_metacall(this, _c, _id, _a);
        _id -= 29;
    }
    if (_c == QMetaObject::RegisterMethodArgumentMetaType) {
        if (_id < 29)
            *reinterpret_cast<QMetaType *>(_a[0]) = QMetaType();
        _id -= 29;
    }
    if (_c == QMetaObject::ReadProperty || _c == QMetaObject::WriteProperty
            || _c == QMetaObject::ResetProperty || _c == QMetaObject::BindableProperty
            || _c == QMetaObject::RegisterPropertyMetaType) {
        qt_static_metacall(this, _c, _id, _a);
        _id -= 60;
    }
    return _id;
}

</syntaxhighlight>

===Signals===
moc needs to generate the implementation of each signal, which is just a call to QMetaObject::activate.

<u>This is something that isn't possible in P2996.</u> Since signals are "ordinary" member functions, we can't inject a definition for them. We would need something like injection of token sequences (P3294, which is C++29 material at this point) in order to synthesize each definition (assuming that we get a "define_function" or similar APIs from reflection).

Of course we could concoct a brand new design for declaring signals. For instance, instead of them being member functions (needing a definition), they could be some other declaration (an inner typedef? a static data member?), and needing some other facility to activate them (rather than just "calling" them like we do today). That's brand new territory to explore and widely incompatible with our 30y of Qt history, so possibly not worth it...?

== JSON output ==
moc also generates a JSON file with interesting metaobject information for the class. This sounds impossible to achieve with reflection.