D-Pointer

English 简体中文 Українська Български Español فارسی [toc align_right="yes" depth="3"]

What is the d-pointer

If you have looked into Qt source files like "this one":http://qt.gitorious.org/qt/qt/blobs/HEAD/src/gui/widgets/qlabel.cpp, you will find it generously sprinkled with

Q_D

and

Q_Q

macros. This article unravels the purpose of these macros. The

Q_D

and

Q_Q

macros are part of a design pattern called the d-pointer (also called the "opaque pointer":http://en.wikipedia.org/wiki/Opaque_pointer) where the implementation details of a library may be hidden from its users and changes to the implementation can be made to a library without breaking binary compatibility.

Binary compatibility, what is that?

When designing libraries like Qt, it is desirable that applications that dynamically link to Qt continue to run without recompiling even after the Qt library is upgraded/replaced with another version. For example, if your application CuteApp was based on Qt 4.5, you should be able to upgrade the Qt libraries (on Windows shipped with the application, on Linux often comes from package manager automatically!) from version 4.5 to Qt 4.6 and your CuteApp that was built with Qt 4.5 should still be able to run.

What breaks binary compatibility

So, when does a change in the library require a recompilation of the application? Let's take a simple example:

 class Widget {
 …
 private:
 Rect m_geometry;
 };

class Label : public Widget {
 public:
 …
 String text() const { return m_text; }
 private:
 String m_text;
 };

Here we have a Widget that contains geometry as a member variable. We compile our Widget and ship it as WidgetLib 1.0.

For WidgetLib 1.1, someone comes up with the bright idea to add support for stylesheets. No sweat, we just add new methods and add a new data member.

 class Widget {
 …
 private:
 Rect m_geometry;
 String m_stylesheet; // NEW in WidgetLib 1.1
 };

class Label : public Widget {
 public:
 …
 String text() const { return m_text; }
 private:
 String m_text;
 };

We ship WidgetLib 1.1 with the above change only to find that CuteApp that was compiled and ran just fine with WidgetLib 1.0 crashes gloriously!

Why did it crash?

The reason is that by adding a new data member, we ended up changing the size of Widget and Label objects. Why does this matter? When your C++ compiler generates code, it uses 'offsets' to access data within an object.

Here's an oversimplified version of how the above POD objects might look in memory.

Label object layout in WidgetLib 1.0	Label object layout in WidgetLib 1.1
m_geometry <offset 0>	m_geometry <offset 0>
—————	m_stylesheet <offset 1>
m_text <offset 1>	——————
—————	m_text <offset 2>

In WidgetLib 1.0, the text member of Label was at (logical) offset 1. The code generated by the compiler in the application for the method

Label::text()

translates to accessing offset 1 of the label object in the application. In WidgetLib 1.1, the text member of Label has shifted to (logical) offset 2! Since the application has not been recompiled, it continues to think that

text

is at offset 1 and ends up accessing the

stylesheet

variable! I am sure at this point there are a few who are wondering why the

Label::text()

's offset calculation code ended up in the CuteApp binary and not in the WidgetLib binary. The answer is that the code for

Label::text()

was defined in the header file and the compiler ended up "inlining":http://en.wikipedia.org/wiki/Inline_function it. So, does the situation change if

Label::text()

had not been inlined? Say,

Label::text()

was moved to the source file? Well, no. The C++ compiler relies on the size of objects being the same at compile time and run-time. For example, stack winding/unwinding - if you created a Label object on the stack, the compiler generated code to allocate space on the stack based on the Label's size at compile time. Since the size of Label is different at run time in WidgetLib 1.1, Label's constructor overwrites existing stack data and ends up corrupting the stack.

Never change the size of an exported C++ class

In summary, never ever change the size or layout (don't move the data around) of exported (i.e visible to the user) C++ classes once your library has been released. A C++ compiler generates code assuming that the size or the ordering of data in a class does not change after the application has been compiled.

So, how can one not change the size of the object and yet add new features?

The d-pointer

The trick is to keep the size of all public classes of a library constant by only storing a single pointer. This pointer points to a private/internal data structure that contains all the data. The size of this internal structure can shrink or grow without having any side-effect on the application because the pointer is accessed only in the library code and from the application's point of view the size of the object never changes - it's always the size of the pointer. This pointer is called the d-pointer.

The spirit of this pattern is outlined in the code below (all code in this article doesn't have destructors, of course you should add them in real code).

widget.h

 /* Since d_ptr is a pointer and is never referended in header file
 (it would cause a compile error) WidgetPrivate doesn't have to be included,
 but forward-declared instead.
 The definition of the class can be written in widget.cpp or
 in a separate file, say widget_p.h*/

class WidgetPrivate;

class Widget {
 …
 Rect geometry() const;
 …
 private:
 WidgetPrivate *d_ptr;
 };

widget_p.h, which is the private header file of the widget class

 /* widget_p.h (_p means private) */
 struct WidgetPrivate {
 Rect geometry;
 String stylesheet;
 };

widget.cpp

 // With this #include, we can access WidgetPrivate.
 #include "widget_p.h"
 Widget::Widget()
 : d_ptr(new WidgetPrivate) {
 // Creation of private data
 }

Rect Widget::geometry() const {
 // The d-ptr is only accessed in the library code
 return d_ptr->geometry;
 }

Next, there's an example of a child class based on Widget.

label.h

 class Label : public Widget {
 …
 String text();
 private:
 // Each class maintains its own d-pointer
 LabelPrivate *d_ptr;
 };

label.cpp

 // Unlike WidgetPrivate, the author decided LabelPrivate to be defined in the source file itself
 struct LabelPrivate {
 String text;
 };

Label::Label()
 : d_ptr(new LabelPrivate) {
 }

String Label::text() {
 return d_ptr->text;
 }

With the above structure, CuteApp never accesses the d-pointer directly. And since the d-pointer is only ever accessed in WidgetLib and WidgetLib is recompiled for every release, the Private class can freely change with no impact on CuteApp.

Other benefits of d-pointer

It's not all about binary compatibility. The d-pointer has other benefits:

• Hide implementation details - We can ship WidgetLib with just the header files and the binaries. The .cpp files can be closed source.
• The header file is clean of implementation details and can serve as the API reference.
• Since the header files needed for implementation are moved from the header file into the implementation (source) file, compiles are much faster.

It is indeed true that the above benefits appear trivial. The real reason to use d-pointers in Qt is for binary compatibility and the fact that Qt started out closed source.

The q-pointer

So far, we have only seen the d-pointer as a C-style data structure. In reality, it contains private methods (helper functions). For example,

LabelPrivate

might have a

getLinkTargetFromPoint()

helper function that is required to find the link target when the mouse is clicked. In many cases, these helper methods require access to the public class i.e some functions from Label or from its base class Widget. For example, a helper method,

setTextAndUpdateWidget()

might want to call

Widget::update()

which is a public method to schedule a redraw the Widget. So, the

WidgetPrivate

stores a pointer to the public class called the q-pointer. Modifying the code above for the q-pointer, we get:

widget.h

 class WidgetPrivate;

class Widget {
 …
 Rect geometry() const;
 …
 private:
 WidgetPrivate *d_ptr;
 };

widget_p.h

 struct WidgetPrivate {
 // Constructor that initializes the q-ptr
 WidgetPrivate(Widget *q) : q_ptr(q) { }
 Widget *q_ptr; // q-ptr points to the API class
 Rect geometry;
 String stylesheet;
 };

widget.cpp

 #include "widget_p.h"
 // Create private data.
 // Pass the 'this' pointer to initialize the q-ptr
 Widget::Widget()
 : d_ptr(new WidgetPrivate(this)) {
 }

Rect Widget::geometry() const {
 // the d-ptr is only accessed in the library code
 return d_ptr->geometry;
 }

Next, another class based on Widget.

label.h

 class Label : public Widget {
 …
 String text() const;
 private:
 LabelPrivate *d_ptr;
 };

label.cpp

 // Unlike WidgetPrivate, the author decided LabelPrivate to be defined in the source file itself
 struct LabelPrivate {
 LabelPrivate(Label *q) : q_ptr(q) { }
 Label *q_ptr;
 String text;
 };

Label::Label()
 : d_ptr(new LabelPrivate(this)) {
 }

String Label::text() {
 return d_ptr->text;
 }

Inheriting d-pointers for optimization

In the above code, creating a single Label results in the memory allocation for

LabelPrivate

and

WidgetPrivate

. If we were to employ this strategy for Qt, the situation becomes quite worse for classes like

QListWidget

- it is 6 levels deep in the class inheritance hierarchy and it would result in upto 6 memory allocations!

This is solved by having an inheritance hierarchy for our private classes and having the class getting instantiated pass on a the d-pointer all the way up.

Notice that when inheriting d-pointers, the declaration of the private class has to be in a separate file, for example widget_p.h. It's no longer possible to declare it in the widget.cpp file.

widget.h

 class Widget {
 public:
 Widget();
 …
 protected:
 // only sublasses may access the below
 Widget(WidgetPrivate &d); // allow subclasses to initialize with their own concrete Private
 WidgetPrivate *d_ptr;
 };

widget_p.h

 struct WidgetPrivate {
 WidgetPrivate(Widget *q) : q_ptr(q) { } // constructor that initializes the q-ptr
 Widget *q_ptr; // q-ptr that points to the API class
 Rect geometry;
 String stylesheet;
 };

widget.cpp

 Widget::Widget()
 : d_ptr(new WidgetPrivate(this)) {
 }

Widget::Widget(WidgetPrivate &d)
 : d_ptr(&d) {
 }

label.h

 class Label : public Widget {
 public:
 Label();
 …
 protected:
 Label(LabelPrivate &d); // allow Label subclasses to pass on their Private
 // notice how Label does not have a d_ptr! It just uses Widget's d_ptr.
 };

label.cpp

 #include "widget_p.h"

class LabelPrivate : public WidgetPrivate {
 public:
 String text;
 };

Label::Label()
 : Widget(*new LabelPrivate) // initialize the d-pointer with our own Private {
 }

Label::Label(LabelPrivate &d)
 : Widget(d) {
 }

Do you see the beauty? When we now create a

Label

object, it will create a

LabelPrivate

(which subclasses

WidgetPrivate

). It passes on the concrete d-pointer to Widget's protected constructor! Now, when a

Label

object is created, there is only one memory allocation. Label also has a protected constructor that can be used by its subclasses to provide their own private classes.

d-pointers in Qt

In Qt, practically every public class uses the d-pointer approach. The only cases where it's not used is when it is known in advance that the class will never ever have extra member variables added to it. For example, for classes like

QPoint

,

QRect

, no new members are expected to be added and hence the data members are stored straight into the class itself instead of using the d-pointer. Notice that in Qt, the base class of all Private objects is

QObjectPrivate

.

Q_D and Q_Q

A side effect of the optimization that we did in the previous step is that the q-ptr and d-ptr are of type

Widget

and

WidgetPrivate

. This means that the following won't work.

 void Label::setText(const String &text) {
 // won't work! since d_ptr is of type WidgetPrivate even though it points to LabelPrivate object
 d_ptr->text = text;
 }

Hence, when accessing the d-pointer in a subclass, we need to static_cast to the appropriate type.

 void Label::setText(const String &text) {
 LabelPrivate '''d = static_cast<LabelPrivate'''>(d_ptr); // cast to our private type
 d->text = text;
 }

As you can see, it's not pretty having static_cast all over the place. Instead, there are two macros defined in src/corelib/global/qglobal.h which make it straighforward:

global.h

 #define Q_D(Class) Class##Private * const d = d_func()
 #define Q_Q(Class) Class * const q = q_func()

label.cpp

//With Q_D you can use the members of LabelPrivate from Label
 void Label::setText(const String &text) {
 Q_D(Label);
 d->text = text;
 }
//With Q_Q you can use the members of Label from LabelPrivate
 void LabelPrivate::someHelperFunction() {
 Q_Q(Label);
 q->selectAll();
 }

Q_DECLARE_PRIVATE and Q_DECLARE_PUBLIC

Qt classes have a

Q_DECLARE_PRIVATE

a macro in the public class. The macro reads:

qglobal.h

 #define Q_DECLARE_PRIVATE(Class)  inline Class##Private* d_func() { return reinterpret_cast<Class##Private '''>(qGetPtrHelper(d_ptr)); }  inline const Class##Private''' d_func() const {  return reinterpret_cast<const Class##Private *>(qGetPtrHelper(d_ptr)); }  friend class Class##Private;

This macro can be used this way:

qlabel.h

 class QLabel {
 private:
 Q_DECLARE_PRIVATE(QLabel);
 };

The idea is that

QLabel

provides a function

d_func()

that allows access to its private internal class. The method itself is private (since the macro is inside a private section in qlabel.h). The

d_func()

can however be invoked by friends (C++ friend) of

QLabel

. This is primarily useful for access of information by Qt classes which cannot get access of some

QLabel

information using public api. As a bizarre example,

QLabel

might keep track of how many times the user has clicked on a link. However, there is no public API to access this information.

QStatistics

is a class that needs this information. A Qt developer will add

QStatistics

as a friend of

QLabel

and

QStatistics

can then do

label->d_func()->linkClickCount

. The

d_func

also has the advantage to enforce const-correctness: In a const member function of MyClass you need a Q_D(const MyClass) and thus you can only call const member functions in MyClassPrivate. With a "naked" d_ptr you could also call non-const functions.