Is the PIMPL idiom really used in practice?

Question

I am reading the book "Exceptional C++" by Herb Sutter, and in that book I have learned about the PIMPL idiom. Basically, the idea is to create a structure for the private objects of a class and dynamically allocate them to decrease the compilation time (and also hide the private implementations in a better manner).

For example:

class X
{
private:
  C c;
  D d;
} ;

could be changed to:

class X
{
private:
  struct XImpl;
  XImpl* pImpl;
};

and, in the .cpp file, the definition:

struct X::XImpl
{
  C c;
  D d;
};

This seems pretty interesting, but I have never seen this kind of approach before, neither in the companies I have worked, nor in open source projects that I've seen the source code. So, I am wondering it this technique is really used in practice?

Should I use it everywhere, or with caution? And is this technique recommended to be used in embedded systems (where the performance is very important)?

Is this essentially the same as a deciding that X is an (abstract) interface, and Ximpl is the implementation? `struct XImpl : public X`. That feels more natural to me. Is there some other issue I've missed? — Aaron McDaid, Jan 23 '12 at 14:02
@AaronMcDaid: It's similar, but has the advantages that (a) member functions don't have to be virtual, and (b) you don't need a factory, or the definition of the implementation class, to instantiate it. — Mike Seymour, Jan 23 '12 at 14:06
@AaronMcDaid The pimpl idiom avoids virtual function calls. It's also a bit more C++-ish (for some conception of C++-ish); you invoke constructors, rather than factory functions. I've used both, depending on what is in the existing code base---the pimpl idiom (originally called the Cheshire cat idiom, and predating Herb's description of it by at least 5 years) seems to have a longer history and be more widely used in C++, but otherwise, both work. — James Kanze, Jan 23 '12 at 14:09
In C++, pimpl should be implemented with `const unique_ptr` rather than `XImpl*`. — Neil G, Jan 23 '12 at 18:22
"never seen this kind of approach before, neither in the companies I have worked, nor in open source projects". Qt is hardly ever using it NOT. — ManuelSchneid3r, Sep 25 '15 at 14:31
https://stackoverflow.com/questions/2346163/pimpl-idiom-vs-bridge-design-pattern — parasrish, Nov 01 '17 at 16:54
@NeilG That would require the definition of XImpl to be visible to clients, which would defeat the PImpl pattern. If X contains a unique_ptr member, X is not fully defined unless unique_ptr is defined, which is not defined unless XImpl is defined. So, all client cpps other than X.cpp will fail to compile, unless we lose half the benefits of Pimpl by including the definition of XImpl in X.h. — Keith Russell, Jun 21 '18 at 16:36
Yes, I did try it. The code in your most recent comment does not require the definition of XImpl or std::unique_ptr, so it will compile. — Keith Russell, Jun 21 '18 at 21:53
I did try it, to double-check. I am right. It doesn’t compile. — Keith Russell, Jun 21 '18 at 23:15
@KeithRussell Your question is "Do you need to define `XImpl` before declaring `unique_ptr` as a member of some class `X`?" You seem to think the answer is yes. — Neil G, Jun 21 '18 at 23:25
I feel that this much back-and-forth is only diluting the comments section on this question. Instead of simply disagreeing with me, please replace the pointer in the OP’s example with a unique_ptr, then include X.h in a second Y.cpp file (not X.cpp, where the definition of XImpl lives). If your compiler lets you even get that far, try instantiating an X in that Y.cpp file. — Keith Russell, Jun 21 '18 at 23:27
@KeithRussell [You are wrong.](https://herbsutter.com/gotw/_101/) `XImpl` needs to only be defined in the implementation file; it can be simply declared as a class in the header file. Also, see this question: https://stackoverflow.com/questions/9020372/how-do-i-use-unique-ptr-for-pimpl — Neil G, Jun 21 '18 at 23:31
We are both wrong. That example contains a small-looking but critical difference: Unlike the OP’s code, GotW’s code has an explicit destructor for X, which prevents the destructor for unique_ptr from being defined inline in an implicit destructor for X. This is a great way to get around the problem, which definitely does occur in OP’s code if we replace the opaque pointer with a unique_ptr. (And I’m going to start using this trick — but I’ll comment it where I do, because it relies upon private details of STL.) — Keith Russell, Jun 21 '18 at 23:44
unique_ptr will generate a call to XImpl’s destructor regardless of whether we (explicitly) define a constructor for X. This is the point of using unique_ptr. The rub is just that XImpl’s destructor must be declared in any translation unit containing a definition (implicit or no) of X’s destructor, or else unique_ptr’s generation of that call will fail. For this reason, in order to preserve the PImpl pattern, X’s destructor must be explicit (and defined outside of the header). — Keith Russell, Jun 21 '18 at 23:57
Yes. What I was missing was that STL can define template classes “piecemeal”. As long as we hide all calls to most of unique_ptr’s constructors and to its destructor, we’re fine — and we can do this by suppressing the implicit definition of X’s constructor and destructor. In your original comment, you just mentioned using unique_ptr, but did not mention that OP’s code as written could not simply have its opaque pointer replaced with a unique_ptr — OP would also have to define X’s destructor. — Keith Russell, Jun 22 '18 at 00:16
@KeithRussell —which you have to do anyway whenever you're using the pimpl idiom. — Neil G, Jun 22 '18 at 00:31
True. (Looking at it again, OP’s code lacks both a constructor and a destructor for his opaque-pointer version of X, even though it seems apparent he intends X to own pImpl in a RAII sense.) — Keith Russell, Jun 22 '18 at 00:49
Addendum: Implicit, generated assignment operator and copy-constructor must also be suppressed, as with the implicit default constructor and destructor. — Keith Russell, Jun 22 '18 at 14:09
@NeilG now I guess the recommended way is to use std::experimental::propagate_const> pImpl — pratikpc, Sep 22 '20 at 07:23
@KeithRussell the generated assignment operator and copy-constructors will be deleted simply due to unique_ptr not having available assignment operator / copy constructor. And the move assignment operator and move constructor are not implicitly declared when there is a user declared destructor in the class. So I think the GotW example is, unsurprisingly, correct. But in general using a unique_ptr on an incomplete type seems like dangerous territory. — Andy Borrell, Apr 13 '21 at 12:44
@AndyBorrell Don’t fear it! I’ve been doing it for years since this question, and I love it. Can’t do it in a lot of cases, but the toolchain will tell you when that is. — Keith Russell, Apr 13 '21 at 13:56

score 137 · Accepted Answer · edited Apr 11 '19 at 09:11

So, I am wondering it this technique is really used in practice? Should I use it everywhere, or with caution?

Of course it is used. I use it in my project, in almost every class.

Reasons for using the PIMPL idiom:

Binary compatibility

When you're developing a library, you can add/modify fields to XImpl without breaking the binary compatibility with your client (which would mean crashes!). Since the binary layout of X class doesn't change when you add new fields to Ximpl class, it is safe to add new functionality to the library in minor versions updates.

Of course, you can also add new public/private non-virtual methods to X/XImpl without breaking the binary compatibility, but that's on par with the standard header/implementation technique.

Data hiding

If you're developing a library, especially a proprietary one, it might be desirable not to disclose what other libraries / implementation techniques were used to implement the public interface of your library. Either because of Intellectual Property issues, or because you believe that users might be tempted to take dangerous assumptions about the implementation or just break the encapsulation by using terrible casting tricks. PIMPL solves/mitigates that.

Compilation time

Compilation time is decreased, since only the source (implementation) file of X needs to be rebuilt when you add/remove fields and/or methods to the XImpl class (which maps to adding private fields/methods in the standard technique). In practice, it's a common operation.

With the standard header/implementation technique (without PIMPL), when you add a new field to X, every client that ever allocates X (either on stack, or on heap) needs to be recompiled, because it must adjust the size of the allocation. Well, every client that doesn't ever allocate X also need to be recompiled, but it's just overhead (the resulting code on the client side will be the same).

What is more, with the standard header/implementation separation XClient1.cpp needs to be recompiled even when a private method X::foo() was added to X and X.h changed, even though XClient1.cpp can't possibly call this method for encapsulation reasons! Like above, it's pure overhead and is related with how real-life C++ build systems work.

Of course, recompilation is not needed when you just modify the implementation of the methods (because you don't touch the header), but that's on par with the standard header/implementation technique.

Is this technique recommended to be used in embedded systems (where the performance is very important)?

That depends on how powerful your target is. However the only answer to this question is: measure and evaluate what you gain and lose. Also, take into consideration that if you're not publishing a library meant to be used in embedded systems by your clients, only the compilation time advantage applies!

In the Qt library this method is also used in smart pointer situations. So QString keeps its contents as a immutable class internally. When the public class is "copied" the private member's pointer is copied instead of the entire private class. These private classes then also use smart pointers, so you basically get garbage collection with most of the classes, in addition to greatly improved performance due to pointer copying instead of full class copying — Timothy Baldridge, Jan 23 '12 at 14:54
Even more, with the pimpl idiom Qt can maintain both forward and back binary compatibility within a single major version (in most cases). IMO this is by far the most significant reason to use it. — whitequark, Jan 27 '12 at 12:39
Also less chance of namespace collision in the headers, since most of the includes needed for data members can be moved into the impl definition — Mark Borgerding, Aug 14 '15 at 17:26
It is also useful for implementing platform-specific code, as you can retain same API. — doc, Jan 10 '17 at 13:41

score 51 · Answer 2 · edited Mar 30 '21 at 13:44

It seems that a lot of libraries out there use it to stay stable in their API, at least for some versions.

But as for all things, you should never use anything everywhere without caution. Always think before using it. Evaluate what advantages it gives you, and if they are worth the price you pay.

The advantages it may give you are:

helps in keeping binary compatibility of shared libraries
hiding certain internal details
decreasing recompilation cycles

Those may or may not be real advantages to you. Like for me, I don't care about a few minutes recompilation time. End users usually also don't, as they always compile it once and from the beginning.

Possible disadvantages are (also here, depending on the implementation and whether they are real disadvantages for you):

Increase in memory usage due to more allocations than with the naïve variant
increased maintenance effort (you have to write at least the forwarding functions)
performance loss (the compiler may not be able to inline stuff as it is with a naïve implementation of your class)

So carefully give everything a value, and evaluate it for yourself. For me, it almost always turns out that using the PIMPL idiom is not worth the effort. There is only one case where I personally use it (or at least something similar):

My C++ wrapper for the Linux stat call. Here the struct from the C header may be different, depending on what #defines are set. And since my wrapper header can't control all of them, I only #include <sys/stat.h> in my .cxx file and avoid these problems.

It should almost always be used for system interfaces, to make the interface code system independent. My `File` class (which exposes much of the information `stat` would return under Unix) uses the same interface under both Windows and Unix, for example. — James Kanze, Jan 23 '12 at 14:12
@JamesKanze: Even there I personally would first sit for a moment and think about if it is not maybe sufficient to have a few `#ifdef`s to make the wrapper as thin as possible. But everyone has different goals, the important thing is to take the time to think about it instead of blindly following something. — PlasmaHH, Jan 23 '12 at 14:34

score 32 · Answer 3 · edited Mar 10 '21 at 15:56

32

I agree with all the others about the goods, but let me put in evidence about a limit: doesn't work well with templates.

The reason is that template instantiation requires the full declaration available where the instantiation took place. (And that's the main reason you don't see template methods defined into .cpp files.)

You can still refer to templatised subclasses, but since you have to include them all, every advantage of "implementation decoupling" on compiling (avoiding to include all platform-specific code everywhere, shortening compilation) is lost.

It is a good paradigm for classic OOP (inheritance based), but not for generic programming (specialization based).

edited Mar 10 '21 at 15:56

Peter Mortensen

28,342
21
95
123

answered Jan 23 '12 at 15:21

Emilio Garavaglia

18,858
2
41
60

4

You have to be more precise: there's absolutely no problem when **using** PIMPL classes as template type arguments. Only if the implementation class itself needs to be parametrized on the outer class's template arguments, it can't be hidden from the interface header anymore, even if it's still a private class. If you can delete the template argument, you can certainly still do "proper" PIMPL. With type deletion you can also do the PIMPL in a base non-template class, and then have the template class derive from it. – Kuba hasn't forgotten Monica Mar 06 '14 at 22:17

score 25 · Answer 4 · edited Mar 30 '21 at 13:46

Other people have already provided the technical up/downsides, but I think the following is worth noting:

First and foremost, don't be dogmatic. If PIMPL works for your situation, use it - don't use it just because "it's better OO since it really hides implementation", etc. Quoting the C++ FAQ:

encapsulation is for code, not people (source)

Just to give you an example of open source software where it is used and why: OpenThreads, the threading library used by the OpenSceneGraph. The main idea is to remove from the header (e.g., <Thread.h>) all platform-specific code, because internal state variables (e.g., thread handles) differ from platform to platform. This way one can compile code against your library without any knowledge of the other platforms' idiosyncrasies, because everything is hidden.

score 12 · Answer 5 · answered Jan 23 '12 at 14:04

12

I would mainly consider PIMPL for classes exposed to be used as an API by other modules. This has many benefits, as it makes recompilation of the changes made in the PIMPL implementation does not affect the rest of the project. Also, for API classes they promote a binary compatibility (changes in a module implementation do not affect clients of those modules, they don't have to be recompiled as the new implementation has the same binary interface - the interface exposed by the PIMPL).

As for using PIMPL for every class, I would consider caution because all those benefits come at a cost: an extra level of indirection is required in order to access the implementation methods.

answered Jan 23 '12 at 14:04

Ghita

4,088
4
36
65

"an extra level of indirection is required in order to access the implementation methods." It is? – xaxxon Jun 11 '17 at 12:28
1

@xaxxon yes, it is. pimpl is slower if the methods are low level. never use it for stuff that lives in a tight loop, for example. – Erik Aronesty Dec 04 '18 at 16:01
1

@xaxxon I would say in general case an extra level is required. If inlining is performed than no. But inlinning would not be an option in code compiled in a different dll. – Ghita Dec 11 '18 at 08:56

score 5 · Answer 6 · answered Aug 10 '15 at 16:16

I used to use this technique a lot in the past but then found myself moving away from it.

Of course it is a good idea to hide the implementation detail away from the users of your class. However you can also do that by getting users of the class to use an abstract interface and for the implementation detail to be the concrete class.

The advantages of pImpl are:

Assuming there is just one implementation of this interface, it is clearer by not using abstract class / concrete implementation
If you have a suite of classes (a module) such that several classes access the same "impl" but users of the module will only use the "exposed" classes.
No v-table if this is assumed to be a bad thing.

The disadvantages I found of pImpl (where abstract interface works better)

Whilst you may have only one "production" implementation, by using an abstract interface you can also create a "mock" inmplementation that works in unit testing.
(The biggest issue). Before the days of unique_ptr and moving you had restricted choices as to how to store the pImpl. A raw pointer and you had issues about your class being non-copyable. An old auto_ptr wouldn't work with forwardly declared class (not on all compilers anyway). So people started using shared_ptr which was nice in making your class copyable but of course both copies had the same underlying shared_ptr which you might not expect (modify one and both are modified). So the solution was often to use raw pointer for the inner one and make the class non-copyable and return a shared_ptr to that instead. So two calls to new. (Actually 3 given old shared_ptr gave you a second one).
Technically not really const-correct as the constness isn't propagated through to a member pointer.

In general I have therefore moved away in the years from pImpl and into abstract interface usage instead (and factory methods to create instances).

score 5 · Answer 7 · edited Mar 30 '21 at 13:41

I think this is one of the most fundamental tools for decoupling.

I was using PIMPL (and many other idioms from Exceptional C++) on embedded project (SetTopBox).

The particular purpose of this idiom in our project was to hide the types XImpl class uses. Specifically, we used it to hide details of implementations for different hardware, where different headers would be pulled in. We had different implementations of XImpl classes for one platform and different for the other. Layout of class X stayed the same regardless of the platform.

ncsc · Answer 8 · 2015-09-04T11:20:04.483

As many other said, the Pimpl idiom allows to reach complete information hiding and compilation independency, unfortunately with the cost of performance loss (additional pointer indirection) and additional memory need (the member pointer itself). The additional cost can be critical in embedded software development, in particular in those scenarios where memory must be economized as much as possible. Using C++ abstract classes as interfaces would lead to the same benefits at the same cost. This shows actually a big deficiency of C++ where, without recurring to C-like interfaces (global methods with an opaque pointer as parameter), it is not possible to have true information hiding and compilation independency without additional resource drawbacks: this is mainly because the declaration of a class, which must be included by its users, exports not only the interface of the class (public methods) needed by the users, but also its internals (private members), not needed by the users.

score 3 · Answer 9 · edited Mar 10 '21 at 15:43

Here is an actual scenario I encountered, where this idiom helped a great deal. I recently decided to support DirectX 11, as well as my existing DirectX 9 support, in a game engine.

The engine already wrapped most DX features, so none of the DX interfaces were used directly; they were just defined in the headers as private members. The engine uses DLL files as extensions, adding keyboard, mouse, joystick, and scripting support, as week as many other extensions. While most of those DLLs did not use DX directly, they required knowledge and linkage to DX simply because they pulled in headers that exposed DX. In adding DX 11, this complexity was to increase dramatically, however unnecessarily. Moving the DX members into a PIMPL, defined only in the source, eliminated this imposition.

On top of this reduction of library dependencies, my exposed interfaces became cleaner as I moved private member functions into the PIMPL, exposing only front facing interfaces.

score 2 · Answer 10 · edited Mar 10 '21 at 15:54

2

It is used in practice in a lot of projects. It's usefulness depends heavily on the kind of project. One of the more prominent projects using this is Qt, where the basic idea is to hide implementation or platform-specific code from the user (other developers using Qt).

This is a noble idea, but there is a real drawback to this: debugging As long as the code hidden in private implementations is of premium quality this is all well, but if there are bugs in there, then the user/developer has a problem, because it just a dumb pointer to a hidden implementation, even if he/she has the implementations source code.

So as in nearly all design decisions there are pros and cons.

edited Mar 10 '21 at 15:54

Peter Mortensen

28,342
21
95
123

answered Jan 23 '12 at 14:08

codencandy

1,641
10
19

9

it's dumb but it's typed... why can't you follow the code in the debugger? – UncleZeiv Jan 23 '12 at 14:31
2

Generally speaking, to debug into Qt code you need to build Qt yourself. Once you do, there's no problem stepping into PIMPL methods, and inspecting contents of the PIMPL data. – Kuba hasn't forgotten Monica Mar 06 '14 at 22:20

score 0 · Answer 11 · answered Sep 21 '16 at 01:45

One benefit I can see is that it allows the programmer to implement certain operations in a fairly fast manner:

X( X && move_semantics_are_cool ) : pImpl(NULL) {
    this->swap(move_semantics_are_cool);
}
X& swap( X& rhs ) {
    std::swap( pImpl, rhs.pImpl );
    return *this;
}
X& operator=( X && move_semantics_are_cool ) {
    return this->swap(move_semantics_are_cool);
}
X& operator=( const X& rhs ) {
    X temporary_copy(rhs);
    return this->swap(temporary_copy);
}

PS: I hope I'm not misunderstanding move semantics.

Is the PIMPL idiom really used in practice?

11 Answers11

Reasons for using the PIMPL idiom:

Binary compatibility

Data hiding

Compilation time

Linked

Related