34

In C++ the standard library is wrapped in the std namespace and the programmer is not supposed to define anything inside that namespace. Of course the standard include files don't step on each other names inside the standard library (so it's never a problem to include a standard header).

Then why isn't the whole standard library included by default instead of forcing programmers to write for example #include <vector> each time? This would also speed up compilation as the compilers could start with a pre-built symbol table for all the standard headers.

Pre-including everything would also solve some portability problems: for example when you include <map> it's defined what symbols are taken into std namespace, but it's not guaranteed that other standard symbols are not loaded into it and for example you could end up (in theory) with std::vector also becoming available.

It happens sometimes that a programmer forgets to include a standard header but the program compiles anyway because of an include dependence of the specific implementation. When moving the program to another environment (or just another version of the same compiler) the same source code could however stop compiling.

From a technical point of view I can image a compiler just preloading (with mmap) an optimal perfect-hash symbol table for the standard library. This should be faster to do than loading and doing a C++ parse of even a single standard include file and should be able to provide faster lookup for std:: names. This data would also be read-only (thus probably allowing a more compact representation and also shareable between multiple instances of the compiler).

These are however just shoulds as I never implemented this.

The only downside I see is that we C++ programmers would lose compilation coffee breaks and Stack Overflow visits :-)

EDIT

Just to clarify the main advantage I see is for the programmers that today, despite the C++ standard library being a single monolithic namespace, are required to know which sub-part (include file) contains which function/class. To add insult to injury when they make a mistake and forget an include file still the code may compile or not depending on the implementation (thus leading to non-portable programs).

6502
  • 104,192
  • 14
  • 145
  • 251
  • That's not the way it has been done historically. The headers are big; including all of them in every program would grow the amount of code that the compiler has to handle. You don't include libraries; you include headers that declare the facilities provided by the library. When you link, then the library is added to your program. You're right about the 'missing header' problem. You might look up 'include what you use' (IWYU) via Google. IIRC, it is/was a Google code project (so probably now on Github). – Jonathan Leffler Sep 12 '15 at 07:19
  • 2
    Related: http://stackoverflow.com/questions/31816095/why-should-i-not-include-bits-stdc-h – πάντα ῥεῖ Sep 12 '15 at 07:21
  • 4
    @JonathanLeffler: Don't you think that loading (`mmap`-ping) a pre-parsed data structure with the knowledge of the standard library would be faster than loading standard files every time? – 6502 Sep 12 '15 at 07:21
  • This is hardly unique to C++... – Oliver Charlesworth Sep 12 '15 at 07:22
  • I've not done the measurements; I don't know. It increases the amount of lookup that must be done when parsing the code you write. I'm old school enough to be worried about the idea, but that might not be relevant any more. – Jonathan Leffler Sep 12 '15 at 07:28
  • @πάνταῥεῖ: the sense of the answers are more or less "no... C++ programmers must suffer this annoyance because the standard committee thought is was fun to doom them to search on google which header provides a certain feature". – 6502 Sep 12 '15 at 07:29
  • @JonathanLeffler: note that pre-analyzing the standard library provides also this advantage (e.g. perfect hashing). – 6502 Sep 12 '15 at 07:31
  • 3
    Note that the previous question [Why not include all the standard headers always?](http://stackoverflow.com/questions/16830134/why-not-include-all-the-standard-headers-always) addresses some of the issues implicit in this question. – Jonathan Leffler Sep 12 '15 at 07:38
  • 2
    This has nothing to do with the standards committee, its down to the compiler writers how they implement header inclusion. Nothing says the headers need to be a file on disk. As far as the standards go they could be built in to the compiler. – Galik Sep 12 '15 at 07:59
  • @Galik: the language **requires** programmers to include the specific header for any parts of the standard library they use. AFAIK the C++ program `int main(int argc, const char *argv[]){ std::cout << "Hello, world.\n"; }` must not compile in a compliant implementation. – 6502 Sep 12 '15 at 08:55
  • @6502 Sure the program file must use the include statement, but how that include statement is implemented is up to the compiler. It could just turn on a switch to prevent a compiler error about missing the include file for example. – Galik Sep 12 '15 at 09:03
  • Including `` can have a runtime impact, depending on how the initialization of the `cout` etc. objects is implemented. libstdc++ seems to use a Schwartz Counter, for example (with atomic operations). – dyp Sep 12 '15 at 13:21
  • @dyp: implementation quality issue, it would be perfectly within the implementation right to do its own work first (and last) without relying on header inclusion. After all, nobody else defines when your globals are built/destroyed! – Matthieu M. Sep 12 '15 at 17:29
  • @JonathanLeffler: *It increases the amount of lookup that must be done when parsing the code you write.* => are you talking about header resolution or symbol resolution? For the former it's a gain (file I/O are so slow) and for the latter the number of symbols defined by the C++ Standard Library in any "big" C++ programs is a drop in the ocean. – Matthieu M. Sep 12 '15 at 17:37
  • @6502: "must not compile in a compliant implementation" -- it can compile but a diagnostic must be issued by a compliant implementation. – Steve Jessop Sep 12 '15 at 18:12
  • @SteveJessop: "must not compile" meant "must not compile cleanly". Otherwise we can discuss that even the source program "asdfghjkl;" can "compile" :-D – 6502 Sep 12 '15 at 19:08
  • You're basically asking why C++ doesn't have a module system? – user253751 Sep 13 '15 at 02:35
  • @immibis: No. Python has a module system and a partitioned namespace and you still have to declare which part of the standard library you want to use. C++ instead has a single monolithic namespace with all the standard library in it... yet the programmers are required to know and to tell the compiler which sub-part (include file) contains which function/class. – 6502 Sep 13 '15 at 06:29
  • @6502 Oops, I might've meant to comment on another related question (in that case, I don't know which one now). It does sort of fit with the 3rd/4th/edit paragraphs. – user253751 Sep 13 '15 at 07:40

7 Answers7

18

Short answer is because it is not the way the C++ language is supposed to be used

There are good reasons for that:

  • namespace pollution - even if this could be mitigated because std namespace is supposed to be self coherent and programmer are not forced to use using namespace std;. But including the whole library with using namespace std; will certainly lead to a big mess...
  • force programmer to declare the modules that he wants to use to avoid inadvertently calling a wrong standard function because standard library is now huge and not all programmers know all modules
  • history: C++ has still strong inheritance from C where namespace do not exist and where the standard library is supposed to be used as any other library.

To go in your sense, Windows API is an example where you only have one big include (windows.h) that loads many other smaller include files. And in fact, precompiled headers allows that to be fast enough

So IMHO a new language deriving from C++ could decide to automatically declare the whole standard library. A new major release could also do it, but it could break code intensively using using namespace directive and having custom implementations using same names as some standard modules.

But all common languages that I know (C#, Python, Java, Ruby) require the programmer to declare the parts of the standard library that he wants to use, so I suppose that systematically making available every piece of the standard library is still more awkward than really useful for the programmer, at least until someone find how to declare the parts that should not be loaded - that's why I spoke of a new derivative from C++

Jonathan Leffler
  • 666,971
  • 126
  • 813
  • 1,185
Serge Ballesta
  • 121,548
  • 10
  • 94
  • 199
  • 8
    `using namespace std` is never guaranteed to work if you have any name that conflicts with any header of the standard library (the reason is that for example `` **can** legally take in also `std::vector`). Indeed `using namespace std` is by itself a bad mistake (except for toy examples) and basically defeats the purpose of namespaces. – 6502 Sep 12 '15 at 08:14
  • IMO the only point with substance in your list is "history"... in other words "it's this way just because. fullstop.". Polluting the namespace is nonsense because that namespace is forbidden (except `swap`) for the programmers and if you import all of it with `using namespace std` and clash a standard name your program is non portable anyway (even if you didn't list the official include file for the name you stepped over). Same goes for calling a standard function without knowing (it can happen anyway with today standard). – 6502 Sep 13 '15 at 06:47
  • 1
    As a compromise they could define a standard header `` that pulls in all of the standard library headers. Then old code won't break (because it won't include ``) and new code will be able to just include `` if the author desires so (and will still have the option of including individual headers instead if that's what the author prefers). – celtschk Sep 13 '15 at 08:26
10

Most of the C++ standard libraries are template based which means that the code they'll generate will depend ultimately in how you use them. In other words, there is very little that could be compiled before instantiate a template like std::vector<MyType> m_collection;.

Also, C++ is probably the slowest language to compile and there is a lot parsing work that compilers have to do when you #include a header file that also includes other headers.

Darien Pardinas
  • 5,176
  • 1
  • 32
  • 43
  • 8
    The fact that C++ is an horrible beast to parse is an argument in **favor** of pre-parsing the whole standard library, not against it. – 6502 Sep 12 '15 at 08:16
  • @6502 True, but it would mean that C++ compilers would be essentially required to implement support for pre-parsing, regardless of whether it fit their particular needs. It would also make it impossible to use preprocessor macros to configure the standard library. – Sneftel Sep 12 '15 at 13:04
  • 1
    The template stuff depends to an extent on whether the implementation implements two-phase lookup. There was a time when pretty much the best MSVC++ could do with a template definition, before it was instantiated, was to tokenize it and check there are the same number of open and close brackets (I exaggerate somewhat). But for a compiler to do two-phase lookup, including more recent efforts from Microsoft, basically *means* that it can and does do more than "very little" useful work compiling the template before knowing the parameters. – Steve Jessop Sep 12 '15 at 18:34
7

Well, first thing first, C++ tries to adhere to "you only pay for what you use".
The standard-library is sometimes not part of what you use at all, or even of what you could use if you wanted.
Also, you can replace it if there's a reason to do so: See libstdc++ and libc++.
That means just including it all without question isn't actually such a bright idea.

Anyway, the committee are slowly plugging away at creating a module-system (It takes lots of time, hopefully it will work for C++1z: C++ Modules - why were they removed from C++0x? Will they be back later on?), and when that's done most downsides to including more of the standard-library than strictly neccessary should disappear, and the individual modules should more cleanly exclude symbols they need not contain.
Also, as those modules are pre-parsed, they should give the compilation-speed improvement you want.

Community
  • 1
  • 1
Deduplicator
  • 41,806
  • 6
  • 61
  • 104
  • A module system is a different thing and partitioning the standard library may be a good idea (but I don't like too much current proposal that IIUC is about injecting module exported names in compilation unit namespace). – 6502 Sep 13 '15 at 07:08
6

You offer two advantages of your scheme:

  • Compile-time performance. But nothing in the standard prevents an implementation from doing what you suggest[*] with a very slight modification: that the pre-compiled table is only mapped in when the translation unit includes at least one standard header. From the POV of the standard, it's unnecessary to impose potential implementation burden over a QoI issue.

  • Convenience to programmers: under your scheme we wouldn't have to specify which headers we need. We do this in order to support C++ implementations that have chosen not to implement your idea of making the standard headers monolithic (which currently is all of them), and so from the POV of the C++ standard it's a matter of "supporting existing practice and implementation freedom at a cost to programmers considered acceptable". Which is kind of the slogan of C++, isn't it?

Since no C++ implementation (that I know of) actually does this, my suspicion is that in point of fact it does not grant the performance improvement you think it does. Microsoft provides precompiled headers (via stdafx.h) for exactly this performance reason, and yet it still doesn't give you an option for "all the standard libraries", instead it requires you to say what you want in. It would be dead easy for this or any other implementation to provide an implementation-specific header defined to have the same effect as including all standard headers. This suggests to me that at least in Microsoft's opinion there would be no great overall benefit to providing that.

If implementations were to start providing monolithic standard libraries with a demonstrable compile-time performance improvement, then we'd discuss whether or not it's a good idea for the C++ standard to continue permitting implementations that don't. As things stand, it has to.

[*] Except perhaps for the fact that <cassert> is defined to have different behaviour according to the definition of NDEBUG at the point it's included. But I think implementations could just preprocess the user's code as normal, and then map in one of two different tables according to whether it's defined.

Steve Jessop
  • 257,525
  • 32
  • 431
  • 672
  • Your point for `` is indeed generic, as many "debug" options exist for the various Standard Library implementations... and that is of course without counting the platform for which you compile (which impacts the size and alignment of the types!). Nonetheless, pre-compiled headers can either branch-out (one pre-compiled header per set of options) or be "all-inclusive" and anything in between. – Matthieu M. Sep 12 '15 at 18:56
  • @MatthieuM: good point, the combinations of options might be intractable for some implementations. My wild-ass guess is that the precompiled table for a gcc target might be smaller than all the code required to support an instruction set, but in many cases is larger than the additional support/config for each little architecture variant and so would bloat the compiler installation itself if it wasn't generated and cached when first used. – Steve Jessop Sep 12 '15 at 19:01
  • The main advantage is for the programmer. The C++ standard library is a single monolithic namespace but the programmers are required to know which sub-part contains which function/class (with the added twist that when you make a mistake the code on some compilers could compile anyway, leading to silently non-portable programs). The fact that I think it will even be (albeit marginally) faster and lighter is just incidental. – 6502 Sep 13 '15 at 06:33
  • @6502: well, if performance isn't important and convenience is, then any programmer can just write a file that lists all the standard headers and include it in everything. At which point it will rapidly transpire that actually performance *is* significant, and in C++ listing only the standard headers needed is one way to improve it. Or maybe (in the implementation you're using) that won't be the case, in which case you can just do that regardless of what the standard says. The standard generally lets programmers and implementers figure these things out before considering mandating them. – Steve Jessop Sep 13 '15 at 10:09
  • @SteveJessop: I actually made a little test building with `g++ -c -std=c++11 -x c++-header std` a precompiled header `std.gch` containing all the standard includes I could get to precompile (resulting .gch is ~78Mb). After that compiling an "hello world" takes on my PC 160ms when using `#include "std"` and about 50ms when using `#include `. If however I include the headers I most frequently use (vector, map, string, functional, cstdio, unordered_map, sstring, iostream, atomic, thread, mutex) the compilation of the same hello world goes to 550ms. – 6502 Sep 13 '15 at 20:12
  • Your workround for `` isn't. You can include it multiple times in a single translation unit, and it does different things depending on the definition of NDEBUG at the point of inclusion. I think `` would have to be left out of the pre-compiled standard library. – Martin Bonner supports Monica Nov 24 '17 at 08:45
4

First of all, I am afraid that having a prelude is a bit late to the game. Or rather, seeing as preludes are not easily extensible, we have to content ourselves with a very thin one (built-in types...).

As an example, let's say that I have a C++03 program:

#include <boost/unordered_map.hpp>

using namespace std;
using boost::unordered_map;

static unordered_map<int, string> const Symbols = ...;

It all works fine, but suddenly when I migrate to C++11:

error: ambiguous symbol "unordered_map", do you mean:
- std::unordered_map
- boost::unordered_map

Congratulations, you have invented the least backward compatible scheme for growing the standard library (just kidding, whoever uses using namespace std; is to blame...).

Alright, let's not pre-include them, but still bundle the perfect hash table anyway. The performance gain would be worth it, right?

Well, I seriously doubt it. First of all because the Standard Library is tiny compared to most other header files that you include (hint: compare it to Boost). Therefore the performance gain would be... smallish.

Oh, not all programs are big; but the small ones compile fast already (by virtue of being small) and the big ones include much more code than the Standard Library headers so you won't get much mileage out of it.

Note: and yes, I did benchmark the file look-up in a project with "only" a hundred -I directives; the conclusion was that pre-computing the "include path" to "file location" map and feeding it to gcc resulted in a 30% speed-up (after using ccache already). Generating it and keeping it up-to-date was complicated, so we never used it...

But could we at least include a provision that the compiler could do it in the Standard?

As far as I know, it is already included. I cannot remember if there is a specific blurb about it, but the Standard Library is really part of the "implementation" so resolving #include <vector> to an internal hash-map would fall under the as-if rule anyway.

But they could do it, still!

And lose any flexibility. For example Clang can use either the libstdc++ or the libc++ on Linux, and I believe it to be compatible with the Dirkumware's derivative that ships with VC++ (or if not completely, at least greatly).

This is another point of customization: if the Standard library does not fit your needs, or your platforms, by virtue of being mostly treated like any other library you can replace part of most of it with relative ease.

But! But!

#include <stdafx.h>

If you work on Windows, you will recognize it. This is called a pre-compiled header. It must be included first (or all benefits are lost) and in exchange instead of parsing files you are pulling in an efficient binary representation of those parsed files (ie, a serialized AST version, possibly with some type resolution already performed) which saves off maybe 30% to 50% of the work. Yep, this is close to your proposal; this is Computer Science for you, there's always someone else who thought about it first...

Clang and gcc have a similar mechanism; though from what I've heard it can be so painful to use that people prefer the more transparent ccache in practice.

And all of these will come to naught with modules.

This is the true solution to this pre-processing/parsing/type-resolving madness. Because modules are truly isolated (ie, unlike headers, not subject to inclusion order), an efficient binary representation (like pre-compiled headers) can be pre-computed for each and every module you depend on.

This not only means the Standard Library, but all libraries.

Your solution, more flexible, and dressed to the nines!

Matthieu M.
  • 251,718
  • 39
  • 369
  • 642
  • Apologies for the fact that much of what I say is included here. +1 but I think my presentation is usefully different :-) – Steve Jessop Sep 12 '15 at 18:43
  • @SteveJessop: No need to apologize; as a matter of fact I like your answer. Mine was constructed more as a simulated discussion versus an imaginary petulant programmer which I hope makes it diverting but I fully admit is not exactly concise. – Matthieu M. Sep 12 '15 at 18:58
  • @6502: Well, your edit substantially clarified your rant question, but I still think I answered it in the very first part of my answer. To re-iterate: languages with a prelude cannot easily extend said prelude with time, and as far as I know ALL languages have the issue that you have to know where to look. That's generally where IDE or good compilers step in; use whatever you wish for, they'll tell you which file/module include. – Matthieu M. Sep 13 '15 at 10:55
  • @6502: I do feel slightly concerned about your attitude though. (1) Please keep a civil tongue with you, we are after all trying to help you, (2) Avoid judging projects you have never seen. – Matthieu M. Sep 13 '15 at 10:58
3

I think the answer comes down to C++'s philosophy of not making you pay for what you don't use. It also gives you more flexibility: you aren't forced to use parts of the standard library if you don't need them. And then there's the fact that some platforms might not support things like throwing exceptions or dynamically allocating memory (like the processors used in the Arduino, for example). And there's one other thing you said that is incorrect. As long as it's not a template class, you are allowed to add swap operators to the std namespace for your own classes.

  • C++ is different from C and basically you cannot for example avoid exceptions (the program "int main(){ new char[1000]; }" is valid C++, doesn't use any `#include` and can throw exceptions). Free store allocation is also part of the core language. Containers would be absent from a freestanding implementation just because of implementers being lazy (they don't use anything else). Pre-including the standard library (or implementation-defined parts of it in a freestanding implementation) would still be possible. – 6502 Sep 13 '15 at 07:19
  • Now that you mention it, the AVR chips I mentioned do support throwing exceptions. However, the overhead for doing so was very bad. But as far as dynamic memory allocation, those chips do not support that. (Or at least, they certainly didn't a few years ago.). So apparent `malloc`-ing and such was implemented by reserving (maximum) memory areas *on the stack*. (And `free` et al. are NOPs). –  Sep 13 '15 at 09:04
3

One could use an alternative implementation of the C++ Standard Library to the one shipped with the compiler. Or wrap headers with one's definitions, to add, enable or disable features (see GNU wrapper headers). Plain text headers and the C inclusion model are a more powerful and flexible mechanism than a binary black box.

Liviu Nicoara
  • 83
  • 4
  • The standard doesn't consider the possibility of replacing parts of the standard library. Actually programs are forbidden to step over any standard name. Implementations may provide this kind of facility but this would remain to be true even with a pre-included standard library (e.g. providing specific tools to package the prologue). – 6502 Sep 13 '15 at 07:04
  • I am not talking about replacing parts. Portable, third-party implementations of the C++ Standard Library exist and are/were in large use for decades (see Apache stdcxx, STLPort, etc.). Wrapper headers are a well-known feature with at least two compilers, gcc and clang. Both are possible and at the same time enable uses that would otherwise be impossible or hard to get with black-box approaches. – Liviu Nicoara Sep 13 '15 at 19:53
  • From a **formal** point of view there is no way to write a "third-party implementation of the C++ standard library". Of course most C++ compilers are indeed using regular files for standard includes... but as far the standard goes there could be a compiler for which you cannot replace the implementation of `` because this implementation is hardwired in the compiler and no include file actually exists. – 6502 Sep 13 '15 at 20:04
  • And in the meantime such implementations exist. The standard can be read to mean that the headers are an abstract concept, possibly without any permanent storage. However, it is quite clear why that would be a stretch, given the language history. – Liviu Nicoara Sep 13 '15 at 21:00