C++: Is there a standard definition for end-of-line in a multi-line string constant?

Question

If I have a multi-line string C++11 string constant such as

R"""line 1
line 2
line3"""

Is it defined what character(s) the line terminator/separator consist of?

Answer 1

The intent is that a newline in a raw string literal maps to a single '\n' character. This intent is not expressed as clearly as it should be, which has led to some confusion.

Citations are to the 2011 ISO C++ standard.

First, here's the evidence that it maps to a single '\n' character.

A note in section 2.14.5 [lex.string] paragraph 4 says:

[ Note: A source-file new-line in a raw string literal results in a new-line in the resulting execution string-literal. Assuming no whitespace at the beginning of lines in the following example, the assert will succeed:

    const char *p = R"(a\
    b
    c)";
    assert(std::strcmp(p, "a\\\nb\nc") == 0);

— end note ]

This clearly states that a newline is mapped to a single '\n' character. It also matches the observed behavior of g++ 6.2.0 and clang++ 3.8.1 (tests done on a Linux system using source files with Unix-style and Windows-style line endings).

Given the clearly stated intent in the note and the behavior of two popular compilers, I'd say it's safe to rely on this -- though it would be interesting to see how other compilers actually handle this.

However, a literal reading of the normative wording of the standard could easily lead to a different conclusion, or at least to some uncertainty.

Section 2.5 [lex.pptoken] paragraph 3 says (emphasis added):

Between the initial and final double quote characters of the raw string, any transformations performed in phases 1 and 2 (trigraphs, universal-character-names, and line splicing) are reverted; this reversion shall apply before any d-char, r-char, or delimiting parenthesis is identified.

The phases of translation are specified in 2.2 [lex.phases]. In phase 1:

Physical source file characters are mapped, in an implementation-defined manner, to the basic source character set (introducing new-line characters for end-of-line indicators) if necessary.

If we assume that the mapping of physical source file characters to the basic character set and the introduction of new-line characters are "tranformations", we might reasonably conclude that, for example, a newline in the middle of a raw string literal in a Windows-format source file should be equivalent to a \r\n sequence. (I can imagine that being useful for Windows-specific code.)

(This interpretation does lead to problems with systems where the end-of-line indicator is not a sequence of characters, for example where each line is a fixed-width record. Such systems are rare these days.)

As "Cheers and hth. - Alf"'s answer points out, there is an open Defect Report for this issue. It was submitted in 2013 and has not yet been resolved.

Personally, I think the root of the confusion is the word "any" (emphasis added as before):

Between the initial and final double quote characters of the raw string, any transformations performed in phases 1 and 2 (trigraphs, universal-character-names, and line splicing) are reverted; this reversion shall apply before any d-char, r-char, or delimiting parenthesis is identified.

Surely the mapping of physical source file characters to the basic source character set can reasonably be thought of as a transformation. The parenthesized clause "(trigraphs, universal-character-names, and line splicing)" seems to be intended to specify which transformations are to be reverted, but that either attempts to change the meaning of the word "transformations" (which the standard does not formally define) or contradicts the use of the word "any".

I suggest that changing the word "any" to "certain" would express the apparent intent much more clearly:

Between the initial and final double quote characters of the raw string, certain transformations performed in phases 1 and 2 (trigraphs, universal-character-names, and line splicing) are reverted; this reversion shall apply before any d-char, r-char, or delimiting parenthesis is identified.

This wording would make it much clearer that "trigraphs, universal-character-names, and line splicing" are the only transformations that are to be reverted. (Not everything done in translation phases 1 and 2 is reverted, just those specific listed transformations.)

Answer 2

The standard seems to indicate that:

R"""line 1
line 2
line3"""

is equivalent to:

"line 1\nline 2\nline3"

From 2.14.5 String literals of the C++11 standard:

4 [ Note: A source-file new-line in a raw string literal results in a new-line in the resulting execution string literal. Assuming no whitespace at the beginning of lines in the following example, the assert will succeed:

const char *p = R"(a\
b
c)";
assert(std::strcmp(p, "a\\\nb\nc") == 0);

—end note ]

5 [ Example: The raw string

R"a(
)\
a"
)a"

is equivalent to "\n)\\\na\"\n".

Answer 3

^{Note: the question has changed substantially since the answers were posted. Only half of it remains, namely the pure C++ aspect. The network focus in this answer addresses the original question's “sending a multi-line string to a server with well-defined end-of-line requirements”. I do not chase question evolution in general.}

Internally in the program, the C++ standard for newline is \n. This is used also for newline in a raw literal. There is no special convention for raw literals.

Usually \n maps to ASCII linefeed, which is the value 10.

I'm not sure what it maps to in EBCDIC, but you can check that if needed.

On the wire, however, it's my impression that most protocols use ASCII carriage return plus linefeed, i.e. 13 followed by 10. This is sometimes called CRLF, after the ASCII abbreviations CR for carriage return and LF for linefeed. When the C++ escapes are mapped to ASCII this is simply \r\n in C++.

You need to abide by the requirements of the protocol you're using.

For ordinary file/stream i/o the C++ standard library takes care of mapping the internal \n to whatever convention the host environment uses. This is called text mode, as opposed to binary mode where no mapping is performed.

For network i/o, which is not covered by the standard library, the application code must do this itself, either directly or via some library functions.

---

There is an active issue about this, core language defect report #1655 “Line endings in raw string literals”, submitted by Mike Miller 2013-04-26, where he asks,

” is it intended that, for example, a CRLF in the source of a raw string literal is to be represented as a newline character or as the original characters?

Since line ending values differ depending on the encoding of the original file, and considering that in some file systems there is not an encoding of line endings, but instead lines as records, it's clear that the intention is not to represent the file contents as-is – since that's impossible to do in all cases. But as far as I can see this DR is not yet resolved.