2128

I'm trying to replace each , in the current file by a new line:

:%s/,/\n/g 

But it inserts what looks like a ^@ instead of an actual newline. The file is not in DOS mode or anything.

What should I do?

If you are curious, like me, check the question Why is \r a newline for Vim? as well.

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
Vinko Vrsalovic
  • 244,143
  • 49
  • 315
  • 361
  • 2
    Stack Overflow is a site for programming and development questions. This question appears to be off-topic because it is not about programming or development. See [What topics can I ask about here](http://stackoverflow.com/help/on-topic) in the Help Center. Perhaps [Super User](http://superuser.com/) or [Unix & Linux Stack Exchange](http://unix.stackexchange.com/) would be a better place to ask. – jww May 20 '18 at 04:00
  • 5
    @jww this question is 10 years old... seems to be too old to migrate. There are lots of questions like this, for example: https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl/ – Vinko Vrsalovic May 20 '18 at 07:46
  • 25
    @jww vim is a tool commonly used for programmers, and questions about tools commonly used for programmers are on topic in [so]. Although obviously this is more suitable on [vim.se]. – user202729 Aug 26 '18 at 15:18

11 Answers11

2776

Use \r instead of \n.

Substituting by \n inserts a null character into the text. To get a newline, use \r. When searching for a newline, you’d still use \n, however. This asymmetry is due to the fact that \n and \r do slightly different things:

\n matches an end of line (newline), whereas \r matches a carriage return. On the other hand, in substitutions \n inserts a null character whereas \r inserts a newline (more precisely, it’s treated as the input CR). Here’s a small, non-interactive example to illustrate this, using the Vim command line feature (in other words, you can copy and paste the following into a terminal to run it). xxd shows a hexdump of the resulting file.

echo bar > test
(echo 'Before:'; xxd test) > output.txt
vim test '+s/b/\n/' '+s/a/\r/' +wq
(echo 'After:'; xxd test) >> output.txt
more output.txt
Before:
0000000: 6261 720a                                bar.
After:
0000000: 000a 720a                                ..r.

In other words, \n has inserted the byte 0x00 into the text; \r has inserted the byte 0x0a.

Konrad Rudolph
  • 482,603
  • 120
  • 884
  • 1,141
  • 138
    /r is treated as pressing the Enter/Return key. It works on all platforms. – Luka Marinko Oct 12 '08 at 11:41
  • 1
    for whatever reason, replacing all '\n with ',\n works when doing: %s/'\n/',\r/g – tipu Oct 27 '11 at 20:20
  • 78
    See also [Why is \r a newline for Vim?](http://stackoverflow.com/questions/71417/why-is-r-a-newline-for-vim). – Andrew Marshall Apr 05 '12 at 06:14
  • 16
    I wish this worked for classic vi. On AIX v6.1, `\r` doesn't work like this. But you can press `Ctrl-V` `Enter` in place of typing `\r`, and it works. – eksortso Apr 26 '13 at 19:52
  • Even this works Type V followed by M holding control key which looks like this ^M which can be used instead of \r or \n – Nagaraju Aug 20 '14 at 10:10
  • 5
    I'm late to the party. If `\r` inserts `` and `\n` inserts a null, how would I replace something with a carriage return? – Mr. Llama Jun 03 '15 at 21:18
  • 1
    @KonradRudolph: Ages ago when people were using teletypes, before computers, when you reach the end of the line it took the time of inserting two characters for the print head to reach from the far right to the left to begin a new line. So if any character came when the head was moving from right to left, it was lost. To solve this problem people inserted a Carriage Return (CR) and a Line Feed (LF). After the speed of the printhead movement was reduced, different people took different routes- unix used LF, DOS used CR and LF, Apple used CR. – 0x0 May 15 '16 at 02:21
  • So, to maintain compatibility, DOS removes CR when the file is opened as ASCII, and has both when opened as binary. I'm not sure why this is a problem in unix, but this probably also has some quirks like that. – 0x0 May 15 '16 at 02:23
  • @Sunil I’m aware of the history of carriage return and line feed in different systems; however, it doesn’t explain the asymmetry between the meaning of `\r` and `\n` in searching and replacing. – Konrad Rudolph May 16 '16 at 09:10
  • It is interesting that to go back I would have to do: `%s/\n/,/g`, not `%s/\r/,/g`. – Akavall Jan 18 '17 at 23:32
  • 1
    This answer has really come along way since the beginning https://stackoverflow.com/revisions/71334/1 – LangeHaare Oct 24 '17 at 09:44
  • 3
    @SunnyRaj are you sure about the history of CR and LF? I was under impression that originally LF moved paper one row forward but did not move the printhead, and CR moved the printhead but did not move the paper. As a result, if your OS did not convert the input before printing, you could not just use just LF nor CR to get the correct output. MS DOS used raw printer data as the text file format, Mac OS used CR and converted from that to printer's raw format and UNIX used LF and converted from that to printer's raw format. – Mikko Rantalainen Dec 21 '17 at 07:46
  • 1
    I had a long line containing a lot of explicit '\n' which I wanted to expand to a line each. after reading this thread, `%s/\\n/\r/g` did the trick! (notice the extra \, otherwise it searches for actual newlines) – tormodatt Mar 22 '18 at 12:46
  • Actually, I always do something like: `:%s/text/^M/g`, where I just press `^V` and Enter for the second part. Hope that help someone – Chromium May 25 '18 at 02:29
  • 1
    @Chromium I'd always done it like that too, but I'm glad to learn this new way, which I think is better. The advantage is it can be described in a recipe, which is more-or-less copy-pastable directly into vim, without having to explain the "where I just press `^V` and Enter for the second part" part. – Don Hatch Oct 27 '19 at 04:18
219

Here's the trick:

First, set your Vi(m) session to allow pattern matching with special characters (i.e.: newline). It's probably worth putting this line in your .vimrc or .exrc file:

:set magic

Next, do:

:s/,/,^M/g

To get the ^M character, type Ctrl + V and hit Enter. Under Windows, do Ctrl + Q, Enter. The only way I can remember these is by remembering how little sense they make:

A: What would be the worst control-character to use to represent a newline?

B: Either q (because it usually means "Quit") or v because it would be so easy to type Ctrl + C by mistake and kill the editor.

A: Make it so.

Community
  • 1
  • 1
Logan
  • 3,931
  • 3
  • 18
  • 33
  • 9
    I'm using GVim on Windows, and I need neither the `:set magic` (it's not in my ~/_vimrc either) or `ctrl-q`. Just a simple `ctrl-v` followed by enter creates the `^M` character for me just fine. – Chris Phillips Sep 14 '11 at 21:02
  • 6
    C-v doesn't represent a newline; it's the "escape next literal character" command. I dunno what C-v is a mnemonic for either, but there's a reason it doesn't mentally map to newline. – Jim Stewart Jun 22 '12 at 21:26
  • 29
    Ctrl-v is a mnemonic for "verbatim" - i.e. escape next key pressed to its "verbatim" keycode/character. In Windows it's paste: to keep things familiar. Ctrl-Q is for "(un)Quote" maybe. Quite stupid, anyway - but you can use it in binary files - e.g. to search for Ctrl-A through Ctrl-Z (Ascii 1-26 I guess). – Tomasz Gandor Nov 06 '13 at 09:49
  • 1
    `Ctrl-C` doesn't actually kill the editor, although it can cancel you back to Normal mode. `Ctrl-V` means verbatim, and `Ctrl-Q` means that someone made the mistake of loading the `$VIMRUNTIME/mswin.vim` configuration file. You don't need mswin. Just use your own vimrc instead. – 00dani Oct 12 '16 at 04:28
  • wow----this is amazing. I used to do `sed -n l ` till now. Good to know that the same can be achieved with `Ctrl-v` in vim. – arpit May 15 '20 at 04:35
106

In the syntax s/foo/bar, \r and \n have different meanings, depending on context.


Short:

For foo:

\r == "carriage return" (CR / ^M)
\n == matches "line feed" (LF) on Linux/Mac, and CRLF on Windows

For bar:

\r == produces LF on Linux/Mac, CRLF on Windows
\n == "null byte" (NUL / ^@)

When editing files in linux (i.e. on a webserver) that were initially created in a windows environment and uploaded (i.e. FTP/SFTP) - all the ^M's you see in vim, are the CR's which linux does not translate as it uses only LF's to depict a line break.


Longer (with ASCII numbers):

NUL == 0x00 == 0 == Ctrl + @ == ^@ shown in vim
LF == 0x0A == 10 == Ctrl + J
CR == 0x0D == 13 == Ctrl + M == ^M shown in vim

Here is a list of the ASCII control characters. Insert them in Vim via Ctrl + V,Ctrl + ---key---.

In Bash or the other Unix/Linux shells, just type Ctrl + ---key---.

Try Ctrl + M in Bash. It's the same as hitting Enter, as the shell realizes what is meant, even though Linux systems use line feeds for line delimiting.

To insert literal's in bash, prepending them with Ctrl + V will also work.

Try in Bash:

echo ^[[33;1mcolored.^[[0mnot colored.

This uses ANSI escape sequences. Insert the two ^['s via Ctrl + V, Esc.

You might also try Ctrl + V,Ctrl + M, Enter, which will give you this:

bash: $'\r': command not found

Remember the \r from above? :>

This ASCII control characters list is different from a complete ASCII symbol table, in that the control characters, which are inserted into a console/pseudoterminal/Vim via the Ctrl key (haha), can be found there.

Whereas in C and most other languages, you usually use the octal codes to represent these 'characters'.

If you really want to know where all this comes from: The TTY demystified. This is the best link you will come across about this topic, but beware: There be dragons.


TL;DR

Usually foo = \n, and bar = \r.

sjas
  • 15,508
  • 11
  • 75
  • 80
  • 1
    So I'm intrigued how you would substitute a character with a carriage return – codeshot Apr 07 '15 at 22:25
  • 2
    @codeshot `:s/x/^M/g` should do. Insert the `^M` via `ctrl-v` followed by `ctrl-m`. – sjas Apr 08 '15 at 08:25
  • 3
    Thanks sjas, You know this question is one of the weirdest of all time. 1008 votes for the answer which basically says nothing more than "vim does what you found. That's because vim does what you found. Never forget that vim does what you found." I'd hoped to find a shortlist of codes for interesting characters in the pattern, the replacement and the reason for the weirdness so its easy to remember and predict other similar weirdness. That would have got my vote. – codeshot Apr 08 '15 at 11:32
  • @codeshot a list of ascii control characters might help you. See http://www.cs.tut.fi/~jkorpela/chars/c0.html for further reference. I will update my answer to include two links. – sjas Apr 08 '15 at 14:11
60

You need to use:

:%s/,/^M/g

To get the ^M character, press Ctrl + v followed by Enter.

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
dogbane
  • 242,394
  • 72
  • 372
  • 395
43

\r can do the work here for you.

Arslan Ali
  • 16,294
  • 7
  • 51
  • 65
Lasar
  • 4,525
  • 4
  • 22
  • 22
25

With Vim on Windows, use Ctrl + Q in place of Ctrl + V.

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
grantc
  • 1,693
  • 12
  • 13
17

This is the best answer for the way I think, but it would have been nicer in a table:

Why is \r a newline for Vim?

So, rewording:

You need to use \r to use a line feed (ASCII 0x0A, the Unix newline) in a regex replacement, but that is peculiar to the replacement - you should normally continue to expect to use \n for line feed and \r for carriage return.

This is because Vim used \n in a replacement to mean the NIL character (ASCII 0x00). You might have expected NIL to have been \0 instead, freeing \n for its usual use for line feed, but \0 already has a meaning in regex replacements, so it was shifted to \n. Hence then going further to also shift the newline from \n to \r (which in a regex pattern is the carriage return character, ASCII 0x0D).

Character                | ASCII code | C representation | Regex match | Regex replacement
-------------------------+------------+------------------+-------------+------------------------
nil                      | 0x00       | \0               | \0          | \n
line feed (Unix newline) | 0x0a       | \n               | \n          | \r
carriage return          | 0x0d       | \r               | \r          | <unknown>

NB: ^M (Ctrl + V Ctrl + M on Linux) inserts a newline when used in a regex replacement rather than a carriage return as others have advised (I just tried it).

Also note that Vim will translate the line feed character when it saves to file based on its file format settings and that might confuse matters.

Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
codeshot
  • 1,155
  • 1
  • 9
  • 20
11

From Eclipse, the ^M characters can be embedded in a line, and you want to convert them to newlines.

:s/\r/\r/g
Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
rickfoosusa
  • 949
  • 16
  • 22
9

But if one has to substitute, then the following thing works:

:%s/\n/\r\|\-\r/g

In the above, every next line is substituted with next line, and then |- and again a new line. This is used in wiki tables.

If the text is as follows:

line1
line2
line3

It is changed to

line1
|-
line2
|-
line3
Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
Kiran K Telukunta
  • 1,808
  • 2
  • 20
  • 19
5

Here's the answer that worked for me. From this guy:

----quoting Use the vi editor to insert a newline char in replace


Something else I have to do and cannot remember and then have to look up.

In vi, to insert a newline character in a search and replace, do the following:

:%s/look_for/replace_with^M/g

The command above would replace all instances of “look_for” with “replace_with\n” (with \n meaning newline).

To get the “^M”, enter the key combination Ctrl + V, and then after that (release all keys) press the Enter key.


Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
5

If you need to do it for a whole file, it was also suggested to me that you could try from the command line:

sed 's/\\n/\n/g' file > newfile
Peter Mortensen
  • 28,342
  • 21
  • 95
  • 123
Evan Donovan
  • 706
  • 9
  • 18
  • 1
    Note that this requires GNU sed. Try `printf 'foo\\nbar\n' | sed 's/\\n/\n/g'` to see if it will work on your system. (Credit to the good people of #bash on freenode for this suggestion.) – Evan Donovan Feb 09 '12 at 23:02
  • Yes, but the question was about Vim. There is Stack Overflow question *[How can I replace a newline (\n) using sed?](https://stackoverflow.com/questions/1251999)*. – Peter Mortensen Nov 15 '19 at 10:04