'^M' character at end of lines

Question

When I run a particular SQL script in Unix environments, I'm am seeing a '^M' character at the end of each line of the SQL script as it is echoed to the command-line. I don't know on which OS the SQL script was originally created.

What is causing this and how do I fix it?

score 80 · Accepted Answer · answered Sep 15 '08 at 17:07

80

It's caused by the DOS/Windows line-ending characters. Like Andy Whitfield said, the Unix command dos2unix will help fix the problem. If you want more information, you can read the man pages for that command.

answered Sep 15 '08 at 17:07

Thomas Owens

107,741
94
299
427

3

On some systems (i.e. Ubuntu) the name of this command is "fromdos" – bobwienholt Dec 04 '12 at 16:12
6

You can get the tool on OSX very easy with `brew install dos2unix` when you have homebrew installed – philipp Jun 12 '13 at 17:04

score 76 · Answer 2 · edited Jul 29 '16 at 02:02

76

Fix line endings in vi by running the following:

:set fileformat=unix

:w

edited Jul 29 '16 at 02:02

s g

4,057
6
39
67

answered Sep 16 '08 at 15:52

Tim Abell

8,954
8
70
99

2

This is a brilliant answer. Many thanks. (saved installing dos2unix, a tool I'd probably only use once) – Jamsi Feb 04 '14 at 22:29
3

this doesnt remove the `^M`s for some reason. reference file: `/etc/timidity/fluidr3_gm.cfg`. – phil294 Apr 06 '17 at 01:11

score 39 · Answer 3 · edited Jan 25 '13 at 12:26

39

The cause is the difference between how a Windows-based based OS and a Unix based OS store the end-of-line markers.

Windows based operating systems, thanks to their DOS heritage, store an end-of-line as a pair of characters - 0x0D0A (carriage return + line feed). Unix-based operating systems just use 0x0A (a line feed). The ^M you're seeing is a visual representation of 0x0D (a carriage return).

dos2unix will help with this. You probably also need to adjust the source of the scripts to be 'Unix-friendly'.

edited Jan 25 '13 at 12:26

Andriy M

71,352
17
87
142

answered Sep 15 '08 at 17:11

ColinYounger

6,570
5
28
33

I wouldn't say current versions of Windows have any kind of DOS *heritage*. They still have compatibility restraints, though. – Joey Mar 08 '13 at 12:30
This is the easy way, is you do an automatic conversion tool. Thank's – Pjl Apr 09 '15 at 13:57
But why `^M`? Why the '^'? Why the 'M'? – 1737973 Jul 28 '15 at 04:47
Because it's a "control character". "^" is the visual representation of clicking the control key. Underneath its just specific bytes, the ^ is how the editor represents them. – Hejazzman Aug 05 '15 at 22:10

score 24 · Answer 4 · edited Jan 21 '14 at 17:41

24

The easiest way is to use vi. ~~I know that sounds terrible but~~ its simple and already installed on most UNIX environments. The ^M is a new line from Windows/DOS environment.

from the command prompt: $ vi filename

Then press ":" to get to command mode.

Search and Replace all Globally is :%s/^M//g "Press and hold control then press V then M" which will replace ^M with nothing.

Then to write and quit enter ":wq" Done!

edited Jan 21 '14 at 17:41

Ionică Bizău

93,552
74
254
426

answered Sep 15 '08 at 17:32

Bernie Perez

12,073
13
41
54

How to replace it in emacs? – herbertD Nov 01 '13 at 03:09
3

Thanks for the expanation on how to type the ^M character! I would replace it with \r instead. So I did :%s/^M/\r/g – aharris88 Nov 24 '14 at 20:05

score 13 · Answer 5 · answered Sep 15 '08 at 17:06

13

Try using dos2unix to strip off the ^M.

answered Sep 15 '08 at 17:06

Andy Whitfield

2,295
2
19
22

score 10 · Answer 6 · edited Dec 04 '12 at 16:06

10

In vi, do a :%s/^M//g

To get the ^M hold the CTRL key, press V then M (Both while holding the control key) and the ^M will appear. This will find all occurrences and replace them with nothing.

edited Dec 04 '12 at 16:06

Brad Mace

26,280
15
94
141

answered Sep 15 '08 at 17:10

dogbane

242,394
72
372
395

2

To replace the ^M with a unix friendly line break: `:%s/^M/\r/g` – Gary Oak Feb 26 '14 at 19:46

score 8 · Answer 7 · answered Sep 15 '08 at 17:10

8

The SQL script was originally created on a Windows OS. The '^M' characters are a result of Windows and Unix having different ideas about what to use for an end-of-line character. You can use perl at the command line to fix this.

perl -pie 's/\r//g' filename.txt

answered Sep 15 '08 at 17:10

Bill the Lizard

369,957
201
546
842

Sure, you CAN use perl, but would you suggest perl over dos2unix? – Thomas Owens Sep 15 '08 at 17:11
2

I'm just providing an alternative, since four people already said to use dos2unix. – Bill the Lizard Sep 15 '08 at 17:17
2

Yes, I found this useful because I am on a backward workstation working in an office with a prehistoric IT department. Except I used a variation: perl -pi -e "s/\x0D/\n/g" file.csv – Rimian Feb 10 '10 at 23:36

score 7 · Answer 8 · answered Sep 15 '08 at 17:08

7

The ^M is typically caused by the Windows operator newlines, and translated onto Unix looks like a ^M. The command dos2unix should remove them nicely

dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]

answered Sep 15 '08 at 17:08

jW.

8,860
12
44
50

score 5 · Answer 9 · edited Mar 08 '13 at 12:02

C:\tmp\text>dos2unix hello.txt helloUNIX.txt

Sed is even more widely available and can do this kind of thing also if dos2unix is not installed

C:\tmp\text>sed s/\r// hello.txt > helloUNIX.txt

You could also try tr:

cat hello.txt | tr -d \r > helloUNIX2.txt

Here are the results:

C:\tmp\text>dumphex hello.txt  
00000000h: 48 61 68 61 0D 0A 68 61 68 61 0D 0A 68 61 68 61 Haha..haha..haha  
00000010h: 0D 0A 0D 0A 68 61 68 61 0D 0A                   ....haha..  

C:\tmp\text>dumphex helloUNIX.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.  

C:\tmp\text>dumphex helloUNIX2.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.

score 4 · Answer 10 · answered Apr 17 '13 at 14:34

4

To replace ^M characters in vi editor use below

open the text file say t1.txt

vi t1.txt

Enter command mode by pressing shift + :

then press keys as mentioned %s/^M/\r/g

in above ^M is not (shift + 6)M instead it is (ctrl + V)(ctrl + M)

answered Apr 17 '13 at 14:34

leela

536
4
10

Your last line is what I was missing from all the previous answers. I kept getting 'no matches found bc I was doing shift+6, so I did what every hacker would and circumvented my misunderstanding with my own solution: record a macro to do $ to go to end of each line and then press x, just repeat macro for num of lines in file. – darethas Aug 13 '13 at 05:46

score 3 · Answer 11 · edited Jul 18 '13 at 07:20

3

An alternative to dos2unix command would be using standard utilities like sed.

For example, dos to unix:

sed 's/\r$//' dos.txt > unix.txt

unix to dos:

sed 's/$/\r/' unix.txt > dos.txt

edited Jul 18 '13 at 07:20

MikroDel

6,389
7
37
69

answered Jul 18 '13 at 07:03

g4th

925
9
14

score 1 · Answer 12 · answered Feb 07 '14 at 16:05

1

You can remove ^M from the files directly via sed command, e.g.:

sed -i'.bak' s/\r//g *.*

If you're happy with the changes, remove the .bak files:

rm -v *.bak

answered Feb 07 '14 at 16:05

kenorb

118,428
63
588
624

score 1 · Answer 13 · edited May 23 '17 at 11:47

1

Convert DOS/Windows (\r\n) line endings to Unix (\n) line endings, with tr:

tr '\r\n' '\n' < dosFile.txt > unixFile.txt

Post about replacing newlines from the Unix command line

edited May 23 '17 at 11:47

Community

1
1

answered Jun 03 '14 at 20:10

Mason Wright

1
2

score 0 · Answer 14 · answered Mar 07 '13 at 04:32

0

In Perl, if you don't want to set the $/ variable and use chomp() you can also do:

$var =~ /\r\n//g;

My two cents

answered Mar 07 '13 at 04:32

Ariel Monaco

2,481
1
17
18

score 0 · Answer 15 · answered Sep 15 '08 at 19:54

0

od -a $file is useful to explore those types of question on Linux (similar to dumphex in the above).

answered Sep 15 '08 at 19:54

Allan Wind

7,923
3
20
30

Scottie T · Answer 16 · 2008-09-15T19:55:28.377

-1

Another vi command that'll do: :%s/.$// This removes the last character of each line in the file. The drawback to this search and replace command is that it doesn't care what the last character is, so be careful not to call it twice.

edited Sep 15 '08 at 19:55

answered Sep 15 '08 at 18:24

Scottie T

10,481
9
43
59

Why mention it if you know it's not reliable? – minexew Feb 07 '14 at 16:10

'^M' character at end of lines

16 Answers16

Linked

Related