How can I run dos2unix on an entire directory?

Question

I have to convert an entire directory using dos2unix. I am not able to figure out how to do this.

score 671 · Accepted Answer · edited Feb 25 '17 at 04:00

671

find . -type f -print0 | xargs -0 dos2unix

Will recursively find all files inside current directory and call for these files dos2unix command

edited Feb 25 '17 at 04:00

therealjumbo

959
1
8
14

answered Aug 13 '12 at 06:56

CyberDem0n

13,057
1
29
21

99

Would break if you had spaces in filename. `find . -type f -print0 | xargs -0 dos2unix` would solve the problem I think. – Noufal Ibrahim Aug 13 '12 at 06:57
1

i gave like this find . -type f -exec dos2unix {} /home/venuk/Desktop/NEO_Src and it gave the error find: missing argument to `-exec' – Vivek Gaur Aug 13 '12 at 07:05
type f = Just 'F'files and not directories for example! – rafa.ferreira Jun 19 '13 at 13:09
@vivekgaur, The folder path doesn't go at the end: it goes immediately after the `find`. In this example, it's the dot (i.e., the current directory). – TRiG Jan 03 '14 at 15:38
1

Note the backslash that escapes the semi-colon ensures the dos2unix commands are separated by semi-colons so that they don't end up mashed together. If you want to run another command afterwards you'll need another semi-colon, so `\;; echo Hello` – Josh May 20 '14 at 11:43
1

Be careful with this command though, it doesn't check whether the file needs the conversion, so even binary files will be "converted", corrupting them. – vguzmanp Dec 16 '14 at 16:15
Would this go into subdirectories as well? – grepsedawk Apr 25 '15 at 15:05
@Pachonk Just tested that and it does. – Spotlight Jun 22 '15 at 23:09
3

@vguzmanp True, the `find` invocation does not do this check (although that would be simple enough to add), but modern `dos2unix` correctly skips binary files. – Kyle Strand Jan 07 '16 at 16:58
@therealjumbo - If you want to use `find` with `-print0` and `xargs`, then you should probably add your own answer. `-print0` is not Posix, so the change could have a negative effect on some users. Considering NickC already provided that variant of the answer, there's little reason to change CyberDem0n's answer. – jww Feb 25 '17 at 01:18
My only problem with this command is that it didn't do the dot files in the directory. Using this on a Chef cookbook, there are a few dot files such as `.kitchen.yml`, or `.gitignore` and those were not picked up. – FilBot3 Apr 17 '17 at 18:49
and to narrow it on a specific extension: `find . -type f -name '*.py' -print0 | xargs -0 dos2unix` – hardmooth May 16 '17 at 06:35
you need to pass the `-i` switch to `dos2unix`, or it will just print the converted files to stdout. – eadmaster Nov 26 '18 at 13:46
1

Warning: If you run this command in a directory containing a `.git` index, your git index will be corrupted. – mbomb007 Mar 18 '20 at 20:34
1

To avoid `.git` index corruption and preserve UTF-8 BOM, I've ended up doing `find . -type f -print0 | xargs -0 dos2unix -ic0 | xargs -0 dos2unix -b`. This command touches only those files (thanks to `-ic0` filtering option), that contain windows line breaks, all other files are skipped. `-b` option is also helpful if your repository contains files with UTF-8 BOM, it preserves the BOM. – Konard Nov 28 '20 at 11:03

score 77 · Answer 2 · answered Mar 20 '14 at 00:23

77

If it's a large directory you may want to consider running with multiple processors:

find . -type f -print0 | xargs -0 -n 1 -P 4 dos2unix

This will pass 1 file at a time, and use 4 processors.

answered Mar 20 '14 at 00:23

nikc

2,198
17
21

1

This method has the advantage, that it continues, even though dos2unix encounters any problems! Something like a "--force" method. Thank you for that! – freeo Oct 10 '14 at 16:29
Wow - just saved me a lot of problem solving time while attempting to convert a Windows developers code additions that had bed added to git incorrectly. Thank you!! – The NetYeti Feb 20 '15 at 00:26
1

Starting a new dos2unix process for each individual file will introduce massively unnecessary overhead. I'd bump that `n` up by an order of magnitude or two (depending on how many files we're talking about here) – JonoCoetzee Apr 23 '15 at 12:24
upvoted for teaching me about xargs -P switch – siliconrockstar Jan 22 '21 at 21:01

Mathias Dolidon · Answer 3 · 2017-07-22T11:41:28.207

26

As I happened to be poorly satisfied by dos2unix, I rolled out my own simple utility. Apart of a few advantages in speed and predictability, the syntax is also a bit simpler :

endlines unix *

And if you want it to go down into subdirectories (skipping hidden dirs and non-text files) :

endlines unix -r .

endlines is available here https://github.com/mdolidon/endlines

edited Jul 22 '17 at 11:41

answered Sep 03 '14 at 10:32

Mathias Dolidon

3,066
17
27

1

This is perfect! The closest one-liner I could get to this is here: https://unix.stackexchange.com/a/365679/112190 – phyatt May 17 '17 at 17:33
2

How does it improve on `dos2unix`? Genuinely curious. – Walf Jul 17 '17 at 05:26
3

1/ Mainly, there are tons of different `dos2unix`, with varying capabilities (some read UTF32 for example, while some don't ; endlines does not). There's only one `endlines`, which capabilities are well known. 2/ liberal on input, not all `dos2unix` are. 3/ efficient file tree exploration, designed to be fast and practical on tens of thousands of files. 4/ runs out of the box on OSX - which is less important now that Brew package exists. – Mathias Dolidon Jul 17 '17 at 11:04
1

In my case it is exactly what I needed thanks a lot. Features that are welcome are the easy way to process a directory tree and the fast execution time. – AlSavi Mar 10 '20 at 17:10

Kyle Strand · Answer 4 · 2016-06-02T18:05:54.060

19

It's probably best to skip hidden files and folders, such as .git. So instead of using find, if your bash version is recent enough or if you're using zsh, just do:

dos2unix **

Note that for Bash, this will require:

shopt -s globstar

....but this is a useful enough feature that you should honestly just put it in your .bashrc anyway.

If you don't want to skip hidden files and folders, but you still don't want to mess with find (and I wouldn't blame you), you can provide a second recursive-glob argument to match only hidden entries:

dos2unix ** **/.*

Note that in both cases, the glob will expand to include directories, so you will see the following warning (potentially many times over): Skipping <dir>, not a regular file.

edited Jun 02 '16 at 18:05

answered Feb 28 '15 at 00:32

Kyle Strand

14,120
3
59
143

1

This didn't work for me. Does the globstar syntax work for dos2unix? I've used globstar elsewhere with success, but couldn't get this to work. I'm using Bash 4.3.11(1) – CloudNinja Jun 02 '16 at 14:25
@NSduToit *NIX style shells don't allow executables to alter the behavior of argument-expansion, because (unlike in Windows) argument-expansion is performed before the executable ever receives the arguments. So the only thing I can think of is that you have some kind of `dos2unix` alias that's affecting how arguments are expanded. What is the output of `type dos2unix` on your system? – Kyle Strand Jun 02 '16 at 15:56
Apologies - seems it works...sort of... Seems the problem I'm encountering has to do with `dos2unix` and not the actual `globstar`. It seems if I use `dos2unix` like that it just blindly ignores files that are hidden (start with '.', eg '.vimrc')......? But the `globstar` itself seems to work - the output of `ls -a **` is as one would expect... – CloudNinja Jun 02 '16 at 16:55
[Script to test](https://justpaste.it/uwc2) --- you'll see the `dos2unix` command never operates on the `.vimrc` file. Not the behavior I expected - any insights appreciated :) – CloudNinja Jun 02 '16 at 16:59
1

@NSduToit There's some confusion here. My answer explicitly states that the *point* of using `**` instead of `find` is to "skip hidden files and folders, such as `.git`". `dos2unix` never sees the hidden files, because `**` does not expand to show them. If you *want* to automatically run `dos2unix` on hidden files and folders, you can use `find` or `dos2unix ** **/.*` The `**/.*` will expand *only* the hidden files and folders, including `.` (the root dir), `..` (the parent dir), and any other hidden entries in the current folder. – Kyle Strand Jun 02 '16 at 17:01
`type dos2unix` returns `dos2unix is hashed (/usr/bin/dos2unix)` – CloudNinja Jun 02 '16 at 17:01
1

(Note that `dos2unix` simply prints `Skipping , not a regular file.` when run on a directory, so running on `..` and `.` is safe.) Additionally, combining `ls` with a glob is *not* a good way to check how the glob is expanded; use `echo` instead: `echo **` will print the arguments that `dos2unix` receives from `dos2unix **`. – Kyle Strand Jun 02 '16 at 17:02
1

Thank you! Makes sense! Okay, I had a totally wrong idea on what `**` implied, and using `echo` makes it clear. – CloudNinja Jun 02 '16 at 17:12
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/113668/discussion-between-kyle-strand-and-ns-du-toit). – Kyle Strand Jun 02 '16 at 18:03
@feeela The entire point of the globstar `**` is to make globs themselves exercise recursive descent. – Kyle Strand May 02 '18 at 15:11

score 11 · Answer 5 · answered Feb 19 '19 at 05:49

A common use case appears to be to standardize line endings for all files committed to a Git repository:

git ls-files | xargs dos2unix

Keep in mind that certain files (e.g. *.sln, *.bat) etc are only used on Windows operating systems and should keep the CRLF ending:

git ls-files '*.sln' '*.bat' | xargs unix2dos

If necessary, use .gitattributes

score 6 · Answer 6 · answered Nov 01 '14 at 22:43

For any Solaris users (am using 5.10, may apply to newer versions too, as well as other unix systems):

dos2unix doesn't default to overwriting the file, it will just print the updated version to stdout, so you will have to specify the source and target, i.e. the same name twice:

find . -type f -exec dos2unix {} {} \;

score 3 · Answer 7 · answered Feb 19 '19 at 14:36

3

I have had the same problem and thanks to the posts here I have solved it. I knew that I have around a hundred files and I needed to run it for *.js files only. find . -type f -name '*.js' -print0 | xargs -0 dos2unix

Thank you all for your help.

answered Feb 19 '19 at 14:36

Strabek

1,755
1
24
34

score 2 · Answer 8 · edited May 14 '18 at 13:54

2

for FILE in /var/www/html/files/*
do
 /usr/bin/dos2unix FILE
done

edited May 14 '18 at 13:54

phuclv

27,258
11
104
360

answered Oct 20 '17 at 15:37

rafapedroche

77
3

1

Welcome to Stack Overflow. Although your solution is a valid solution, it would be great if you could add some explanation to it. You might also consider to reference other answers to justify your answer. Please have a look at [answer] for more information. – kvantour May 14 '18 at 16:04

score 1 · Answer 9 · answered Mar 02 '20 at 16:04

1

I think the simplest way is:

dos2unix $(find . -type f)

answered Mar 02 '20 at 16:04

Sahand

6,046
14
51
105

score 1 · Answer 10 · answered Apr 23 '20 at 03:03

I've googled this like a million times, so my solution is to just put this bash function in your environment.

.bashrc or .profile or whatever

dos2unixd() {
  find $1 -type f -print0 | xargs -0 dos2unix
}

Usage

$ dos2unixd ./somepath

This way you still have the original command dos2unix and it's easy to remember this one dos2unixd.

score -1 · Answer 11 · edited May 14 '18 at 13:54

-1

If there is no sub-directory, you can also take

ls | xargs -I {} dos2unix "{}"

edited May 14 '18 at 13:54

phuclv

27,258
11
104
360

answered Aug 13 '12 at 06:59

Summer_More_More_Tea

11,342
10
45
76

4

If there are no subdirectories, `dos2unix *` is simpler and will actually be more robust than this. (It's generally not recommended to pipe the output of `ls`, because it's a formatting tool and `*` is more reliable for programmatic usage.) – Kyle Strand Dec 09 '16 at 14:55
1

[the reason piping `ls` is not good](https://unix.stackexchange.com/q/128985/44425) – phuclv May 14 '18 at 13:55

How can I run dos2unix on an entire directory?

11 Answers11

Linked