72

I have a directory, 'Dst Directory', which has files and folders in it and I have 'src Directory' which also has files and folders in it. What I want to do is move the contents of 'src Directory' to 'Dst Directory' and overwrite anyfiles that exist with the same name. So for example 'Src Directory\file.txt' needs to be moved to 'Dst Directory\' and overwrite the existing file.txt. The same applies for some folders, moving a folder and merging the contents with the same folder in 'dst directory'

I'm currently using shutil.move to move the contents of src to dst but it won't do it if the files already exist and it won't merge folders; it'll just put the folder inside the existing folder.

Update: To make things a bit clearer, what I'm doing is unzipping an archive to the Dst Directory and then moving the contents of Src Directory there and rezipping, effectively updating files in the zip archive. This will be repeated for adding new files or new versions of files etc which is why it needs to overwrite and merge

Solved: I solved my problem by using distutils.dir_util.copy_tree(src, dst), this copies the folders and files from src directory to dst directory and overwrites/merges where neccesary. Hope that helps some people!

Hope that makes sense, thanks!

iliketocode
  • 6,652
  • 4
  • 41
  • 57
Artharos
  • 847
  • 1
  • 7
  • 9
  • Note that [`distutils.dir_util.copy_tree`](https://docs.python.org/dev/distutils/apiref.html#distutils.dir_util.copy_tree) is not able to copy special files, e.g. [named pipes](https://en.wikipedia.org/wiki/Named_pipe) (throws `distutils.errors.DistutilsFileError`). – patryk.beza Aug 05 '17 at 09:16

7 Answers7

61

This will go through the source directory, create any directories that do not already exist in destination directory, and move files from source to the destination directory:

import os
import shutil

root_src_dir = 'Src Directory\\'
root_dst_dir = 'Dst Directory\\'

for src_dir, dirs, files in os.walk(root_src_dir):
    dst_dir = src_dir.replace(root_src_dir, root_dst_dir, 1)
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
    for file_ in files:
        src_file = os.path.join(src_dir, file_)
        dst_file = os.path.join(dst_dir, file_)
        if os.path.exists(dst_file):
            # in case of the src and dst are the same file
            if os.path.samefile(src_file, dst_file):
                continue
            os.remove(dst_file)
        shutil.move(src_file, dst_dir)

Any pre-existing files will be removed first (via os.remove) before being replace by the corresponding source file. Any files or directories that already exist in the destination but not in the source will remain untouched.

vacing
  • 383
  • 2
  • 8
Ray
  • 169,974
  • 95
  • 213
  • 200
  • 3
    That is nice, thanks for that! I think that's what Brandon Craig Rhodes was talking about, but thanks for providing a snippet! Unfortunately you can't have two correct answers ^^ – Artharos Sep 14 '11 at 17:52
  • 2
    And to copy them over is as easy as replacing "shutil.move" with "shutil.copy". – Karoh Nov 05 '12 at 17:47
  • 1
    If root_dst_dir == 'dir1\\dir2\\', os.mkdir(dst_dir) would throw an error saying "No such file or directory: 'dir1\\dir2\\'". Use os.makedirs(dst_dir) can avoid this problem. – Brian Aug 11 '14 at 06:48
  • This is a problem: `src_dir.replace(root_src_dir, root_dst_dir)`. Consider some directory 'a' being merged into directory 'b'. If there is a subdirectory `a/hahaha`, it will be moved to `b/hbhbhb`. You need `src_dir.replace(root_src_dir, root_dst_dir, 1)` to ensure you only do the first replacement. – cgmb Oct 03 '15 at 07:32
  • 1
    I've edited the answer to include the fixes pointed out by @lyen and myself. This solution has worked quite well for me. Thanks! – cgmb Oct 29 '15 at 02:06
  • 1
    How to delete the empty directories in the root_src_dir? I get Errorno 16 when using rmdir – Ram May 08 '20 at 01:26
53

Use copy() instead, which is willing to overwrite destination files. If you then want the first tree to go away, just rmtree() it separately once you are done iterating over it.

http://docs.python.org/library/shutil.html#shutil.copy

http://docs.python.org/library/shutil.html#shutil.rmtree

Update:

Do an os.walk() over the source tree. For each directory, check if it exists on the destination side, and os.makedirs() it if it is missing. For each file, simply shutil.copy() and the file will be created or overwritten, whichever is appropriate.

Brandon Rhodes
  • 69,820
  • 15
  • 101
  • 136
  • 1
    copy() can't copy folders though, can it? – Artharos Sep 14 '11 at 17:04
  • No, but `move()` on each file also does not create the destination directories, so I assumed that your code already did an `os.makedirs()` on destination folders that did not exist. Ah! I think I understand now — you were doing a `move()` on the *whole* tree at once? Gotchya. Will update my answer. – Brandon Rhodes Sep 14 '11 at 17:16
  • Thanks for the update, the problem with that is that the files to copy are always changing (new files added etc.) so I'd have to update the code every time I added new files for moving, if you understand that. Anyway, I managed it with distutils.dir_util.copy_tree(src, dst) which copys the folders and files and overwrites/merges where neccesary, thanks for the help – Artharos Sep 14 '11 at 17:22
  • `os.walk()` gives you a fresh listing of the source files each time it runs, so it's okay if the list changes each time, if I understand your question. But good luck with the `distutils` solution, though — it's always an adventure when someone depends on `distutils` rather than the stdlib! I suspect they'll keep that function working, though. – Brandon Rhodes Sep 14 '11 at 17:26
  • Oh I see, I didn't really understand the os.walk method, I assumed you'd have to define the files to copy with shutil.copy. I understand it now, haven't been using os.walk for very long. I'll correct answer this then as I can see it works but I'll stick with distutils for now becuase of the simplicity – Artharos Sep 14 '11 at 17:30
9

Since none of the above worked for me, so I wrote my own recursive function. Call Function copyTree(dir1, dir2) to merge directories. Run on multi-platforms Linux and Windows. 

def forceMergeFlatDir(srcDir, dstDir):
    if not os.path.exists(dstDir):
        os.makedirs(dstDir)
    for item in os.listdir(srcDir):
        srcFile = os.path.join(srcDir, item)
        dstFile = os.path.join(dstDir, item)
        forceCopyFile(srcFile, dstFile)

def forceCopyFile (sfile, dfile):
    if os.path.isfile(sfile):
        shutil.copy2(sfile, dfile)

def isAFlatDir(sDir):
    for item in os.listdir(sDir):
        sItem = os.path.join(sDir, item)
        if os.path.isdir(sItem):
            return False
    return True


def copyTree(src, dst):
    for item in os.listdir(src):
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if os.path.isfile(s):
            if not os.path.exists(dst):
                os.makedirs(dst)
            forceCopyFile(s,d)
        if os.path.isdir(s):
            isRecursive = not isAFlatDir(s)
            if isRecursive:
                copyTree(s, d)
            else:
                forceMergeFlatDir(s, d)
ALLOY
  • 91
  • 1
  • 2
  • What's scenarios didn't work for you when using other answers? – cgmb Oct 03 '15 at 03:50
  • 1
    Worth noting, if src has a file with the same name as a directory in dst, this solution will put the file inside the directory that shares its name, while [Ray Vega's solution](http://stackoverflow.com/a/7420617/331041) will throw `OSError: [Errno 21] Is a directory`. – cgmb Oct 03 '15 at 06:50
  • This worked perfect. No OSError 10/12/16/21 ... I got so many errors trying the sys.move(). Thank you . – Ram May 05 '20 at 17:37
4

If you also need to overwrite files with read only flag use this:

def copyDirTree(root_src_dir,root_dst_dir):
"""
Copy directory tree. Overwrites also read only files.
:param root_src_dir: source directory
:param root_dst_dir:  destination directory
"""
for src_dir, dirs, files in os.walk(root_src_dir):
    dst_dir = src_dir.replace(root_src_dir, root_dst_dir, 1)
    if not os.path.exists(dst_dir):
        os.makedirs(dst_dir)
    for file_ in files:
        src_file = os.path.join(src_dir, file_)
        dst_file = os.path.join(dst_dir, file_)
        if os.path.exists(dst_file):
            try:
                os.remove(dst_file)
            except PermissionError as exc:
                os.chmod(dst_file, stat.S_IWUSR)
                os.remove(dst_file)

        shutil.copy(src_file, dst_dir)
Vit Bernatik
  • 2,978
  • 1
  • 25
  • 37
1

I had a similar problem. I wanted to move files and folder structures and overwrite existing files, but not delete anything which is in the destination folder structure.

I solved it by using os.walk(), recursively calling my function and using shutil.move() on files which I wanted to overwrite and folders which did not exist.

It works like shutil.move(), but with the benefit that existing files are only overwritten, but not deleted.

import os
import shutil

def moverecursively(source_folder, destination_folder):
    basename = os.path.basename(source_folder)
    dest_dir = os.path.join(destination_folder, basename)
    if not os.path.exists(dest_dir):
        shutil.move(source_folder, destination_folder)
    else:
        dst_path = os.path.join(destination_folder, basename)
        for root, dirs, files in os.walk(source_folder):
            for item in files:
                src_path = os.path.join(root, item)
                if os.path.exists(dst_file):
                    os.remove(dst_file)
                shutil.move(src_path, dst_path)
            for item in dirs:
                src_path = os.path.join(root, item)
                moverecursively(src_path, dst_path)
the
  • 323
  • 3
  • 8
1

You can use this to copy directory overwriting existing files:

import shutil
shutil.copytree("src", "dst", dirs_exist_ok=True)

dirs_exist_ok argument was added in Python 3.8.

See docs: https://docs.python.org/3/library/shutil.html#shutil.copytree

ArtemSBulgakov
  • 514
  • 3
  • 17
1

Have a look at: os.remove to remove existing files.

phimuemue
  • 30,699
  • 8
  • 74
  • 109
  • Problem with that is that the files I want to add to the folder will change (new ones will be added and old ones updated) so I can't have a set list of what to remove, thanks though – Artharos Sep 14 '11 at 16:42