128

When using os.system() it's often necessary to escape filenames and other arguments passed as parameters to commands. How can I do this? Preferably something that would work on multiple operating systems/shells but in particular for bash.

I'm currently doing the following, but am sure there must be a library function for this, or at least a more elegant/robust/efficient option:

def sh_escape(s):
   return s.replace("(","\\(").replace(")","\\)").replace(" ","\\ ")

os.system("cat %s | grep something | sort > %s" 
          % (sh_escape(in_filename), 
             sh_escape(out_filename)))

Edit: I've accepted the simple answer of using quotes, don't know why I didn't think of that; I guess because I came from Windows where ' and " behave a little differently.

Regarding security, I understand the concern, but, in this case, I'm interested in a quick and easy solution which os.system() provides, and the source of the strings is either not user-generated or at least entered by a trusted user (me).

martineau
  • 99,260
  • 22
  • 139
  • 249
Tom
  • 36,698
  • 31
  • 90
  • 98
  • 6
    This is also useful without os.system, in situations where subprocess isn't even an option; e.g. generating shell scripts. –  Oct 03 '10 at 20:25
  • 2
    Beware of the security issue! For instance if out_filename is foo.txt; rm -rf / The malicious user can add more command directly interpreted by the shell. – Steve Gury Aug 30 '08 at 09:37
  • An ideal `sh_escape` function would escape out the `;` and spaces and remove the security problem by simply creating a file called something like `foo.txt\;\ rm\ -rf\ /`. – Tom Oct 07 '10 at 03:40
  • In almost all cases, you should use subprocess, not os.system. Calling os.system is just asking for an injection attack. – allyourcode Jan 27 '16 at 01:57

8 Answers8

162

shlex.quote() does what you want since python 3.

(Use pipes.quote to support both python 2 and python 3)

pixelbeat
  • 27,785
  • 9
  • 47
  • 57
  • There's also `commands.mkarg`. It also adds a leading space (outside the quotes) which may or may not be desirable.It's interesting how their implementations are quite different from each other, and also much more complicated than Greg Hewgill's answer. – Laurence Gonsalves Oct 04 '10 at 16:47
  • 3
    For some reason, `pipes.quote` is not mentioned by the [standard library documentation for the pipes module](http://docs.python.org/library/pipes.html) – Day Aug 18 '11 at 22:28
  • 1
    Both are undocumented; `command.mkarg` is deprecated and removed in 3.x, while pipes.quote remained. – Beni Cherniavsky-Paskin Sep 18 '11 at 17:01
  • 9
    Correction: officially documented as [`shlex.quote()`](http://docs.python.org/dev/py3k/library/shlex.html#shlex.quote) in 3.3 , `pipes.quote()` retained for compatibility. [http://bugs.python.org/issue9723] – Beni Cherniavsky-Paskin Sep 18 '11 at 17:32
  • 7
    pipes does NOT work on Windows - adds single quotes insted of double quotes. – Nux May 30 '14 at 15:23
  • shlex appears quite limited and doesn't do what Op wants. For example, the string 'hello"there' should have the double-quote escaped, since that's a special character in sh. But shlex.quote() does not escape it, resulting in a malformed shell call. – Cerin Dec 17 '20 at 21:50
90

This is what I use:

def shellquote(s):
    return "'" + s.replace("'", "'\\''") + "'"

The shell will always accept a quoted filename and remove the surrounding quotes before passing it to the program in question. Notably, this avoids problems with filenames that contain spaces or any other kind of nasty shell metacharacter.

Update: If you are using Python 3.3 or later, use shlex.quote instead of rolling your own.

offby1
  • 5,675
  • 25
  • 43
Greg Hewgill
  • 828,234
  • 170
  • 1,097
  • 1,237
  • escaped singles quotes are not valid within single quotes. – pixelbeat May 11 '09 at 12:05
  • 7
    @pixelbeat: which is exactly why he closes his single quotes, adds an escaped literal single quote, and then reopens his single quotes again. – lhunath May 11 '09 at 13:13
  • 4
    While this is hardly the responsibility of the shellquote function, it might be interesting to note that this will still fail if an unquoted backslash appears just before the return value of this function. Morale: make sure you use this in code that you can trust as safe - (such as part of hardcoded commands) - don't append it to other unquoted user input. – lhunath May 11 '09 at 13:16
  • 10
    Note that unless you absolutely need shell features, you should probably be using Jamie's suggestion instead. – lhunath May 11 '09 at 13:17
  • 6
    Something similar to this is now officially available as [shlex.quote](http://docs.python.org/dev/library/shlex.html#shlex.quote). – Janus Troelsen Jun 16 '12 at 21:17
  • 1
    @lhunath completely wrong on both counts. This is the perfect and proper way to escape a word for POSIX shell (note the wording “a word”, the command structure is up to the user, but trivial to do). Passing as list does not help e.g. after an `ssh` command, since you need to escape multiple levels then… – mirabilos Feb 27 '15 at 13:43
  • 3
    The function provided in this answer does a better job of shell quoting than `shlex` or `pipes`. Those python modules erroneously assume that special characters are the only thing which need to be quoted, which means that shell keywords (like `time`, `case` or `while`) will be parsed when that behaviour is not expected. For that reason I would recommend using the single-quoting routine in this answer, because it doesn't try to be "clever" so doesn't have those silly edge cases. – user3035772 Jan 15 '16 at 12:22
  • Something this solution doesn't address is if there's '/' embedded in a filename. The shell will barf on that because it doesn't allow that character in path name components. – Tom Barron Feb 16 '16 at 20:37
  • Also present in python 2.7 - https://docs.python.org/2.7/library/shlex.html – Aditya Dec 19 '17 at 00:32
  • I believe `"'" + s.replace("'", "'\"'\"'") + "'"` works even better because it doesn't require backslash to be interpreted as special before double quote in shell. – Mikko Rantalainen Apr 17 '19 at 14:36
59

Perhaps you have a specific reason for using os.system(). But if not you should probably be using the subprocess module. You can specify the pipes directly and avoid using the shell.

The following is from PEP324:

Replacing shell pipe line
-------------------------

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Larivact
  • 5
  • 4
Jamie
  • 7,022
  • 4
  • 19
  • 15
  • 6
    `subprocess` (especially with `check_call` etc) is often dramatically superior, but there are a few cases where shell escaping is still useful. The main one I'm running into is when I'm having to invoke ssh remote commands. – Craig Ringer May 23 '13 at 02:28
  • @CraigRinger, yup, ssh remoting is what brought me here. :P I wish ssh had something to help here. – Jürgen A. Erhard May 27 '13 at 16:25
  • @JürgenA.Erhard It does seem odd that it doesn't have an --execvp-remote option (or work that way by default). Doing everything through the shell seems clumsy and risky. OTOH, ssh is full of weird quirks, often things done in a narrow view of "security" that causes people to come up with way-more-insecure workarounds. – Craig Ringer May 27 '13 at 23:31
12

Maybe subprocess.list2cmdline is a better shot?

Gary Shi
  • 635
  • 6
  • 13
  • That looks pretty good. Interesting it isn't documented... (in http://docs.python.org/library/subprocess.html at least) – Tom Jun 04 '12 at 23:40
  • 4
    It does not properly escape \: `subprocess.list2cmdline(["'",'',"\\",'"'])` gives `' "" \ \"` – Tino Oct 28 '12 at 15:33
  • It does not escape shell expansion symbols – grep Jan 07 '13 at 23:51
  • Is subprocess.list2cmdline() intended only for Windows? – JS. Feb 25 '16 at 00:10
  • @JS Yes, `list2cmdline` conforms to Windows cmd.exe syntax ([see the function docstring in the Python source code](https://github.com/python/cpython/blob/3.7/Lib/subprocess.py#L516)). `shlex.quote` conforms to Unix bourne shell syntax, however it isn't usually necessary since Unix has good support for directly passing arguments. Windows pretty much requires that you pass a single string with all your arguments (thus the need for proper escaping). – eestrada Sep 25 '19 at 16:41
6

Note that pipes.quote is actually broken in Python 2.5 and Python 3.1 and not safe to use--It doesn't handle zero-length arguments.

>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1  arg3

See Python issue 7476; it has been fixed in Python 2.6 and 3.2 and newer.

Martijn Pieters
  • 889,049
  • 245
  • 3,507
  • 2,997
John Wiseman
  • 2,907
  • 1
  • 19
  • 30
  • 4
    What version of Python are you using? Version 2.6 seems to produce the correct output: mycommand arg1 '' arg3 (Those are two single-quotes together, though the font on Stack Overflow makes that hard to tell!) – Brandon Rhodes Aug 31 '10 at 13:27
3

Notice: This is an answer for Python 2.7.x.

According to the source, pipes.quote() is a way to "Reliably quote a string as a single argument for /bin/sh". (Although it is deprecated since version 2.7 and finally exposed publicly in Python 3.3 as the shlex.quote() function.)

On the other hand, subprocess.list2cmdline() is a way to "Translate a sequence of arguments into a command line string, using the same rules as the MS C runtime".

Here we are, the platform independent way of quoting strings for command lines.

import sys
mswindows = (sys.platform == "win32")

if mswindows:
    from subprocess import list2cmdline
    quote_args = list2cmdline
else:
    # POSIX
    from pipes import quote

    def quote_args(seq):
        return ' '.join(quote(arg) for arg in seq)

Usage:

# Quote a single argument
print quote_args(['my argument'])

# Quote multiple arguments
my_args = ['This', 'is', 'my arguments']
print quote_args(my_args)
Mr Fooz
  • 95,588
  • 5
  • 65
  • 95
Rockallite
  • 14,477
  • 5
  • 51
  • 45
3

I believe that os.system just invokes whatever command shell is configured for the user, so I don't think you can do it in a platform independent way. My command shell could be anything from bash, emacs, ruby, or even quake3. Some of these programs aren't expecting the kind of arguments you are passing to them and even if they did there is no guarantee they do their escaping the same way.

pauldoo
  • 16,381
  • 20
  • 85
  • 112
  • 2
    It's not unreasonable to expect a mostly or fully POSIX-compliant shell (at least everywhere but with Windows, and you know what "shell" you have then, anyway). os.system doesn't use $SHELL, at least not here. –  Oct 03 '10 at 20:26
2

The function I use is:

def quote_argument(argument):
    return '"%s"' % (
        argument
        .replace('\\', '\\\\')
        .replace('"', '\\"')
        .replace('$', '\\$')
        .replace('`', '\\`')
    )

that is: I always enclose the argument in double quotes, and then backslash-quote the only characters special inside double quotes.

tzot
  • 81,264
  • 25
  • 129
  • 197
  • Note that you should use '\\"', '\\$' and '\\`', otherwise the escaping doesn't happen. – JanKanis Jul 08 '14 at 13:14
  • 1
    Additionally, [there are issues with using double quotes in some (weird) locales](http://bugs.python.org/issue22187); the suggested fix uses `pipes.quote` which @JohnWiseman pointed out is also broken. Greg Hewgill’s answer is thus the one to use. (It’s also the one the shells use internally for the regular cases.) – mirabilos Feb 27 '15 at 15:11