81

I am trying to execute a curl command within a python script.

If I do it in the terminal, it looks like this:

curl -X POST -d  '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://localhost:8080/firewall/rules/0000000000000001

I've seen recommendations to use pycurl, but I couldn't figure out how to apply it to mine.

I tried using:

subprocess.call([
    'curl',
    '-X',
    'POST',
    '-d',
    flow_x,
    'http://localhost:8080/firewall/rules/0000000000000001'
])

and it works, but is there a better way?

OJFord
  • 8,132
  • 7
  • 52
  • 88
Kiran Vemuri
  • 1,763
  • 1
  • 17
  • 37
  • 1
    You don't have to use `cURL` to POST something to a server. `requests` can do so quite easily (as can `urllib`, with a bit more effort) – roippi Sep 23 '14 at 16:50
  • Check this to know more about executing shell cmds in python http://stackoverflow.com/questions/89228/calling-an-external-command-in-python – Sainath Motlakunta Sep 23 '14 at 16:51

7 Answers7

177

Don't!

I know, that's the "answer" nobody wants. But if something's worth doing, it's worth doing right, right?

This seeming like a good idea probably stems from a fairly wide misconception that shell commands such as curl are anything other than programs themselves.

So what you're asking is "how do I run this other program, from within my program, just to make a measly little web request?". That's crazy, there's got to be a better way right?

Uxio's answer works, sure. But it hardly looks very Pythonic, does it? That's a lot of work just for one little request. Python's supposed to be about flying! Anyone writing that is probably wishing they just call'd curl!

it works, but is there a better way?

Yes, there is a better way!

Requests: HTTP for Humans

Things shouldn’t be this way. Not in Python.

Let's GET this page:

import requests
res = requests.get('https://stackoverflow.com/questions/26000336')

That's it, really! You then have the raw res.text, or res.json() output, the res.headers, etc.

You can see the docs (linked above) for details of setting all the options, since I imagine OP has moved on by now, and you - the reader now - likely need different ones.

But, for example, it's as simple as:

url     = 'http://example.tld'
payload = { 'key' : 'val' }
headers = {}
res = requests.post(url, data=payload, headers=headers)

You can even use a nice Python dict to supply the query string in a GET request with params={}.

Simple and elegant. Keep calm, and fly on.

Community
  • 1
  • 1
OJFord
  • 8,132
  • 7
  • 52
  • 88
  • I am using python 2.4.3. can't use requests. ImportError: No module named requests. – Gary Mar 17 '16 at 14:52
  • 11
    @Gary `pip install requests` – OJFord Mar 17 '16 at 15:45
  • I had to do json.dumps(payload), but didn't need to use the json.dumps method on headers. – user1045085 Sep 07 '16 at 03:46
  • Right @user1045085, because you wanted to send a JSON payload - but `headers` is a (dictionary, not string) argument to a Python function, so JSON doesn't come into it. – OJFord Sep 07 '16 at 13:37
  • @OllieFord Do you know by any chance if this would this work for Firebase? Im new to Fb and python but they have a REST API which they have documented as accepting curl requests. – marciokoko Sep 21 '16 at 17:13
  • @marciokoko Most definitely! :) `requests` is "just" HTTP under the hood. Use it as you would curl, but fly faster with Python. – OJFord Sep 21 '16 at 17:19
  • 1
    +1. To the best of my knowledge, anything that can be done with cURL can also be done via python requests. Might as well use that one instead. – idungotnosn Jan 09 '17 at 18:37
  • 1
    this worked perfectly for what I came looking for. Thanks @OJFord – galois Sep 29 '17 at 14:45
  • Doesn't work on kibana due to content security policy issues. – I'll Eat My Hat Aug 05 '19 at 14:03
  • @I'll Eat My Hat - refer to Kibana API docs. If that's the case, it's true of using curl directly too, not a problem introduced by `requests`. – OJFord Aug 05 '19 at 22:00
  • It turns out that kibana just returns very bad error messages. – I'll Eat My Hat Aug 07 '19 at 16:04
  • How to insert option then? There are quite a number of option when using curl – Luk Aron Nov 14 '19 at 10:12
  • @LukAron See the docs here: http://requests.readthedocs.org/en/latest/ For converting an existing curl command with several options, it might be easier to use a tool that can convert for you, like Postman app or in the browser at https://curl.trillworks.com/ – OJFord Nov 19 '19 at 10:37
43

You could use urllib as @roippi said:

import urllib2
data = '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}'
url = 'http://localhost:8080/firewall/rules/0000000000000001'
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
for x in f:
    print(x)
f.close()
Uxio
  • 1,181
  • 10
  • 12
32

If you are not tweaking the curl command too much you can also go and call the curl command directly

import shlex
cmd = '''curl -X POST -d  '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://localhost:8080/firewall/rules/0000000000000001'''
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
hyades
  • 2,752
  • 1
  • 14
  • 31
  • Thanks, I implemented it using subprocess.call() – Kiran Vemuri Sep 24 '14 at 19:42
  • 1
    I was very well helped with the suggestion of @Ollie Ford: I have installed Requests under W10. From command line, started up Python and then composed my requested URL. From now I'll have to figure out how to set up (and view the contents of) a stream in a .py file. Suggestions welcome! – user3722096 Nov 25 '16 at 21:28
29

Use this tool (hosted here for free) to convert your curl command to equivalent Python requests code:

Example: This,

curl 'https://www.example.com/' -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Origin: https://www.example.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Cookie: SESSID=ABCDEF' --data-binary 'Pathfinder' --compressed

Gets converted neatly to:

import requests

cookies = {
    'SESSID': 'ABCDEF',
}

headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'Origin': 'https://www.example.com',
    'Accept-Encoding': 'gzip, deflate, br',
}

data = 'Pathfinder'

response = requests.post('https://www.example.com/', headers=headers, cookies=cookies, data=data)
Nitin Nain
  • 3,659
  • 1
  • 33
  • 46
  • 4
    While @OJFord has enlightened us why not to use curl within python, Nitin has depicted the simplest way to implement the same using "requests". I highly recommend this answer. – mdabdullah May 08 '19 at 14:15
  • 1
    WOW amazing! This is not a just "Let me just do it for you" solution. I've been struggling with trying to figure out why my command isn't working even though everything appeared to be correct. This tool pointed out to me that my base url for my request had extra parameters in it that were causing problems. For folks who are new to making these sorts requests this tool is perfect for checking for these sorts of simple oversights. Excellent answer. – VanBantam Jul 19 '20 at 18:39
7

Try with subprocess

CurlUrl="curl 'https://www.example.com/' -H 'Connection: keep-alive' -H 'Cache- 
          Control: max-age=0' -H 'Origin: https://www.example.com' -H 'Accept-Encoding: 
          gzip, deflate, br' -H 'Cookie: SESSID=ABCDEF' --data-binary 'Pathfinder' -- 
          compressed"

Use getstatusoutput to store the results

status, output = subprocess.getstatusoutput(CurlUrl)
Simas Joneliunas
  • 2,522
  • 12
  • 20
  • 28
Ramesh Ponnusamy
  • 586
  • 4
  • 9
2

Rephrasing one of the answers in this post, instead of using cmd.split(). Try to use: 

import shlex

args = shlex.split(cmd)

Then feed args to subprocess.Popen.

Check this doc for more info: https://docs.python.org/2/library/subprocess.html#popen-constructor

Ganesh prasad
  • 137
  • 1
  • 6
0

You can use below code snippet

import shlex
import subprocess
import json

def call_curl(curl):
    args = shlex.split(curl)
    process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    return json.loads(stdout.decode('utf-8'))


if __name__ == '__main__':
    curl = '''curl - X
    POST - d
    '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}'
    http: // localhost: 8080 / firewall / rules / 0000000000000001 '''
    output = call_curl(curl)
    print(output)
Prashant Kanak
  • 11
  • 3