Login to website using PHP and get text from page

Question

I am trying to login to a website and grab content from a page you must be authenticated to see. I have done some research and have seen some examples using both cURL and stream_context_create but I cannot get either way to work. I have the url for the page to login to, and the page that contains the data I need to get. Your help is much appreciated!

Here's what I'm working with:

<?php 
    $pages = array('home' => 
'https://www.53.com/wps/portal/personal', 
               'login' => 
'https://www.53.com/wps/portal/personal', 
               'data' => 
'https://www.53.com/servlet/efsonline/index.html?Messages.SortedBy=DATE,REVERSE'); 
    $ch = curl_init(); 
    //Set options for curl session 
    $options = array(CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)', 
             CURLOPT_SSL_VERIFYPEER => FALSE, 
             CURLOPT_SSL_VERIFYHOST => 2, 
             CURLOPT_HEADER => TRUE, 
             //CURLOPT_RETURNTRANSFER => TRUE, 
             CURLOPT_COOKIEFILE => 'cookie.txt', 
             CURLOPT_COOKIEJAR => 'cookies.txt'); 

    //Hit home page for session cookie 
    $options[CURLOPT_URL] = $pages['home']; 
    curl_setopt_array($ch, $options); 
    curl_exec($ch); 

    //Login 
    $options[CURLOPT_URL] = $pages['login']; 
    $options[CURLOPT_POST] = TRUE; 
    $options[CURLOPT_POSTFIELDS] = 'uid-input=xxx&pw=xxx'; 
    $options[CURLOPT_FOLLOWLOCATION] = FALSE; 
    curl_setopt_array($ch, $options); 
    curl_exec($ch); 

    //Hit data page 
    $options[CURLOPT_URL] = $pages['data']; 
    curl_setopt_array($ch, $options); 
    $data = curl_exec($ch); 

    //Output data
    echo $data; 

    //Close curl session 
    curl_close($ch); 
?>

Cheers,

Anthony

Show the code you can't get to work and somebody will certainly be able to help you out — Pekka, Nov 20 '10 at 13:42
Please check [link][1] and help me [1]: http://stackoverflow.com/questions/11218783/no-response-with-curl here — avinashse, Jun 28 '12 at 09:41

score 0 · Answer 1 · edited May 23 '17 at 12:09

If you have a look at How to post data in PHP using file_get_contents? you probably get what you need.

(I guess you would need to do the same post as when you log in and continue the session from there)

Edit (as response to question below)

In the first request the client will respond with some cookie after the login. This cookie must you set as a header in the second request.

If you get 500 errors it might be that the server is rejecting your request as you do not send sufficient headers to it. It's not uncommon to reject you. Try the following:

$opts = array('http' =>
array('method' => 'Get',
        'Header' => "Host: www.someserver.com\r\n".
        "User-Agent: Mozilla\r\n",
    'user_agent' => 'Mozilla'));

$context = stream_context_create($opts);

$fp = fopen('http://www.someserver.no', 'r', false, $context);
fpassthru($fp);
fclose($fp);

... what headers that you might need to set will vary from server to server.

Right, but I think my problem is I need to authenticate on a separate page and then carry over that session to the 2nd page that contains the content I want. I tried that example and calling file_get_contents twice, once for authentication and once to get the content from the 2nd page. But all I get is "...[function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error..." — Anthony Garand, Nov 20 '10 at 13:51
Do the 500 Internal server error come from your server or the server you send the query to= — Knubo, Nov 20 '10 at 17:56

score 0 · Answer 2 · answered Nov 20 '10 at 13:53

0

Anthony, some time ago I had to build something like that. What you need to focus on, is on the cookies.

Curl handles cookies, what you need to do is to make sure you login first, and in the same connection load the data. I you can't use the same connection, you can use the opts CURLOPT_COOKIE or CURLOPT_COOKIEFILE to help you. More info on that: http://ar2.php.net/manual/en/function.curl-setopt.php

answered Nov 20 '10 at 13:53

Alex Siri

2,707
1
15
22

If you're going to use "file_get_contents", then you'll need to extract the cookie header yourself and parse it, I would not recommend that. – Alex Siri Nov 20 '10 at 13:56
I tried that and found some examples online, look at the original post for the updated code, but it is outputting the login page and it gives the message from the website that cookies are not enabled. – Anthony Garand Nov 20 '10 at 14:10

score 0 · Answer 3 · answered Nov 20 '10 at 22:27

0

You are using the login fields ids instead of names:

$options[CURLOPT_POSTFIELDS] = 'uid-input=xxx&pw=xxx';

should be

$options[CURLOPT_POSTFIELDS] = 'UserName=xxx&Password=xxx';

answered Nov 20 '10 at 22:27

Liam Bailey

5,691
3
30
44

Login to website using PHP and get text from page

3 Answers3