Scraping data through secure login website ex(Curl) php

Question

i have an issue, i need to learn how to access secure login forms so i can get the data that are inside of them. I am new to this and i want to do it with Curl and then scrape the data with simple_html_dom or regex.As you can see from the code below, it's a simple form username and password. My question here is how can i access the form with Curl and any other form without writing in the page username admin and password 123? I know that i have to use CURLOPT_RETURNTRANSFER, CURLOPT_SSL..etc and i also know that scraping websites it's never the same code, every website has different options that you will have to manage with CURLOPT_... but i don't know the structure. what i need and where to start.

Code

<body>
<center>
    <form method="post">
        <label>Username</label>
            <input name="user" type="text">
        <label>Password</label>
            <input name="pass" type="password">
        <button type="submit">Submit</button>
    </form>
</center>


</body>
<html>

<?php

$name="";
$password = "";

if(!empty($_POST['user']) && !empty($_POST['pass'])){
    $name = $_POST['user'];
    $password = $_POST['pass'];
}

if($name == "admin" && $password == "123"){
    $url = "https://stackoverflow.com/";
    $html = file_get_html($url);

    foreach ($html->find('a') as $values){
        echo $values->href."<br>";
    }
}   else{
    echo "Error";
}

You see this code above: First of all the parsing it's not working inside the if statement outside it's fine but my issue as i said is how can i access secure forms with curl and then post the scraped data? forms like that are everywhere. if anyone knows how to guide me somewhere or tell would be fine. Thank you for your time

i don't know a lot about php but you can maybe search curl manual ? I know it's still hard — , Jun 13 '20 at 11:21

score 1 · Accepted Answer · answered Jun 13 '20 at 11:35

1

You do not have to access to form, of course, if you want to do automatic for multiple random websites, then you must use regex to find login forms.

If you already have a website in your mind to login you can use CURL and send a POST request, (please look to already opened question in stackoverflow PHP + curl, HTTP POST sample code). If website form uses "token" in forms for "csrf" protection then your code must find that token, add it to your POST request fields and make a request.

I recommend you guzzlehttp/guzzle, if you do not use composer just download it from github and require/include src/Client.php and write your code. Please look here for POST/Form request documentation and form_params. Have your time read documentation and write good code. Good luck

answered Jun 13 '20 at 11:35

Teymur Mardaliyer Lennon

832
7
15

i will accept your answer thank you a lot. One question can you explain me what it's going on with curl post why we need it etc..cause as i said i am new – ThunderBoy Jun 13 '20 at 11:49
and also you are using guzzle ? not curl ? if yes why ? – ThunderBoy Jun 13 '20 at 12:25
1

CURL is a command line tool for sending requests from command-lines to clients (websites, webservices etc). Yes, mostly I am using guzzle, guzzle is simple and easy it also uses CURL. guzzle library makes life easier thats all :) Btw, mostly curl comes on php, otherwise you have to install curl package (php-curl) to your php. – Teymur Mardaliyer Lennon Jun 13 '20 at 17:49

Scraping data through secure login website ex(Curl) php

1 Answers1