Download webpages using curl in batch file

Question

I'm interested in downloading a bunch of webpages by running a curl statement in a loop in a batch file.

The webpages end in a number which are in a range, of say from 100 to 200, and I would like to download each page as an html file.

When I run the following, the program freezes / throws and error. How can I get this to work?

Thanks in advance!

@echo off

echo @echo pages will begin downloading... > ListOfPages.txt

SETLOCAL ENABLEDELAYEDEXPANSION

For /L %%A IN (100,1,200) do (  
    SET page="http://path/to/webpage/%%A"
    SET destination="C:\path\to\file\sitenumber_%%A.html"

    :: insert curl statements to new file
    echo curl !page! -OutFile !destination! >> ListOfPages.txt
)


:: Loop through each line of new file and execute curl statement
For /F %%G in (C:\path\to\file\ListOfPages.txt) do (

     WHAT GOES HERE???

 )

For that kind of problems, I `echo 'curl ...'`, check if the links are ok. Then `echo` to a new file and run the new file. Don't try to do it in one script. You need a pipeline of checkpoints — lllllllllll, Jun 09 '16 at 23:13
@lllllllllll once I send the curl statements to separate text file, how do I then execute those curl commands? ..I can update the question — mrp, Jun 09 '16 at 23:46
Frankly, I don't know how to do it in Windows. Can you run `curl` from the windows console? Then, save your curl list as bat file and run it from the windows console. It should work, but I am not sure. If you use Cygwin, then `chmod u+x file; your_shell_name file` — lllllllllll, Jun 09 '16 at 23:53
@lllllllllll I will try what you suggested. Unfortunately I can't use Cygwin though. I'm stuck dealing with Windows specific commands. If I could use linux commands this would be WAY easier. Simply: curl "http ://path/to/website/[100-200]" -o sitenumber_#1.html -s — mrp, Jun 09 '16 at 23:58
http://stackoverflow.com/questions/9507353/how-do-i-install-set-up-and-use-curl-on-a-windows — lllllllllll, Jun 10 '16 at 00:01

score 0 · Accepted Answer · edited Feb 27 '18 at 11:56

The answer to this is that you can't run curl (as far as I've tested), however you can use the powershell commands in a bat file.

Per @lllllllllll's suggestion, send the list of sites into a new file, however it should be a .bat file, not a text file. And the command that should be used isn't curl, but PowerShell.exe -Command Invoke-WebRequst:

@echo off

echo @echo pages will begin downloading... > ListOfPages.bat

SETLOCAL ENABLEDELAYEDEXPANSION

For /L %%A IN (100,1,200) do (  
    SET page="http://path/to/webpage/%%A"
    SET destination="C:\path\to\file\sitenumber_%%A.html"

    :: insert curl statements to new file
    echo PowerShell.exe -Command Invoke-WebRequest !page! -OutFile !destination! >> ListOfPages.bat
)

then go back to the PowerShell in that folder where the .bat file is saved, and run:

./ListOfPages.bat

Download webpages using curl in batch file

1 Answers1