Madhav Kobal's Blog

This blog will be dedicated to Linux, Open Source and Technology news, affairs, how-tos and virtually EVERYTHING in these domains.

How to Download Files From The Web Using wget

Posted by madhavkobal on 21/08/2009

Download a zip archive:

wget http://website.com/archive.zip

and archive.zip will be downloaded. But we can download files using a lot of parameters. Read on!

wget -r http://website.com

This download all files recursively: images, html files, etc. But this could get us banned by the server for sending too many download requests so to avoid this:

wget --random-wait --limit-rate=20k -r http://website.com

–random-wait means to download a file and then wait for a random period of time, then download the next file and so on.

–limit-rate=20k indicates that you want to download at a maximum speed of 20k so you don’t get banned.

Or you could also do:

wget --wait=20 --limit-rate=20K -r -p -U Mozilla http://website.com

–wait=20 to wait 20 seconds between each file download, but I think it’s better to download with –random-wait

-p indicates that the files should be displayed as HTML, as if you were actually looking at the page

-U Mozilla will make the website believe you are using a Mozilla browser.

And here is how to download all images, videos or whatever you want, from a website:

wget -r -A=.jpg,.png http://website.com

With this command, you download all jpg and png files from website.com. If you want to download all mp3s, then you would use -A=.mp3

You can also use a GUI for wget if you want. It’s called Gwget and should be in your distribution repositories. For Ubuntu, do:

sudo apt-get install gwget
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: