Troubleshooting `$ wget -A.pdf -r site.com` I am trying to get homework papers with the command but for some reason, it does not return them. Any idea why? $ wget -A.pdf -r -nd My goal is to download all homework and then merge them with `pdftk` and then search over them with some program -- but I cannot proceed (without again going to Firefox DownloadThemAll -plugin) until I get this working. Ideas why not working? Have sysadmins set up some bans for wgets or why does it fail? I get only this robots.txt -file: $ cat robots.txt User-agent: * Disallow: / User-agent: Googlebot Allow: / Disallow: /cgi-bin/

Your problem is in `robots.txt`. Wget looks on it and sees


User-agent: *
Disallow: /


You can use tell wget to ignore the `robots.txt` file:


wget -A.pdf -r -nd -e robots=off


And it will work.