Discussion:
Problem with wget
Add Reply
root
2020-03-19 09:03:11 UTC
Reply
Permalink
I have come across a site which will not accept a download request
from wget, but will accept the same request from curl. My son
runs Ubuntu and his wget can fetch from the site. I am running
14.2 Slackware 64. Thinking that my version of wget might be
too old I updated to the version in current which necessitated
updating a half dozen other packages. No success, however
with the newest version of wget. The site is a small private
operation so I don't want to burden it with a number of
requests so I hope you can suggest a reason why wget might
fail under Slackware and not under Ubuntu, or why curl
works and wget doesn't.

I have tried different user agents as well as no-check-certificate
which has worked on other problem sites.

Thanks for any suggestions.
Rinaldi J. Montessi
2020-03-19 12:16:05 UTC
Reply
Permalink
Post by root
I have come across a site which will not accept a download request
from wget, but will accept the same request from curl. My son
runs Ubuntu and his wget can fetch from the site. I am running
14.2 Slackware 64. Thinking that my version of wget might be
too old I updated to the version in current which necessitated
updating a half dozen other packages. No success, however
with the newest version of wget. The site is a small private
operation so I don't want to burden it with a number of
requests so I hope you can suggest a reason why wget might
fail under Slackware and not under Ubuntu, or why curl
works and wget doesn't.
I have tried different user agents as well as no-check-certificate
which has worked on other problem sites.
Thanks for any suggestions.
I can think of a couple of things:

Switch wget to active or passive ftp in /etc/wgetrc

Send your wgetrc to your son and have him run a diff

Rinaldi
--
Critic, n.:
A person who boasts himself hard to please because nobody tries
to please him.
-- Ambrose Bierce, "The Devil's Dictionary"
root
2020-03-19 16:41:40 UTC
Reply
Permalink
Post by Rinaldi J. Montessi
Switch wget to active or passive ftp in /etc/wgetrc
Send your wgetrc to your son and have him run a diff
Rinaldi
Wow, makes me feel stupid: I have not known about /etc/wgetrc.
This offers lots of possibilities.

Thanks for responding.
Rinaldi J. Montessi
2020-03-19 19:00:29 UTC
Reply
Permalink
Post by root
Post by Rinaldi J. Montessi
Switch wget to active or passive ftp in /etc/wgetrc
Send your wgetrc to your son and have him run a diff
Rinaldi
Wow, makes me feel stupid: I have not known about /etc/wgetrc.
This offers lots of possibilities.
Thanks for responding.
There's aslo a ~/.wgetrc for those who don't have root access ;-)

Rinaldi
--
Critic, n.:
A person who boasts himself hard to please because nobody tries
to please him.
-- Ambrose Bierce, "The Devil's Dictionary"
Eric Pozharski
2020-03-20 07:42:27 UTC
Reply
Permalink
Post by root
I have come across a site which will not accept a download request
from wget, but will accept the same request from curl.
*SKIP*

Also, trying '--debug' could give valuable insight what exactly is
happening, because wget fails this output shouldn't be copious. I mean
-- how about posting it here?
--
Torvalds' goal for Linux is very simple: World Domination
Stallman's goal for GNU is even simpler: Freedom
Henrik Carlqvist
2020-03-20 19:40:02 UTC
Reply
Permalink
Post by Eric Pozharski
Also, trying '--debug' could give valuable insight what exactly is
happening, because wget fails this output shouldn't be copious. I mean
-- how about posting it here?
It could also be possible to use a network sniffing tool like wireshark
to look at the network traffic and that way compare the failing wget with
the successful curl.

regards Henrik
root
2020-03-20 23:09:19 UTC
Reply
Permalink
Post by Henrik Carlqvist
Post by Eric Pozharski
Also, trying '--debug' could give valuable insight what exactly is
happening, because wget fails this output shouldn't be copious. I mean
-- how about posting it here?
It could also be possible to use a network sniffing tool like wireshark
to look at the network traffic and that way compare the failing wget with
the successful curl.
regards Henrik
Thanks to everyone who responded. The problem was my file /etc/wgetrc.
Some time ago, beyond my memory, I had added a user-agent line. I
compared my file to that of my son and that difference stood out.
When I deleted the user-agent line wget worked. This was the
offending line:

user_agent = Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2

Along the way I found that the default behavior of curl is to *not*
follow redirect links. curl -L mimics the behavior of wget in this
respect.

Thanks Henrik, and thanks to everyone who responded.

Loading...