Hi,
I frequently see "curl: (18) transfer closed with outstanding read
data remaining". It seems that it is related to the use of a
http_proxy. Is there an option to make curl more robust so that it
will handle this problem correctly? Thanks.
--
Regards,
Peng
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
I frequently see "curl: (18) transfer closed with outstanding read data
remaining". It seems that it is related to the use of a http_proxy. Is there
an option to make curl more robust so that it will handle this problem
correctly?
It's not really a matter of "robustness". When the TCP connection gets
disconnected, that's not something curl can change or "improve".
What curl *could* do however, is retry the transfer when such an error
occurs
if such an option is used (which it currently doesn't), and then it *could*
ask for the transfer to get resumed from the last byte position it
received.
The latter part was submitted as the incomplete PR #2450 that seems to have
been abandoned...
[#2450] = https://github.com/curl/curl/pull/2450
--
/ daniel.haxx.se
Show quoted text
Hi,
I am not a technical user of curl and a lot of the e-mails I see I don't
understand but I do use it to scrape web pages and download 50,000+ web
pages each containing over 250 records and more.
When I action curl I check the return code. If an error then I retry the
curl command 5 times and if it still fails I log the curl command and url
details etc . and then move onto the next curl statement.
I know that this is probably not the same as what you are doing as I 'throw
way' the fails and start again if I encounter an error but it allows me to
continue on with the processing and to investigate any errors at a later
time
or during the actual processing. I even use the windows command line to
simulate the curl command that is in error and 'see' what is happening with
debug turned on.
Hope this was helpful.
Mike Lambert
Show quoted text
Hi,
I am not a technical user of curl and a lot of the e-mails I see I don't
understand but I do use it to scrape web pages and download 50,000+ web
pages each containing over 250 records and more.
When I action curl I check the return code. If an error then I retry the
curl command 5 times and if it still fails I log the curl command and url
details etc . and then move onto the next curl statement.
I know that this is probably not the same as what you are doing as I
'throw way' the fails and start again if I encounter an error but it allows
me to continue on with the processing and to investigate any errors at a
later time
or during the actual processing. I even use the windows command line to
simulate the curl command that is in error and 'see' what is happening with
debug turned on.
Hope this was helpful.
Mike Lambert
-----Original Message-----
Sent: Tuesday, 23 October 2018 3:21 AM
Subject: Re: curl: (18) transfer closed with outstanding read data remaining
Post by Peng YuI frequently see "curl: (18) transfer closed with outstanding read
data remaining". It seems that it is related to the use of a
http_proxy. Is there an option to make curl more robust so that it
will handle this problem correctly?
It's not really a matter of "robustness". When the TCP connection gets
disconnected, that's not something curl can change or "improve".
What curl *could* do however, is retry the transfer when such an error
occurs if such an option is used (which it currently doesn't), and then it
*could* ask for the transfer to get resumed from the last byte position it
received.
The latter part was submitted as the incomplete PR #2450 that seems to
have been abandoned...
[#2450] = https://github.com/curl/curl/pull/2450
--
/ daniel.haxx.se
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html