[Need your help] Very weird issue: why does curl close the http connection with slow networks?

[A Abc via curl-users]

A long time for investigating and google, but no luck :-(

------------------------------
Problem:
a)
one broken case with slow networks:
hhs-MacBook-Pro:temp hh$ curl -v --limit-rate 100k -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)

GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:06:31 GMT
<
{ [13617 bytes data] 39 1240k   39  489k    0     0  81681      0  0:00:15  0:00:06  0:00:09 70292* transfer closed with 682811 bytes remaining to read
* Curl_http_done: called premature == 1
 46 1240k   46  574k    0     0  95813      0  0:00:13  0:00:06  0:00:07  102k
* Closing connection 0
curl: (18) transfer closed with 682811 bytes remaining to read

b)
one good case with fast networks:
hhs-MacBook-Pro:temp hh$ curl -v -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)

GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:07:21 GMT
<
{ [13617 bytes data]
* Curl_http_done: called premature == 0
100 1240k  100 1240k    0     0  56.5M      0 --:--:-- --:--:-- --:--:-- 57.7M
* Connection #0 to host 192.168.30.212 left intact

c)
one sample for packages by wireshark:
GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

HTTP/1.1 200
Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/javascript
Content-Length: 1270588
Date: Wed, 12 Apr 2017 07:18:50 GMT

!function(e){function t(r){if(n[r])return n[r].exports;var o=n[r]={exports:{},id:r,loaded:!1};return e[r].call(o.exports,o,o.exports,t),o.loaded=!0,o.exports}var n={};return t.m=e,t.c=n,t.p="",t(0)}([function(e,t,n){n(474),n(475),n(476),n(473),e.exports=n(458)},function(e,t,n){"use strict";e.exports=n(82)},function(e,t,n){"use strict";function r(e){return e&&e.__esModule?e:{default:e}}t.__esModule=!0;var o=n(467),i=r(o);t.default=i.default||function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e}},function(e,t,n){var r,o;/*!
          Copyright (c) 2016 Jed Watson.
          Licensed under the MIT License (MIT), see
          http://jedwatson.github.io/classnames
        */!function(){"use strict";function n(){for(var e=[],t=0;t<arguments.length;t++){var r=arguments[t];if(r){var o=typeof r;if("string"===o||"number"===o)e.push(r);else if(Array.isArray(r))e.push(n.apply(null,r));else if("object"===o)for(var a in
... ...



------------------------------
More information:
1.
the server side on CentOS7.2 is a springboot application, that means launch a .jar package with an embedded tomcat 8.5.6
So, if change into .war package, and deploy into an standalone tomcat8.5, then no broken.

2.
if put a Fiddler(v4.6) between curl and server, even with much slower networks, no broken.
Of course, although the speed is much slower like 10k/second, but the speed is smooth, like delay 80ms per 1KB.

3.
chrome(v57.0.2987) has similar issue.
In fact, we found the issue on chrome at first, but for debugging, find that curl has similar issue.
if has network throttling < WIFI, then errors with "net::ERR_CONTENT_LENGTH_MISMATCH"

4.
for most cases, when curl send the FIN+ACK, there are about 1.5second for no any tcp package from server to curl.
For most cases, when there is "1.5second pause" for a data stream from server to client, normally, there are serveral special packages marked by black background from wireshark as the snapshot: anyProblem.png

5:
then, I add a test API on server applicaiton, it will write some bytes into response, then sleep 5 seconds, then continue to write response. But curl is waiting here, no broken, no FIN+ACK.

6:
so far, we just test the static resource file, like javascript file. Since for #5 above, the test is not for static content, so maybe #5 has less point.
workaround is: add gzip mode for static resource.But if the static resource is much bigger, then the issue happens again.

7. also tried "--speed-time <time>", "--speed-limit <speed>", but seems useless.

8. also, tried "--trace <file>", no more information/clues




-------------------------------
Background:
same issues  for curl on:
Mac 10.12.3:
hhs-MacBook-Pro:temp hh$ curl -V
curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

on CentOS7.2:
[***@localhost tmp]$ curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

even on CentOS7.2:
a build from source code with Debug version:
# curl -V
curl 7.54.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.54.0-DEV
Protocols: dict file ftp gopher http imap pop3 rtsp smtp telnet tftp
Features: Debug TrackMemory IPv6 Largefile UnixSockets

----------
My questions:
1. why does curl close the http connection?
2. how to fix it?

Thanks

[A Abc via curl-users]

also same issues with another network speed "--limit-rate 200k"
And also some snapshots are here, which mentioned by the first email.

On Thursday, April 13, 2017 2:09 PM, A Abc <***@yahoo.com> wrote:


A long time for investigating and google, but no luck :-(

------------------------------
Problem:
a)
one broken case with slow networks:
hhs-MacBook-Pro:temp hh$ curl -v --limit-rate 100k -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)

GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:06:31 GMT
<
{ [13617 bytes data] 39 1240k   39  489k    0     0  81681      0  0:00:15  0:00:06  0:00:09 70292* transfer closed with 682811 bytes remaining to read
* Curl_http_done: called premature == 1
 46 1240k   46  574k    0     0  95813      0  0:00:13  0:00:06  0:00:07  102k
* Closing connection 0
curl: (18) transfer closed with 682811 bytes remaining to read

b)
one good case with fast networks:
hhs-MacBook-Pro:temp hh$ curl -v -O http://192.168.30.212:8080/main-11359f6396107106f1d1.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.30.212...
* TCP_NODELAY set
* Connected to 192.168.30.212 (192.168.30.212) port 8080 (#0)

GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

< HTTP/1.1 200
< Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
< Accept-Ranges: bytes
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/javascript
< Content-Length: 1270588
< Date: Wed, 12 Apr 2017 08:07:21 GMT
<
{ [13617 bytes data]
* Curl_http_done: called premature == 0
100 1240k  100 1240k    0     0  56.5M      0 --:--:-- --:--:-- --:--:-- 57.7M
* Connection #0 to host 192.168.30.212 left intact

c)
one sample for packages by wireshark:
GET /main-11359f6396107106f1d1.js HTTP/1.1
Host: 192.168.30.212:8080
User-Agent: curl/7.51.0
Accept: */*

HTTP/1.1 200
Last-Modified: Wed, 12 Apr 2017 04:57:23 GMT
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/javascript
Content-Length: 1270588
Date: Wed, 12 Apr 2017 07:18:50 GMT

!function(e){function t(r){if(n[r])return n[r].exports;var o=n[r]={exports:{},id:r,loaded:!1};return e[r].call(o.exports,o,o.exports,t),o.loaded=!0,o.exports}var n={};return t.m=e,t.c=n,t.p="",t(0)}([function(e,t,n){n(474),n(475),n(476),n(473),e.exports=n(458)},function(e,t,n){"use strict";e.exports=n(82)},function(e,t,n){"use strict";function r(e){return e&&e.__esModule?e:{default:e}}t.__esModule=!0;var o=n(467),i=r(o);t.default=i.default||function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var r in n)Object.prototype.hasOwnProperty.call(n,r)&&(e[r]=n[r])}return e}},function(e,t,n){var r,o;/*!
          Copyright (c) 2016 Jed Watson.
          Licensed under the MIT License (MIT), see
          http://jedwatson.github.io/classnames
        */!function(){"use strict";function n(){for(var e=[],t=0;t<arguments.length;t++){var r=arguments[t];if(r){var o=typeof r;if("string"===o||"number"===o)e.push(r);else if(Array.isArray(r))e.push(n.apply(null,r));else if("object"===o)for(var a in
... ...



------------------------------
More information:
1.
the server side on CentOS7.2 is a springboot application, that means launch a .jar package with an embedded tomcat 8.5.6
So, if change into .war package, and deploy into an standalone tomcat8.5, then no broken.

2.
if put a Fiddler(v4.6) between curl and server, even with much slower networks, no broken.
Of course, although the speed is much slower like 10k/second, but the speed is smooth, like delay 80ms per 1KB.

3.
chrome(v57.0.2987) has similar issue.
In fact, we found the issue on chrome at first, but for debugging, find that curl has similar issue.
if has network throttling < WIFI, then errors with "net::ERR_CONTENT_LENGTH_MISMATCH"

4.
for most cases, when curl send the FIN+ACK, there are about 1.5second for no any tcp package from server to curl.
For most cases, when there is "1.5second pause" for a data stream from server to client, normally, there are serveral special packages marked by black background from wireshark as the snapshot: anyProblem.png

5:
then, I add a test API on server applicaiton, it will write some bytes into response, then sleep 5 seconds, then continue to write response. But curl is waiting here, no broken, no FIN+ACK.

6:
so far, we just test the static resource file, like javascript file. Since for #5 above, the test is not for static content, so maybe #5 has less point.
workaround is: add gzip mode for static resource.But if the static resource is much bigger, then the issue happens again.

7. also tried "--speed-time <time>", "--speed-limit <speed>", but seems useless.

8. also, tried "--trace <file>", no more information/clues




-------------------------------
Background:
same issues  for curl on:
Mac 10.12.3:
hhs-MacBook-Pro:temp hh$ curl -V
curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

on CentOS7.2:
[***@localhost tmp]$ curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

even on CentOS7.2:
a build from source code with Debug version:
# curl -V
curl 7.54.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.54.0-DEV
Protocols: dict file ftp gopher http imap pop3 rtsp smtp telnet tftp
Features: Debug TrackMemory IPv6 Largefile UnixSockets

----------
My questions:
1. why does curl close the http connection?
2. how to fix it?

Thanks

[Daniel Stenberg]

Post by A Abc via curl-users
curl: (18) transfer closed with 682811 bytes remaining to read

If you didn't specify a timeout, this is *the server* (or something in your
network) closing the connection before the transfer was complete.

But when curl does this due to a timeout, it is supposed to return a time-out
error message so I'm inclined to say this was someone else closing the
connection.

Post by A Abc via curl-users
chrome(v57.0.2987) has similar issue.

... which I think only enforces the suspicion that this is not a client-side
problem.

Post by A Abc via curl-users
then, I add a test API on server applicaiton, it will write some bytes into
response, then sleep 5 seconds, then continue to write response. But curl is
waiting here, no broken, no FIN+ACK.

I don't understand this point.

[A Abc via curl-users]

hi Daniel,
Thanks for your reply.
--
I also suspect the issue is from server side.
But did you see the wireshark snapshot attached on my second email. (I also attached the snapshot in the current email)
It's curl who sends the FIN+ACK to server side, then server side sends back ACK.
So, is it possible that server side triggers the connection closing, but client side(curl) sends the FIN+ACK first, then server side sends ACK?This is out of knowledge.

on the snapshot, 10.1.12.180 is the client side(curl), 192.168.30.212 is the http server side.

--
For the case, that you said you don't understand this point.What I want to say is that:From the beginning of that email, we can suspect that, curl may close the connection, due to no any data traffic between curl and server for about 1.5second.
But with my test application, it proved that, curl will not close the connection even there is no any data traffic bewteen curl and server for 5 second.

I will appreciate your more information.
Thanks

Post by A Abc via curl-users
curl: (18) transfer closed with 682811 bytes remaining to read

If you didn't specify a timeout, this is *the server* (or something in your
network) closing the connection before the transfer was complete.

But when curl does this due to a timeout, it is supposed to return a time-out
error message so I'm inclined to say this was someone else closing the
connection.

Post by A Abc via curl-users
chrome(v57.0.2987) has similar issue.

... which I think only enforces the suspicion that this is not a client-side
problem.

Post by A Abc via curl-users
then, I add a test API on server applicaiton, it will write some bytes into
response, then sleep 5 seconds, then continue to write response. But curl is
waiting here, no broken, no FIN+ACK.

I don't understand this point.

[A Abc via curl-users]

sorry, forgot the attachments.


On Wednesday, April 19, 2017 4:37 PM, A Abc <***@yahoo.com> wrote:


hi Daniel,
Thanks for your reply.
--
I also suspect the issue is from server side.
But did you see the wireshark snapshot attached on my second email. (I also attached the snapshot in the current email)
It's curl who sends the FIN+ACK to server side, then server side sends back ACK.
So, is it possible that server side triggers the connection closing, but client side(curl) sends the FIN+ACK first, then server side sends ACK?This is out of knowledge.

on the snapshot, 10.1.12.180 is the client side(curl), 192.168.30.212 is the http server side.

--
For the case, that you said you don't understand this point.What I want to say is that:From the beginning of that email, we can suspect that, curl may close the connection, due to no any data traffic between curl and server for about 1.5second.
But with my test application, it proved that, curl will not close the connection even there is no any data traffic bewteen curl and server for 5 second.

I will appreciate your more information.
Thanks

Post by A Abc via curl-users
curl: (18) transfer closed with 682811 bytes remaining to read

If you didn't specify a timeout, this is *the server* (or something in your
network) closing the connection before the transfer was complete.

But when curl does this due to a timeout, it is supposed to return a time-out
error message so I'm inclined to say this was someone else closing the
connection.

Post by A Abc via curl-users
chrome(v57.0.2987) has similar issue.

... which I think only enforces the suspicion that this is not a client-side
problem.

Post by A Abc via curl-users
then, I add a test API on server applicaiton, it will write some bytes into
response, then sleep 5 seconds, then continue to write response. But curl is
waiting here, no broken, no FIN+ACK.

I don't understand this point.