Curl -K - works fine, but with -Z it outputs some zero length files

[Max Waterman via curl-users]

I have a script in which I have a pipe that I've made output a series of:

"
url="..."
output="..."
"

that pipe into a curl command:

"
| curl -K - lots of other options
"

If I have the curl command as above, it downloads one file at a time and it works just fine.

However, if I attempt to have curl run them in parallel by specifying "-Z", then it creates many of the downloaded files with zero length. This happens even if I specify "--parallel-max=1" which I thought would be the same as running one at a time, but it seems not.

Any idea what is going on, and how to debug this?

[EDIT]
I'm thinking the issue I'm seeing is quite similar to the one described here:

<https://github.com/curl/curl/issues/5611>

So, I guess I'm out of luck. I've subscribed to that issue, but I think I'll have to try and find a different tool.

Regards,

Max.
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
E

[Daniel Stenberg via curl-users]

Post by Max Waterman via curl-users
If I have the curl command as above, it downloads one file at a time and it
works just fine.
However, if I attempt to have curl run them in parallel by specifying "-Z",
then it creates many of the downloaded files with zero length.

You forgot to show us your 'curl -V' output! To me, this sounds like a bug
we've already fixed...

Post by Max Waterman via curl-users
<https://github.com/curl/curl/issues/5611>

I can't rule that out, but as you can get them serially it seems unlikely.

[Max Waterman via curl-users]

Post by Daniel Stenberg via curl-users

Post by Max Waterman via curl-users
If I have the curl command as above, it downloads one file at a time and it
works just fine.
However, if I attempt to have curl run them in parallel by specifying "-Z",
then it creates many of the downloaded files with zero length.

You forgot to show us your 'curl -V' output! To me, this sounds like a bug
we've already fixed...

$ curl -V
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Post by Daniel Stenberg via curl-users

Post by Max Waterman via curl-users
<https://github.com/curl/curl/issues/5611>

I can't rule that out, but as you can get them serially it seems unlikely.

Hrm, a reasonable point, I suppose. I'll see if I can build a more recent version.

Regards,

Max.
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://cur

[Daniel Stenberg via curl-users]

Post by Max Waterman via curl-users
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0

See: https://github.com/curl/curl/issues/4937

Sounds very similar.

Post by Max Waterman via curl-users
Hrm, a reasonable point, I suppose. I'll see if I can build a more recent version.

The fix to #4937 shipped in 7.70.0.

[Max Waterman via curl-users]

I built a more up-to-date curl using the instructions (vcpkg):

$ ~/z/davidmaxwaterman/vcpkg/installed/x64-linux/tools/curl/curl -V
curl 7.74.0-DEV (Linux) libcurl/7.74.0-DEV OpenSSL/1.1.1j zlib/1.2.11
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM SSL TrackMemory UnixSockets

and it seems to work nicely. So, nice :D

The OS version is:

curl 7.68.0

so, I guess I could test the version in my script and force other users to upgrade too...else they'll hit the same problem, and it isn't completely obvious something has gone wrong.

FWIW, this is my speedup:

w/o -Z: 22m5s
with -Z: 5m57s

I did a 'diff -r' on the results, and they're the same, and I also re-ran the old version with -Z and confirmed many files are 0 bytes (total d/l is 119M instead of 230M)...which is why it takes only 4m47s :)

Anyway, thanks for prompting me to do that.

Regards,

Max.

Post by Daniel Stenberg via curl-users

Post by Max Waterman via curl-users
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0

See: https://github.com/curl/curl/issues/4937
Sounds very similar.

Post by Max Waterman via curl-users
Hrm, a reasonable point, I suppose. I'll see if I can build a more recent version.

The fix to #4937 shipped in 7.70.0.
--
/ daniel.haxx.se
| Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://www.wolfssl.com/contact/

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users