Strange redirection to Cloudflare server with Captcha?

Discussion:

Mac-Fly via curl-users

2021-05-15 03:19:44 UTC

Dear all,

since a few days a have the problem that a few pages that used to work just fine now return a page that comes from Cloudflare and wants to input some Captcha.

For example, a call to:
curl https://www.audacityteam.org
...returns a webpage with the title:
<title>Please Wait... | Cloudflare</title>
...and this meta-data:
<meta name="captcha-bypass" id="captcha-bypass" />
...and of course more content.
Strangely the same page called form any browser works just fine. And also, _was_ working fine for ages!
Also I tried to set the user agent to be exactly as my browser with no effect.

Do you have any idea what this may cause and (more important) how to work-around? :-)

The cURL I am using is:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/e

bruce via curl-users

2021-05-15 04:24:54 UTC

Permalink

Hi.

Welcome to the obnoxious world of crawling. A bunch of sites that sign
up to CDN/solns like Cloudflare are implementing recaptcha/cookie
issues.

There's nothing that says a target site has to play nice!!

However, if you walk through the process, and if you inspect the
traffic, you can possibly determine steps around the issue.

I've discovered with various targets that implement captcha, that the
image/process is usually only happening at th begining of the crawl
process. This allows me to perform a manual process where I manually
"solve" the captcha, and analyze the traffic to see what it's doing.

In most cases, the captcha process is creating a "cookie" file of some sort.

I've been able to "capture" the cookie, and to insert this in my
curl/crawl process and things more or less manage to work.

In fact, In most processes, the cookie lasts for hours/days and in
some cases I can craft a process that works through the site using
this method.

Your mileage might vary.

Good Luck!

On Fri, May 14, 2021 at 11:26 PM Mac-Fly via curl-users

Post by Mac-Fly via curl-users
Dear all,
since a few days a have the problem that a few pages that used to work just fine now return a page that comes from Cloudflare and wants to input some Captcha.
curl https://www.audacityteam.org
<title>Please Wait... | Cloudflare</title>
<meta name="captcha-bypass" id="captcha-bypass" />
...and of course more content.
Strangely the same page called form any browser works just fine. And also, _was_ working fine for ages!
Also I tried to set the user agent to be exactly as my browser with no effect.
Do you have any idea what this may cause and (more important) how to work-around? :-)
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiq

Mac-Fly via curl-users

2021-05-16 08:57:18 UTC

Permalink

Dear Bruce et al.,

I am afraid your are right. I tried to follow the steps you've described bit it seems quite difficult to find a good work-around. :-(

What is strange though is: I've used a browser (Firefox) with a completely empty profile (so no cache, no cookies etc...) and I don't see the captcha. So although I am "faking" the user agent to be the same as for Firefox ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0") there must be something more that Firefox is doing or Cloudfare is checking to avoid the captcha.

Post by bruce via curl-users
There's nothing that says a target site has to play nice!!

So - its at least not a cURL issue. That is, what I wanted to know.

Thanks a lot!

------------------------------

Date: Sat, 15 May 2021 00:24:54 -0400
From: bruce <***@gmail.com>
To: the curl tool <curl-***@cool.haxx.se>
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID:
<CAP16ngrbPQcNgadVCkLzo3JHkjmPYfrRSB+***@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

Hi.

Welcome to the obnoxious world of crawling. A bunch of sites that sign
up to CDN/solns like Cloudflare are implementing recaptcha/cookie
issues.

There's nothing that says a target site has to play nice!!

However, if you walk through the process, and if you inspect the
traffic, you can possibly determine steps around the issue.

I've discovered with various targets that implement captcha, that the
image/process is usually only happening at th begining of the crawl
process. This allows me to perform a manual process where I manually
"solve" the captcha, and analyze the traffic to see what it's doing.

In most cases, the captcha process is creating a "cookie" file of some sort.

I've been able to "capture" the cookie, and to insert this in my
curl/crawl process and things more or less manage to work.

In fact, In most processes, the cookie lasts for hours/days and in
some cases I can craft a process that works through the site using
this method.

Your mileage might vary.

Good Luck!

On Fri, May 14, 2021 at 11:26 PM Mac-Fly via curl-users

Post by bruce via curl-users
Dear all,
since a few days a have the problem that a few pages that used to work just fine now return a page that comes from Cloudflare and wants to input some Captcha.
curl https://www.audacityteam.org
<title>Please Wait... | Cloudflare</title>
<meta name="captcha-bypass" id="captcha-bypass" />
...and of course more content.
Strangely the same page called form any browser works just fine. And also, _was_ working fine for ages!
Also I tried to set the user agent to be exactly as my browser with no effect.
Do you have any idea what this may cause and (more important) how to work-around? :-)
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users

Petr Pisar via curl-users

2021-05-16 09:29:54 UTC

Permalink

Post by Mac-Fly via curl-users
What is strange though is: I've used a browser (Firefox) with a completely
empty profile (so no cache, no cookies etc...) and I don't see the captcha.
So although I am "faking" the user agent to be the same as for Firefox
("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
Firefox/88.0") there must be something more that Firefox is doing or
Cloudfare is checking to avoid the captcha.

That works for me. But that can be caused by other variables like my IP
address:

$ curl https://www.audacityteam.org/ --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' --verbose |head -n 10
* Uses proxy env variable no_proxy == 'localhost,router,router.bayer.uni.cx'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700:20::681a:6c:443...
* Connected to www.audacityteam.org (2606:4700:20::681a:6c) port 443 (#0)
* found 147 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: sni.cloudflaressl.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: C=US,ST=CA,L=San Francisco,O=Cloudflare\, Inc.,CN=sni.cloudflaressl.com
* start date: Tue, 22 Dec 2020 00:00:00 GMT
* expire date: Tue, 21 Dec 2021 23:59:59 GMT
* issuer: C=US,O=Cloudflare\, Inc.,CN=Cloudflare Inc ECC CA-3
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55da9cf91fc0)

Post by Mac-Fly via curl-users
GET / HTTP/2
Host: www.audacityteam.org
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

Mac-Fly via curl-users

2021-05-16 13:16:13 UTC

Permalink

Hi Petr,

this was a very good hint!!! Here are my new findings on the very same computer, very same connection.

It *seems* really related to CURL!

1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd

...the issue remains!

2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

...the issue is gone!!!

3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

...the issue is gone!!!

So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?

Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):

------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------

------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs

------------------- </LINUX> -------------------

What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.

------------------------------

Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <***@atlas.cz>
To: curl-***@cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/***@album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"

Post by Mac-Fly via curl-users
GET / HTTP/2
Host: www.audacityteam.org
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ® | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">

-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>

Petr Pisar via curl-users

2021-05-16 13:43:23 UTC

Permalink

Post by Mac-Fly via curl-users
So it looks like either my version is too new or something has changes that
requires a certain flag to be set. Whatever was changed is different between
version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the
SSL layer. But how to find out now whats happening?
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------

You need to compare the whole output (as I showed). Especially debugging of
the TLS version, HTTP version, and HTTP headers.

Post by Mac-Fly via curl-users
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.

It instructs a TCP layer not to buffer data to be sent into larger chunks.
I.e. it sends every byte as soon as possible.

-- Petr

Mac-Fly via curl-users

2021-05-17 19:27:16 UTC

Permalink

Post by Petr Pisar via curl-users
You need to compare the whole output (as I showed).
Especially debugging of the TLS version, HTTP version,
and HTTP headers.

I am afraid I don't know what to look at exactly.

But if you look at the snippets I posted: The IP is different already after the first lines (and these are the first lines of the log in verbose mode). Then it does no change. One IP points to Cloudfare (the windows one), the other does points to audacity (the Linux one). So I _think_ that already at that point something was different for an unknown reason. :-/

Anyway, here is the remaining log for Windows:

* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x2d06268)
} [5 bytes data]

Post by Petr Pisar via curl-users
GET / HTTP/2
Host: www.audacityteam.org
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Sun, 16 May 2021 12:56:42 GMT
< content-type: text/html; charset=UTF-8

...and Linux:

* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2225 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x56713f30)
} [5 bytes data]

Post by Petr Pisar via curl-users
GET / HTTP/2
Host: www.audacityteam.org
Accept: */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

Post by Petr Pisar via curl-users
Send curl-users mailing list submissions to

curl-***@cool.haxx.se

To subscribe or unsubscribe via the World Wide Web, visit
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
or, via email, send a message with subject or body 'help' to
curl-users-***@cool.haxx.se

You can reach the person managing the list at
curl-users-***@cool.haxx.se

When replying, please edit your Subject line so it is more specific
than "Re: Contents of curl-users digest..."

Today's Topics:

1. Re: Strange redirection to Cloudflare server with Captcha?
(Mac-Fly)
2. Re: Strange redirection to Cloudflare server with Captcha?
(Petr Pisar)

----------------------------------------------------------------------

Message: 1
Date: Sun, 16 May 2021 15:16:13 +0200
From: "Mac-Fly" <mac-***@gmx.net>
To: curl-***@cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <20210516.131613.462.2@[0.0.0.0]>
Content-Type: text/plain

Hi Petr,

this was a very good hint!!! Here are my new findings on the very same computer, very same connection.

It *seems* really related to CURL!

1.) Using:
curl 7.76.1 (i386-pc-win32) libcurl/7.76.1 OpenSSL/1.1.1k (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 WinIDN libssh2/1.9.0 nghttp2/1.43.0 libgsasl/1.10.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP Unicode UnixSockets zstd

...the issue remains!

2.) Using Linux (in a VM) with:
curl 7.64.0 (i686-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

...the issue is gone!!!

3.) Using Windows with built-in CURL (in system32 folder):
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

...the issue is gone!!!

So it looks like either my version is too new or something has changes that requires a certain flag to be set. Whatever was changed is different between version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the SSL layer. But how to find out now whats happening?

Here is the difference in the logs of CURL 7.76.1 on Windows versus Linux (note that the IP is different!):

------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------

------------------- <LINUX> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x56713f30)
* Expire in 1 ms for 1 (transfer 0x56713f30)
(a lot of that!)
* Trying 104.26.0.108...
* TCP_NODELAY set
* Connected to www.audacityteam.org (104.26.0.108) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs

------------------- </LINUX> -------------------

What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.

------------------------------

Message: 2
Date: Sun, 16 May 2021 11:29:54 +0200
From: Petr Pisar <***@atlas.cz>
To: curl-***@cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <YKDmEoP2YS/***@album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"

Post by Petr Pisar via curl-users
What is strange though is: I've used a browser (Firefox) with a completely
empty profile (so no cache, no cookies etc...) and I don't see the captcha.
So although I am "faking" the user agent to be the same as for Firefox
("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101
Firefox/88.0") there must be something more that Firefox is doing or
Cloudfare is checking to avoid the captcha.

Post by Petr Pisar via curl-users
GET / HTTP/2
Host: www.audacityteam.org
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Sun, 16 May 2021 09:26:20 GMT
< content-type: text/html; charset=UTF-8
< vary: Accept-Encoding
< cf-cache-status: DYNAMIC
< cf-request-id: 0a16188cb500004df4e8b28000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=B08U%2B6VwaRx3Y9jtuFtV617avTWMcTbGHi9ObnpG9gR%2FV4%2FLSwUep13ZnJZog%2F5cmf8Mm9Yu2RUIZCLudNRyrARB%2FzygWVvcBAr7B3EKGnAGGVtq49AHHpgv4zQF1QkWkw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 6503905abdab4df4-FRA
<
{ [875 bytes data]
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Audacity ? | Free, open source, cross-platform audio software for multi-track recording and editing.</title>
<link rel="profile" href="http://gmpg.org/xfn/11">
<link rel="pingback" href="https://www.audacityteam.org/xmlrpc.php">

-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/698139d8/attachment-0001.sig>

------------------------------

Message: 2
Date: Sun, 16 May 2021 15:43:23 +0200
From: Petr Pisar <***@atlas.cz>
To: curl-***@cool.haxx.se
Subject: Re: Strange redirection to Cloudflare server with Captcha?
Message-ID: <***@album.bayer.uni.cx>
Content-Type: text/plain; charset="utf-8"

Post by Petr Pisar via curl-users
So it looks like either my version is too new or something has changes that
requires a certain flag to be set. Whatever was changed is different between
version 7.76.1 and 7.64.0 on Windows an Linux. This could (of course) be the
SSL layer. But how to find out now whats happening?
------------------- <WINDOWS> -------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.67.74.133:443...
* Connected to www.audacityteam.org (172.67.74.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: D:\Users\Martin\Documents\Code\CodeBlocks_Projects\WebChangeMonitor\curl\bin\curl-ca-bundle.crt
* CApath: none
------------------- </WINDOWS> -------------------

You need to compare the whole output (as I showed). Especially debugging of
the TLS version, HTTP version, and HTTP headers.

Post by Petr Pisar via curl-users
What does TCP_NODELAY mean??? The same is set for Windows built-in curl 7.55.1.

It instructs a TCP layer not to buffer data to be sent into larger chunks.
I.e. it sends every byte as soon as possible.

-- Petr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://cool.haxx.se/pipermail/curl-users/attachments/20210516/1bb2cb8c/attachment-0001.sig>

------------------------------

Subject: Digest Footer

_______________________________________________
curl-users mailing list
curl-***@cool.haxx.se
https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users

------------------------------

End of curl-users Digest, Vol 189, Issue 6
******************************************

Petr Pisar via curl-users

2021-05-18 18:09:32 UTC

Permalink

Post by Mac-Fly via curl-users

Post by Petr Pisar via curl-users
You need to compare the whole output (as I showed).
Especially debugging of the TLS version, HTTP version,
and HTTP headers.

www.audacityteam.org has 6 different IP addresses. You can you use "--resolve
www.audacityteam.org:443:172.67.74.133" option to enforce connecting to
172.67.74.133 IP address.

Post by Mac-Fly via curl-users
Then it does no change.

I did not know that.

Post by Mac-Fly via curl-users
One IP points to Cloudfare (the windows one), the
other does points to audacity (the Linux one).

No. Both belongs to Cloufare. I tried both of them and both of them behave the
same way for me as in your Linux example. Including TLS and HTTP protocol version
and HTTP headers.

Post by Mac-Fly via curl-users
So I _think_ that already at
that point something was different for an unknown reason. :-/

TLS library, SNI support, maybe other things which cannot be seen from the
log. Or it could be IP addess of the client. (Cloudfare can respond differently
to the same requests arrived from different IP address.)

Just for your reference I have:

$ curl -V
curl 7.76.1 (x86_64-pc-linux-gnu) libcurl/7.76.1 GnuTLS/3.7.1 zlib/1.2.11 libidn2/2.3.0 nghttp2/1.41.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL TLS-SRP UnixSockets

I'm also short of ideas.

-- Petr

bruce via curl-users

2021-05-18 19:15:02 UTC

Permalink

Hi.

What's the code/snippet for your testing? Assuming you're doing a
quick/dirty code test, and not just a curl example from the cmd line.

On Tue, May 18, 2021 at 2:16 PM Petr Pisar via curl-users

Post by Petr Pisar via curl-users

Post by Mac-Fly via curl-users

Post by Petr Pisar via curl-users
You need to compare the whole output (as I showed).
Especially debugging of the TLS version, HTTP version,
and HTTP headers.

www.audacityteam.org has 6 different IP addresses. You can you use "--resolve
www.audacityteam.org:443:172.67.74.133" option to enforce connecting to
172.67.74.133 IP address.

Post by Mac-Fly via curl-users
Then it does no change.

I did not know that.

Post by Mac-Fly via curl-users
One IP points to Cloudfare (the windows one), the
other does points to audacity (the Linux one).

No. Both belongs to Cloufare. I tried both of them and both of them behave the
same way for me as in your Linux example. Including TLS and HTTP protocol version
and HTTP headers.

Post by Mac-Fly via curl-users
So I _think_ that already at
that point something was different for an unknown reason. :-/

TLS library, SNI support, maybe other things which cannot be seen from the
log. Or it could be IP addess of the client. (Cloudfare can respond differently
to the same requests arrived from different IP address.)
$ curl -V
curl 7.76.1 (x86_64-pc-linux-gnu) libcurl/7.76.1 GnuTLS/3.7.1 zlib/1.2.11 libidn2/2.3.0 nghttp2/1.41.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL TLS-SRP UnixSockets
I'm also short of ideas.
-- Petr
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etique