Error 1020

[Jeremy Roussak via curl-users]

First post here - inexperienced curl user!

Mac, Catalina.

I have an AppleScript which uses “do shell script” to run curl and then fiddles with the returned html. It’s been working perfectly well for quite a while but over the last couple of weeks has been failing with “error code 1020”. I get the same error if I run curl in Terminal. But if I paste the URL into a browser, the page loads fine.

The URL is “https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only <https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only>” but other URLs into that site behave in the same way.

I assume they’ve changed something to prevent scraping; my needs are just to look at the prices of a few items once a week, so it shouldn’t upset them too much.
Does curl have options which would make it appear to be Safari/Chrome/Firefox/other? I’ve tried —user-agent but it made no difference. Any ideas gratefully received.


Jeremy

Jeremy Roussak
Photography: jbr.is

[Dan Fandrich via curl-users]

Post by Jeremy Roussak via curl-users
Does curl have options which would make it appear to be Safari/Chrome/Firefox/
other? I’ve tried —user-agent but it made no difference. Any ideas gratefully
received.

--user-agent (copying a recent Firefox version) makes that URL work for me. If
you're still having issues, it's possible they've implemented further
restrictions, for example, per IP address. It looks like the site is behind
a Cloudflare CDN, so they're probably fairly limited in adding any
further sophisticated custom checks.

Dan
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.

[Paul Gilmartin via curl-users]

Post by Jeremy Roussak via curl-users
Mac, Catalina.
I have an AppleScript which uses “do shell script” to run curl and then fiddles with the returned html. It’s been working perfectly well for quite a while but over the last couple of weeks has been failing with “error code 1020”. I get the same error if I run curl in Terminal. But if I paste the URL into a browser, the page loads fine.
The URL is “https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only” but other URLs into that site behave in the same way.

Interesting. On Mojave command line I get:
509 $ curl https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
error code: 1020

With a fairly complex wrapper script I get:
curl --location --fail --show-error --header 'From: ***@AIM.com' --remote-time --dump-header ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only.headers --output ./sony_alpha_a7r_iv_mirrorless_digital_camera_body_only https://www.e-infin.com/uk/item/3999/sony_alpha_a7r_iv_mirrorless_digital_camera_body_only
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 16 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 403

511 $ curl --version
curl 7.76.1 (x86_64-apple-darwin18.7.0) libcurl/7.76.1 (SecureTransport) OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

Post by Jeremy Roussak via curl-users
I assume they’ve changed something to prevent scraping; my needs are just to look at the prices of a few items once a week, so it shouldn’t upset them too much.
Does curl have options which would make it appear to be Safari/Chrome/Firefox/other? I’ve tried —user-agent but it made no difference. Any ideas gratefully received.

-- gil



-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se

[Daniel Stenberg via curl-users]

Post by Jeremy Roussak via curl-users
over the last couple of weeks has been failing with “error code 1020”.

In addition to what the others have said, I'd like to clarify that:

The "error code 1020" text is content from the server - it's the response body
data. It is not something curl says itself.

(Adding "--trace-ascii -" to the command line makes this clear.)