curl-users--insecure (Daniel Stenberg)

[Timothe Litt]

Message: 3
Date: Wed, 23 Aug 2017 10:08:37 +0200 (CEST)
Subject: --insecure
Content-Type: text/plain; format=flowed; charset=US-ASCII
Hi friends,
On github alone, "curl --insecure" is used in source code at least 117,000
times. With a possible addition of about 196,000 instances where "curl -k" is
used.
Some of these use cases are probably totally legit, especially when you get
things from localhost or similar, but many of them should probably rather make
the connection to the self-signed server secure by using a cacert for it.
Is there anything we can do to reduce the use of insecure SSL connections done
by curl in the world?

Yes, see below.

https://github.com/curl/curl/pull/1821

I don't see how this helps. I know I specified --insecure - confirming
it doesn't incent me to do otherwise.
A few thoughts:

In my experience, the reason for the common use of --insecure is that
it's too painful to add trust for a self-signed certificate. Getting it
from the server and adding it to a trust store is a multi-step, arcane
process. And once there, it probably doesn't persist. (E.g., openssl
s_client -connect foo.example.org:443 </dev/null | sed ... find and add
to ca_bundle/path [hashing the latter] - and the next cert bundle
removes it. Or add explicit --capath/--cacert to every curl command
that touches such a host. In any case, I ran my curl command; it
failed. I added trust. I try again. Three phases, probably a dozen
commands. Hey, I just want my file... what are the chances I'm being
spoofed? I get paid for results - and it's snack time.) Don't tell me
that a smart guy can do it simpler or can script the process. That's
true - but you shouldn't have to be "smart" to securely get a file. Or
have time to spare.

Things to consider:

a) Encourage DANE (and support it in curl) - though whether curl support
would drive adoption is open

b) Take a leaf from ssh's book: add a simple mechanism for retrieving a
server's certificate and adding it to a trust store. The trust store
could be curl-specific (~/.curl/trusted_certificate_bundle or
~/.curl/trusted_certificate_path/), or add it to the SSH cafile/capath
(with sufficient privs). In the latter case, mk-ca-bundle.pl (or
equivalent) needs a mechanism to include locally-trusted certs; I
suspect adding to capath is easiest.

The most likely to be accepted model would be that on a certificate
failure, there's a prompt similar to ssh's:
The self-signed certificate offered by foo.example.org is not trusted.
Issuer/subject: /C:AQ/L:McMurdo/CN:foo.example.org Valid 1-Apr-2001
thru 31-Mar-2037
12 bit RSA key. Fingerprint: BEEFDAD..., Not revoked.
Do you want to continue connecting? [y/N/a]
Where 'a' adds to (some) trust store.

If the trust store is curl-private, there need to be facilities to list
the content and to revoke trust; probably also purge expired/revoked.

Another model is to have curl (or a separate utility) contact a server
with the sole purpose of obtaining its certificate and adding it to a
trust store. e.g curl --trust-certificate foo.example.org:443

There are quite a few questions to answer about the details of
implementation, among them:

* Should non self-signed certificates be included? (E.g. a new or
local CA) - if so, how far up the trust chain does one go to find
the certificate to add? (The leaf? The first intermediate? The
root?)
* What statuses should be allowed? Are expired certs OK? What if CRL
or OCSP can't be retrieved or indicate revocation/suspension?
* What if this host already has a different certificate registered?
Is it an update, or a MITM attack?
* What permissions are acceptable for the trust store? (E.g. o=w is
almost certainly a bad thing; g=w may be OK) Should it have a
checksum or some other tamper detection mechanism? A cron job that
mails changes to the owner? [You don't want silent trust added...]
* Should non-interactive jobs have a --ok-to-trust option to skip the
prompt? (Test suites would like it). If so, should interactive users?
* How does this interact with --quiet?
* Is this added trust port-specific? (Many, but not all self-signed
certs are for a specific service - e.g. TLS or SMTPS. Restricting
trust to the specific service(s) that a user approved limits exposure.)

There are others - the trick is finding the right balance between "it's
easy" and "it's secure in all cases".

Anyhow, not fully thought out, but perhaps this rough outline will
prompt further development.

Hopefully useful.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

[Daniel Stenberg]

I don't see how this helps. I know I specified --insecure - confirming it
doesn't incent me to do otherwise.

I think a lot of people use -k pasted from somewhere without knowing what it
means - the stackoverflow effect. I also think that curl is used insecurely
from within lots of scripts without all users of these scripts knowing that,
and these users could now spot the warning.

This said, I actually don't think this little warning will change much as we
users are well trained since long at ignoring warnings - especially if
everything still seems to work fine.

a) Encourage DANE (and support it in curl) - though whether curl support
would drive adoption is open

That's a looong term vision. Not to mention how this then expects people to
put their self-signed certs into the DNS, which most certainly will always be
more complicated than just keep using --inscure...

We've had DANE mentioned in the TODO document for years and while we've seen
at least two efforts in making it reality, it is a big chunk of complicated
work. DANE is not exactly getting much love from the browser world so it's not
likely to be a real solution for HTTPS within the forseeable future.

b) Take a leaf from ssh's book: add a simple mechanism for retrieving a
server's certificate and adding it to a trust store.

That's indeed an interesting idea! That's a long list of issues to deal with
though, that are things that makes this much more complicated than the "easy"
case ssh has with known hosts.

In addition to that great list of challenges we also have

* what TLS library backend is used
* HTTPS proxy support (for some TLS backends)

I firmly agree that it would be really neat with a start at this that at least
helps the user more towards getting rid of the --insecure option.

[Timothe Litt]

Post by Daniel Stenberg
This said, I actually don't think this little warning will change much
as we users are well trained since long at ignoring warnings -
especially if everything still seems to work fine.

Agree. I don't think it's harmful, except in that it adds to the 'you
can ignore warnings' training. I would prefer to see curl attempt to
verify the host (regardless of -k), and only retry (and warn) without
verification if the first attempt fails and -k was specified. That way
the user is warned only when the -k actually has an effect. This will
prevent warning overload.

Post by Daniel Stenberg

Post by Timothe Litt
a) Encourage DANE (and support it in curl) - though whether curl
support would drive adoption is open

That's a looong term vision. Not to mention how this then expects
people to put their self-signed certs into the DNS, which most
certainly will always be more complicated than just keep using
--inscure...

But that can be automated too, especially with DNS UPDATE. But yes,
it's non-trivial. However, it is the more secure approach - DNSSEC
protects the certs & their authenticity, and there's only one place to
update - rather than a lot of files on a lot of systems.

Post by Daniel Stenberg
making it reality, it is a big chunk of complicated work.

Agree.

Post by Daniel Stenberg
DANE is not exactly getting much love from the browser world so it's
not likely to be a real solution for HTTPS within the forseeable future.

Agree - but someone has to go first. Browsers aren't anxious to support
because there's no demand; there's no demand because browsers don't
support it. curl could be helpful - but I agree it's a long & hard
road. Which is why I didn't say much about it and suggested plan B as a
pragmatic approach.

Post by Daniel Stenberg

Post by Timothe Litt
b) Take a leaf from ssh's book: add a simple mechanism for retrieving
a server's certificate and adding it to a trust store.

That's indeed an interesting idea! That's a long list of issues to
deal with though, that are things that makes this much more
complicated than the "easy" case ssh has with known hosts.

PKI is inherently complicated. known_hosts/authorized_keys is rather
crude hack - it doesn't scale well, and updating on a key/cipher
change/revocation is painful - every host/client pair has to be updated.

BUT: It's "good enough". Human intervention is minimal, yet it enforces
(more or less) secure behavior. And despite its problems, it has been
effective for a long time. It makes people think, but doesn't get in
the way once trust is established.

This thought was to apply "good enough" thinking to the curl world. The
nice thing about this model is that a user doesn't have to repeat or
modify the command. It's a once/host (certificate) confirmation.

Post by Daniel Stenberg
In addition to that great list of challenges we also have

They're more questions than challenges - implementing any answer doesn't
appear to be particularly difficult.

Post by Daniel Stenberg
* what TLS library backend is used

That doesn't seem especially hard to abstract. They all take a
cafile/capath, so appending a local trust store should be doable, though
the mechanisms vary. It's work - but I don't see an architectural issue.

Post by Daniel Stenberg
* HTTPS proxy support (for some TLS backends)

I don't see how that's involved; at most it's a minor issue. If curl
talks to a proxy, it's the proxy's certificate that has to go to the
local trust store. The proxy has to deal with trust with respect to the
'real' server; this doesn't change that. If curl IS a proxy, the same
mechanism for injecting additional trust used for server verification
can be used for client verification.

Post by Daniel Stenberg
I firmly agree that it would be really neat with a start at this that
at least helps the user more towards getting rid of the --insecure
option.

This could be built incrementally - take the simplest answers to work
out the UI. E.g. a proof of concept might:

* Restrict to self-signed (this is the biggest simplification)
* Warn about not in validity period; ignore revocation for the first pass.
* Warn in prompt if certificate trusted for a host changes
* Reject files/directories with o=w - maybe a compile mask of
forbidden permissions
* It overrides --quiet interactively - human intervention is required,
and the alternative is to fail.
* Make trust default to port-specific for self-signed; allow an
override or wildcard. Default to '*' when non-self-signed support
is added
* Implement only one backend; stub out the others.
* Don't worry about proxy support

Once you have confidence & some user feedback, go back and provide more
capabilities/options.

But that's just my impression of a reasonable approach; I'm not deeply
familiar with curl internals, and this won't be on my project list.

One more question - should curl's local trust store interact with the
browsers'/system's certificate trust store? If so, which one(s) and
how? [e.g., when a user adds a certificate to Firefox, can/should curl
trust it? Can/should curl's trust store be the same as
Firefox/Windows/the TLS library? Or should curl update those?
Optionally? Provide import/export?

Anyhow, these are all more design questions than ones for a users' list.

I don't have time to contribute coding/detailed design to this, but
would be able to review a design if you elect to proceed. I opened
github issue 1822 so that future work/discussion/commits can be linked
to this thread, and I'll see notifications. (I don't read curl-users
regularly.)

Glad to have provoked some thought.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

[Alex Bligh via curl-users]

b) Take a leaf from ssh's book: add a simple mechanism for retrieving a server's certificate and adding it to a trust store.

That's indeed an interesting idea! That's a long list of issues to deal with though, that are things that makes this much more complicated than the "easy" case ssh has with known hosts.

Even a --getcert flag which retrieves the cert, and a --manualcert flag which accepts that particular cert (or similar) would be helpful. I don't think that should be much harder than specifying ones own trust store to the SSL library.
--
Alex Bligh





-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx

[Timothe Litt]

Post by Alex Bligh via curl-users
Even a --getcert flag which retrieves the cert, and a --manualcert flag which accepts that particular cert (or similar) would be helpful. I don't think that should be much harder than specifying ones own trust store to the SSL library.

There is no need for --manualcert - --cacert will do.

You can retrieve a server's certificate with this (or similar) magic:
openssl s_client -connect www.example.net:443 </dev/null 2>&1 | \
sed -ne'/--BEGIN/,/--END/p' >www.example.net.pem

and then use curl with
curl --cacert www.example.net.pem ...
EVERY time you connect to that particular server.

And as I noted earlier, that can be scripted by people willing to invest
the time.
Such a script can be arbitrarily clever about getting & caching
certificates; using
the default trust store when possible, and invoking curl.

But as I wrote previously, this misses the point.

If the goal is to eliminate the use of --insecure, the problem to be
solved is that
any variation of this process is more painful than adding --insecure to
a command
-- or to your alias for curl! Even '--cacert ~/.certs/www.example.com'
is more
typing than '-k' (or '--insecure').

People are lazy - or more charitably, want to make the most efficient
use of their time.
Users add -k because it appears to trivially solve the 'problem' of curl
failing to connect
to some servers. So any complicated scheme is unattractive, and will
not be adopted
by the majority of users.

To eliminate (or drastically reduce) --insecure, curl needs to make
adding and using
trust painless - at most a one-time confirmation. This is why the SSH
'known_hosts'
model is attractive. In its simplest form (and most common use case),
it requires no
additional command line options, and it only asks for confirmation ONCE
per server
certificate. The complexities are hidden from the user. There are
implementation
details for the curl developers to work through; but better them (once)
than users
(every time).

To remove the incentive to type -k, the solution needs to make things
simpler for the
user, not add more work.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

[Daniel Stenberg]

To eliminate (or drastically reduce) --insecure, curl needs to make adding
and using trust painless - at most a one-time confirmation. This is why the
SSH 'known_hosts' model is attractive.

I started to jot down the exact steps that would be needed in order to have
curl do a "trust on first use" approach properly.

Here's what I have so far:

https://github.com/curl/curl/wiki/Trust-On-First-Use

Most of the building stones are already available.