The Problem

I've got one server in a farm which is suddenly unable to correctly handle SSL certificates. Attempting to do a curl command like curl -v https://google.com results in:

curl -v https://google.com

* About to connect() to google.com port 443 (#0)

*   Trying 74.125.137.101... connected

* successfully set certificate verify locations:

*   CAfile: none

  CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS alert, Server hello (2):

* SSL certificate problem, verify that the CA cert is OK. Details:

error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Using openssl s_client is a little more detailed.

# openssl s_client -host google.com -port 443

CONNECTED(00000003)

depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

verify error:num=20:unable to get local issuer certificate

Things Tried So Far

• Reinstalling ca-certificates - already have the latest available version, according to aptitude, Version: 20130906ubuntu0.12.04.1.




• Reconfiguring ca-certificates via dpkg-reconfigure. This appears to rehash the /etc/ssl/certs folder but has no effect on the problem.




• Using update-ca-certificates --fresh to regenerate the symbolic links in that folder




• Grabbing the latest Mozilla ca bundle from curl.haxx.se - by putting that .pem file in /etc/ssl/certs and running the update command.

Weirdness

The certificate that curl claims it cannot find is indeed in the certification path.

# ls -l /etc/ssl/certs/*Geo*



lrwxrwxrwx 1 root root 57 Apr  7 15:57 /etc/ssl/certs/GeoTrust_Global_CA.pem -> /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt

...

The certificate file referenced has the same permissions as every other box on my network, namely 644.

# ls -l /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt

-rw-r--r-- 1 root root 1216 Feb 20 11:49 /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt

Other secure sites such as Github show identical issues with different certificates. I am running the absolute latest version available of all packages for Ubuntu 12.04.4, including curl, openssl, and ca-certificates.

What's going on here?