Cipher Security
Hardening TLS and SSH

by Charles Fisher

Introduction

Encryption and secure communications are critical to our life on the Internet. Without the ability to both authenticate and preserve secrecy, we cannot engage in commerce, nor can we trust the words of our friends and colleagues.

It comes as some surprise, then, that insufficient attention has been paid in recent years to strong encryption, and many of our "secure" protocols have been easily broken. The recent Heartbleed, POODLE, CRIME, and BEAST exploits put at risk our trust in our networks and in one another.

Gathered here are best practice approaches to close known exploits and strengthen communication security. These recommendations are by no means the final word on the subject - the goal here is to draw focus upon continuing best practice.

Please note that many governments and jurisdictions have declared encryption illegal and, even where allowed, law enforcement has become increasingly desperate with growing opaque content. Ensure that both these techniques and the content that they protect are not overtly illegal.

This article focuses on Oracle Linux versions 5, 6, and 7 and close brethren (RedHat, CentOS, and Scientific Linux). From here forward I will refer to these platforms simply as V5, V6, and V7. Oracle's V7 only runs on the x86_64 platform, so that's this article's primary focus.

These products can rightly be considered defective, in spite of constant vendor patches. The library designers would likely argue that their place is to implement mechanism, not policy, but the resulting products are nonetheless critically flawed. Here is how to fix them.

Strong Ciphers in TLS

The Transport Layer Security (TLS) protocols emerged from the older Secure Sockets Layer (SSL) that originated in the Netscape browser and server software.

It should come as no surprise that SSL must not be used in any context for secure communications. The last version, SSLv3, was rendered completely insecure by the recent POODLE exploit. No version of SSL is safe for secure communications of any kind - the design of the protocol is fatally flawed, and no implementation of it can be secure.

TLS version 1.0 is also no longer safe. The immediate preference for secure communication is the modern TLS version 1.2 protocol which, unfortunately, is not (yet) widely used. Despite the lack of popularity, prefer 1.2 if you value security.

Yet even with TLS version 1.2, there are still a number of important weaknesses which must be addressed to meet current best practice as specified in RFC 7525:

There are several implementations of the TLS protocols, and three competing libraries are installed on Oracle Linux systems by default: OpenSSL, NSS, and GnuTLS. All of these libraries can provide Apache with TLS for HTTPS. It has been asserted that GnuTLS is of low code quality and unsafe for binary data, so exercise special care with this particular library in critical applications. This article will focus only upon OpenSSL, as it is the most widely used.

For TLS cipher hardening under OpenSSL, I turn to Hynek Schlawack's website on the subject. He lists the following options for the SSL configuration of the Apache webserver:

SSLProtocol ALL -SSLv2 -SSLv3
SSLHonorCipherOrder On
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

This configuration focuses upon the Advanced Encryption Standard (AES) - also known as the Rijndael cipher (as named by the cipher's originators), with 3DES as a fallback for old browsers. Note that 3DES is generally agreed to provide 80 bits of security, and it is also quite slow. These characteristics do not meet the above criteria, but we allow the legacy Data Encryption Standard (Triple-DES) cipher to provide continued access to older browsers.

On an older V5 system (which does not implement TLS 1.1 or 1.2 in OpenSSL), the list of acceptable ciphers is relatively short:

$ cat /etc/oracle-release /etc/redhat-release 
Oracle Linux Server release 5.11
Red Hat Enterprise Linux Server release 5.11 (Tikanga)

$ openssl ciphers -v 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS'
DHE-RSA-AES256-SHA   SSLv3 Kx=DH  Au=RSA Enc=AES(256)  Mac=SHA1
DHE-RSA-AES128-SHA   SSLv3 Kx=DH  Au=RSA Enc=AES(128)  Mac=SHA1
EDH-RSA-DES-CBC3-SHA SSLv3 Kx=DH  Au=RSA Enc=3DES(168) Mac=SHA1
AES256-SHA           SSLv3 Kx=RSA Au=RSA Enc=AES(256)  Mac=SHA1
AES128-SHA           SSLv3 Kx=RSA Au=RSA Enc=AES(128)  Mac=SHA1
DES-CBC3-SHA         SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1

Please note that TLS version 1.1 introduced new defenses against CBC exploits. CBC is only used above with the 3DES cipher, which calls into question the use of 3DES with TLS version 1.0. Removing 3DES and/or enforcing a minimal protocol of TLS version 1.1 might be required if your security concerns are very grave, but this will adversely impact compatibility with older browsers. Banishing CBC on OpenSSL 0.9.8e will leave you with few working ciphers indeed.

On V7, the list of allowed ciphers is considerably longer:

$ cat /etc/oracle-release /etc/redhat-release 
Oracle Linux Server release 7.1
Red Hat Enterprise Linux Server release 7.1 (Maipo)

$ openssl ciphers -v 'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AES(256) Mac=SHA384
ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH  Au=RSA  Enc=AES(256) Mac=SHA1
ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH Au=ECDSA Enc=AES(256) Mac=SHA1
ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256) Mac=SHA384
ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256) Mac=SHA384
ECDH-RSA-AES256-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(256) Mac=SHA1
ECDH-ECDSA-AES256-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256) Mac=SHA1
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH  Au=RSA  Enc=AES(256) Mac=SHA256
DHE-RSA-AES256-SHA      SSLv3 Kx=DH    Au=RSA  Enc=AES(256) Mac=SHA1
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH      Au=ECDSA Enc=AES(128) Mac=SHA256
ECDHE-RSA-AES128-SHA    SSLv3 Kx=ECDH  Au=RSA  Enc=AES(128) Mac=SHA1
ECDHE-ECDSA-AES128-SHA  SSLv3 Kx=ECDH Au=ECDSA Enc=AES(128) Mac=SHA1
ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128) Mac=SHA256
ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128) Mac=SHA256
ECDH-RSA-AES128-SHA     SSLv3 Kx=ECDH/RSA Au=ECDH Enc=AES(128) Mac=SHA1
ECDH-ECDSA-AES128-SHA   SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128) Mac=SHA1
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH  Au=RSA  Enc=AES(128) Mac=SHA256
DHE-RSA-AES128-SHA      SSLv3 Kx=DH    Au=RSA  Enc=AES(128) Mac=SHA1
ECDHE-RSA-DES-CBC3-SHA  SSLv3 Kx=ECDH  Au=RSA  Enc=3DES(168) Mac=SHA1
ECDHE-ECDSA-DES-CBC3-SHA SSLv3 Kx=ECDH Au=ECDSA Enc=3DES(168) Mac=SHA1
ECDH-RSA-DES-CBC3-SHA   SSLv3 Kx=ECDH/RSA Au=ECDH Enc=3DES(168) Mac=SHA1
ECDH-ECDSA-DES-CBC3-SHA SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=3DES(168) Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH    Au=RSA  Enc=3DES(168) Mac=SHA1
AES256-GCM-SHA384       TLSv1.2 Kx=RSA Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA Au=RSA  Enc=AES(256) Mac=SHA256
AES256-SHA              SSLv3 Kx=RSA   Au=RSA  Enc=AES(256) Mac=SHA1
AES128-SHA256           TLSv1.2 Kx=RSA Au=RSA  Enc=AES(128) Mac=SHA256
AES128-SHA              SSLv3 Kx=RSA   Au=RSA  Enc=AES(128) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA   Au=RSA  Enc=3DES(168) Mac=SHA1

If possible under your release of Apache, also issue an "SSLCompression Off" directive. Compression should not be used with TLS because of the CRIME attack.

If you have connectivity problems with web clients, try disabling the Cipher Order directive first. Custom HTTP clients may not fully implement the TLS negotiation, which might be solved by allowing the client to pick the cipher.

The cipher selector above also prevents any exploit of the "Logjam" (weak Diffie-Hellman primes) security flaw that has recently surfaced. If your version of Apache supports an alternate dh-prime configuration, it is recommend that you follow this procedure:

openssl dhparam -out /home/httpd/conf/dhparams.pem 2048

Then add the following line to your Apache SSL configuration:

SSLOpenSSLConfCmd DHParameters "/home/httpd/conf/dhparams.pem"

Ensure that you have appropriate permissions on your dhparams.pem file, and note that V5 does not support this configuration.

When you have applied these configuration changes to your Apache web server, use the SSLlabs.com scan tool to rate your server. If you are on an older V5 platform that uses the OpenSSL 0.9.8e release, the grade assigned to your server should be a "B" - your final security grade will be higher if you are on a later release.

It is also important to restart your TLS webserver for key regeneration every day, as is mentioned in the Apache changelog: "Session ticket creation uses a random key created during web server startup and recreated during restarts. No other key recreation mechanism is available currently. Therefore using session tickets without restarting the web server with an appropriate frequency (e.g. daily) compromises perfect forward secrecy." This information is not well known, and has been met with some surprise and dismay in the security community: "You see, it turns out that generating fresh RSA keys is a bit costly. So modern web servers don't do it for every single connection. In fact, Apache mod_ssl by default will generate a single export-grade RSA key when the server starts up, and will simply re-use that key for the lifetime of that server."

Note that Hynek Schlawack's site provides configuration instructions for nginx and HAProxy in addition to Apache. Several other applications allow a custom cipher specification - two that I will mention here are stunnel and sendmail.

The stunnel "TLS shim" allows cleartext socket applications to be transparently wrapped in TLS encryption. In your stunnel configuration, specify the "cipher=" directive with the above string to force stunnel to best practice. Also, on the V7 platform, supply the "fips=no" directive, otherwise you will be locked to the TLS version 1 protocol with the message 'sslVersion = TLSv1' is required in FIPS mode.

The sendmail transport agent has received recent patches to fully specify ciphers. You can add the following options to your sendmail.cf to force best practice ciphers:

O CipherList=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_CIPHER_SERVER_PREFERENCE
O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_CIPHER_SERVER_PREFERENCE

With these settings you will see encryption information in your mail logs:

May 12 10:17:58 myhost sendmail[1234]: STARTTLS=client, relay=mymail.linuxjournal.com., version=TLSv1/SSLv3, verify=FAIL, cipher=AES128-SHA, bits=128/128
May 12 10:38:28 myhost sendmail[5678]: STARTTLS=client, relay=mymail.linuxjournal.com., version=TLSv1/SSLv3, verify=FAIL, cipher=AES128-SHA, bits=128/128

The "verify=FAIL" indicates that your keys are not signed by a certificate authority (which is not as important for an SMTP server). The encryption is listed as AES128-SHA.

For a public mailserver, it is important to be more permissive with the allowed ciphers to prevent SMTP sessions from going cleartext. Behind a corporate firewall, however, it is likely better to force strong TLS ciphers more rigorously.

It is also important to promptly apply vendor patches for TLS. It was recently discovered that later TLS versions were using SSLv3 padding functions directly in violation of the standards, rendering the latest versions vulnerable (this was more a concern for NSS than OpenSSL). Prompt patching is a requirement for a secure TLS configuration.

I would like to thank Hynek Schlawack for his contribution to and thoughtful commentary on TLS security.

Strong Ciphers in SSH

It is now well-known that (some) SSH sessions can be decrypted (potentially in real time) by an adversary with sufficient resources. SSH best practice has changed in the years since the protocols were developed, and what was reasonably secure in the past is now entirely unsafe.

The first concern for an SSH administrator is to disable protocol 1 as it is thoroughly broken. Despite a stream of vendor updates, older Linux releases maintain this flawed configuration, requiring the system manager to remove it by hand. Do so by ensuring "Protocol 2" appears in your sshd_config, and all reference to "Protocol 2,1" is deleted. Encouragement is also offered to remove it from client SSH applications also, in case a server is inaccessible or otherwise overlooked.

For further hardening of Protocol 2 ciphers, I turn to the Stribika SSH Guide. These specifications are for the very latest versions of SSH, and directly apply only to Oracle Linux 7.1.

For older versions of SSH, I turn to the Stribika Legacy SSH Guide, which contains relevant configuration details for Oracle Linux 5, 6 and 7.

There are only two recommended sshd_config changes for Oracle Linux 5:

Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-ripemd160

Unfortunately, the PUTTY suite of SSH client programs for Win32 are incompatible with the "MACs hmac-ripemd160" setting, and will not connect to a V5 server when this configuration is implemented. As PUTTY has quietly become a corporate standard, this likely is an insurmountable incompatibility, so most enterprise deployments will implement only the Cipher directive.

Version 0.58 of PUTTY also does not implement the strong AES-CTR ciphers (these appear to have been introduced in the 0.60 release) and likewise will not connect to an SSH server where they are used exclusively. It is strongly recommended that you implement the Cipher directive, as it removes RC4 (arcfour) which is totally inappropriate for modern SSH. It is not unreasonable to expect corporate clients to run the latest versions of PUTTY, as new releases are trivially easy to install.

Oracle Linux 5 has a role of special importance as it is the underlying OS for the Linux version of the Oracle Exadata architecture (the alternate OS being Solaris). If you are an Exadata customer, confirm with Oracle that you will retain vendor support if you change cipher and protocol settings on a supported Exadata appliance.

V5's default SSH ciphers will will be pruned especially hard:

$ man sshd_config | col -b | awk "/Ciphers/,/ClientAlive/"

Ciphers

Specifies the ciphers allowed for protocol version 2.  Multiple
ciphers must be comma-separated.  The supported ciphers are
3des-cbc, aes128-cbc, aes192-cbc, aes256-cbc,
aes128-ctr, aes192-ctr, aes256-ctr, arcfour128,
arcfour256, arcfour, blowfish-cbc, and cast128-cbc.  The
default is

	aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,
	aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,
	aes256-cbc,arcfour

It is possible to install a newer version of OpenSSH on V5, but it is not trivial. Attempting to compile the latest release results in the following error:

error: OpenSSL >= 0.9.8f required (have "0090802f (OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008)")

It is possible to compile OpenSSH without OpenSSL dependencies with the following:

--without-openssl Disable use of OpenSSL; use only limited internal crypto **EXPERIMENTAL**

Enterprise deployments are likely unwilling to use experimental code, so we will not go into further details. If you obtain binary RPMs for upgrade, ensure that you know how they were produced.

Oracle Linux 7 lacks a few ciphers from the latest releases of SSH, and differs only slightly from the recommended settings:

HostKey /etc/ssh/ssh_host_rsa_key
Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
KexAlgorithms diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

Oracle Linux 7.1 can be configured exactly as recommended, including the new ed25519 hostkey:

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

The Stribika Guide immediately dismisses the 3DES cipher, which is likely reasonable as it is slow and relatively weak, but also goes to some length to criticize the influence of NIST and the NSA. In the long view, this is not entirely fair, as the U.S. government's influence over the field of cryptography has been largely productive. To quote cryptographer Bruce Schneier, "It took the academic community two decades to figure out that the NSA 'tweaks' actually improved the security of DES... DES did more to galvanize the field of cryptanalysis than anything else." Despite unfortunate recent events, modern secure communication has much to owe to the Data Encryption Standard and those who were involved in its introduction.

Stribika levels specific criticism "...advising against the use of NIST elliptic curves because they are notoriously hard to implement correctly. So much so, that I wonder if it's intentional. Any simple implementation will seem to work but leak secrets through side channels. Disabling them doesn't seem to cause a problem; clients either have Curve25519 too, or they have good enough DH support. And ECDSA (or regular DSA for that matter) is just not a good algorithm, no matter what curve you use."

In any case, there is technical justification for leaving 3DES in TLS, but removing it from SSH - there is a greater financial cost when browsers and customers cannot reach you than when your administrators are inconvenienced by a software standards upgrade.

If you are using ssh-agent with a private key, you can strengthen the encryption of the password on the key using this method documented by Martin Kleppmann with PKCS#8. Here is the procedure summarized from the author:

cd ~/.ssh

mv ~/.ssh/id_rsa ~/.ssh/id_rsa.old

openssl pkcs8 -topk8 -v2 des3 -in ~/.ssh/id_rsa.old -out ~/.ssh/id_rsa

chmod 600 ~/.ssh/id_rsa

The author estimates that this procedure provides the equivalent benefit of adding two extra characters to the password. It is important to note, however, that the putty agent is not able to read the new format produced here. If you use pagent with putty (or expect to), convert your OpenSSH key to pagent first, then run this procedure, assuming that retention of your key in both formats is allowed. It is likely wise to retain a copy of the original private key on offline media. It is also important to note that this procedure does not add any extra protection from a keylogger.

User SSH keypairs are likely superior to passwords for many aspects of security. SSH servers cannot enforce password standards on remote keys (minimum password length, change frequency, reuse prevention, etc.), and there are definite risks in forwarding the ssh-agent that would compromise server security. If you allow your users to authenticate with SSH keypairs that they generate, you should understand how they can be (ab)used.

Finally, be aware that keystroke delay duration can be used as a side channel exploit in SSH via the application of the Viterbi Algorithm. Interactive SSH sessions are more revealing of content than most expect, and should be avoided for those with high security requirements. Send batches of ssh commands, or implement a bandwidth "fuzzer" in a secondary session on the same channel if an interactive session is required but security is critical. Of particular note:

I would like to thank Stribika for his contribution to and thoughtful commentary on SSH security.

Unbreakable Encryption

While the best practices above are helpful, these protocols have been entirely inadequate in assuring private communication channels, and they have been broken many times.

If your needs for secure communication are so dire that any risk of interception is too great, then you should likely consider encryption tools that do not appear to have as yet been broken.

A careful parse of recent evidence indicates that the Gnu Privacy Guard implementation of Pretty Good Privacy (PGP) continues to present insurmountable difficulty to eavesdropping and unauthorized decryption.

This utility is installed in all recent versions of Oracle Linux by default. It should be your first thought for secure communications, and you should realize that all the techniques described above are compromises for the sake of expedience.