The Rise of Opaque encryption Resets the Network Security Industry

{Note this writeup was done in summer of 2017 while I was at Skyport Systems as Founder and VP of Engineering}

During my time at Cisco and Juniper (1999-2013), our marketing departments unveiled many campaigns around the “Intelligent Network.” To deliver on this message, we kept building products and features that would use stateful views of deep packet data to help reinforce the marketing story about the value of the network. During the same time, a multi-billion-dollar industry grew out of providing visibility into and control of enterprise traffic, including next-generation firewalls, features in routers, IPS/IDS, UTM, web proxies, data loss prevention, anti-virus, and secure web gateways.

We knew that while uncommon at the time, that most of those features we were adding would become irrelevant if the day came that all the data flowing through our devices was encrypted, but that seemed a very far off possibility.

Fast forward to 2017. Not only is encryption here with a vengeance, it will soon cause most network traffic to go dark from the point of view of IT departments. This year the threshold was crossed with [now over 50% of web traffic using HTTPS] (https://www.wired.com/2017/01/half-web-now-encrypted-makes-everyone-safer/). We are also seeing the rise of new encryption protocols that do not accommodate man-in-the-middle (MITM) monitoring. As a result, most traffic will speed by, opaque to inspection from the standpoint of IT and security administrators.

These MITM boxes (arguably) own one side of each leg in internal data communications (with the definition that the enterprise IT department owns all client/server endpoints in its network) and for years MITM proxies terminated encrypted sessions and re-originated them, in which case everything is visible. However, moving forward with new protocols increasingly they will only be able to gather extremely superficial intelligence from the traffic headers flowing through them in clear text.

MITM today

Before launching into new developments with encryption, here are the main uses of traffic inspection today:

• Malware and botnet detection to contain the lateral spread of malware and reach-out from enterprise to command and control points
• Authentication/validation of access to critical segments and systems
• Detecting policy violations related to content and data access
• Detecting data exfiltration (theft)
• General visibility and analytics

Over the past two decades, MITM boxes evolved as invisible inspectors (transparent bumps in the wire) in the enterprise network. This model was relatively easy to operationalize and vendors provided a range of product options.

Today, there are two main scenarios for deploying MITM boxes. The left side of the diagram below shows traditional inspection of traffic between enterprise servers within their perimiter to external points like clients, SaaS and IaaS services. The right side of the diagram shows the trend over the past 5-7 years towards deploying MITM devices inside the enterprise to provide visibility and segment the enterprise internally. For these internal segmentation cases the network security boxes can intercept traffic between servers, or across boundaries that connect servers to databases and storage.

What is Opaque Crypto

Opaque cryptography covers a broad set of developments that has a heavy focus on HTTPS, but includes other protocol stacks also as shown below.

Connection Between	New Developments
Application<->Database	Mongo, Postgres, etc w TLS
Application<->Storage	SMBv3, NFS4.1 w Kerberized Crypto
Application<->Application	Increased ease of use of TLS Libraries
Client to Server	HTTP/2, TLS1.3, QUIC
Iaas to On-Premises	Encrypted VPNs with IPSEC

New developments in network traffic encryption

Without doing a deep dive on each of these new developments, we can categorize the trends that make it hard to leverage the existing proxy technologies :

• A general trend towards use of Perfect Forward Secrecy (PFS) and acceptance of it as a valid goal for encryption standards. PFS is a property of a key exchange that deliberately prevents after-the-fact decryption and defeats all passive MITM technologies.
• Increased use of non-SSL/TLS encryption protocols with no accompanying MITM infrastructure. For example, Google’s QUIC is an alternative to TLS that vendors are lagging behind (currently unable to support).
• Multi-path to improve performance. Multi-path makes decryption difficult because a single MITM box only sees a portion of the needed state.
• Compressed header state and fewer clear text headers. This makes it difficult to determine what traffic to decrypt.
  • TLS 1.3 does provide a clear text extension (SNI) to communicate a target server’s host name. While it seems useful for decryption determination, it can be easily spoofed.
• Increased use of certificate and public key pinning, which are hard blocks to traditional transparent MITM.
  • Traditionally if you read network security vendor documents the response to this issue was that SaaS vendors simply will not risk their own business by pinning.
  • However, the submissiveness of SaaS vendors is changing with a growing number of services that cross consumer and enterprise boundaries, as well as growing number of SaaS vendors that value the trust of the end users over friction with IT admins.
• TCP connection re-use (enabled through HTTP/2 and TLS1.3) complicates state machines for MITM devices.
• Technologies that are crossing enterprise perimeter that were never meant for transparently inserting network security in the middle, for example IPSEC based tunnels.

Reading through the above list, what is interesting is that while it seems reasonable that many of these developments could be countered to some degree by network security vendors, taken as a sum total the level of disruption in such a short time is staggering.

Forces resulting in Opaque Crypto

It is frequently called out that the rise of machine learning in past few years is due to the simultaneous occurrence of 3 factors : 1) significant increases in computation, 2) availability of massive data sets, and 3) Improvement in fundamental algorithms.

There is a parallel set of forces that are causing a rise of encryption in a form that removes visibility (i.e. Opaque Crypto). These forces which are detailed in following sub-sections are :

• 1. Consumer web companies now lead definition of Web protocols
• 2. The cost of encryption goes to zero
• 3. Cloud ripple effects

Force #1 : Consumer web companies now lead definition of Web protocols

At one time, vendors like Cisco, working with service providers and enterprise customers, drove the key web and internet standards. The key force driving the technology choices at end of the day was what would sell more routers, switches, and networking security boxes.

While the fact that technology gets driven for economic ends has not changed, the special interests that are now getting prioritized is that of web companies that want to minimize costs, and maximize value to their consumer customers. In the pursuit of pro-consumer profits, web properties focus on characteristics like low latency, high throughput, good user experience, and trust.

On the topic of trust, consumer web companies don’t respect the MITM model that financials so love, because in the consumer arena there is no legitimate reason to insert a MITM (only bad reasons like stealing credit card numbers). Additionally in the post Edward Snoweden era, the general negative public reaction to web companies supporting snooping of any kind has added additional pressure on these companies to resist incept models. The result is that web companies believe to earn trust it requires strong end-end security models, and in the pursuit of profit, that is what consumer web companies are delivering.

A good example of consumer web companies' hostility towards MITM architecture is a 2017 paper that includes authors from Google and Cloudflare that essentially condemns HTTPS interception.

This paper is the result of a massive investment. The authors tested a vast combination of security products and software versions from top vendors, and offer this finding in summary:

Most concerningly, 62% of traffic that traverses a network middlebox has reduced security and 58% of middlebox connections have severe vulnerabilities. We investigated popular antivirus and corporate proxies, finding that nearly all reduce connection security and that many introduce vulnerabilities

After publication of this paper, a US agency issued a CERT (Alert TA17-075A) citing it and issuing a similar warning about the risks of HTTPS interception.

In recent discussions with a household name financial about these trends, the customer noted that they were talking to Google about the disruption that standards like Google’s QUIC/TLS1.3 could cause across their industry due to these standards' implicit push against interception. While the frustration is understandable given that financials are heavily invested in the current model for monitoring/protecting their perimeter, the new direction looks clear barring major government interference.

Force #2 : The cost of encryption goes to zero

Though not strictly a cause, a drop in the cost of encryption was necessary to ease its adoption. Improvements to key encryption standards in hardware–and more importantly software–have made it easier to support very strong encryption algorithms, and at a level that performance is no longer an issue for either the client or server side of a connection.

Improvements like Intel Advanced Encryption Standard instructions speed up encryption, and there are ongoing efforts to improve all aspects of TLS performance, including making TLS a kernel service.

Force #3 : Cloud ripple effects

This force was harder to pin down, but outside of web traffic and the falling cost of encryption there was clearly at least one more force driving the rise of opaque encryption. Two example trends that are not explained by the two forces above are the recent moves towards encrypted storage traffic with NFS4.1 using kerberized encryption as well as increased use of SMB 3.0 with encryption.

So why is the above happening ? Lets start with the assumption that data and workloads without notable business value will continue quickly moving to public cloud. If you are an enterprise vendor of ISV software or systems, you are flailing around, looking to counter this trend with new features, and to push the notion that your customers should be keeping important stuff in a enterprise owned data centers. This leads to features like encrypted data at rest, as well as encrypted data traffic for storage, databases and applications.

Beyond these vendor driven features, cloud in the form of SaaS has produced thousands of enterprise SaaS services. Beyond client connections, these SaaS services most often involve agents or VMs sitting in the enterprise data center that reach out to their cloud service origin (most commonly on AWS) to make some sort of data connection and do data integrations. These SaaS services are following in the foot steps of their consumer web brethren and all will be increasingly opaque.

Finally going to the core trend of IaaS adoption in enterprise, services like AWS are resulting in many (poorly managed) enterprises sprinkling VPC VPN gateways all over their internal network. Each of these end points is rendering the outer perimeter of the enterprise meaningless as each is a bi-directional gateway between an entire world encapsulated in the AWS VPC to the heart of the data center. And again these are all encrypted traffic that is not being proxied by those HTTPS interception MITM devices.

Enterprise vendors' responses to opaque crypto

The network security industry has largely been quiet about the trends discussed here. Some companies like Bluecoat have been visible in their struggles to keep up with protocols like TLS1.3. But overall there has been little substantive industry-wide discussion of the issues or possible solutions.

One exception is Cisco, which has highlighted several issues with opaque crypto and advocated a potential solution. In mid-2017 Cisco introduced Cisco Encrypted Traffic Analytics, and they deserve credit for being one of the only established security vendors to call out this issue. They also deserve credit for publishing academic papers related to their solutions and making available software for feature extraction from packet traces, plus an analysis tool on github that seem related to their for-profit product.

So what does Cisco propose? As their graphic below shows, first they look at clear text TLS information (also known as TLS fingerprinting). This can be as simple as the order of presentation of available bulk encryption algorithms, but of course is usually more complex. Also, they offer nice real time data collection of the TLS packet streams across a variety of extracted features, including old favorites like packet size and inter-packet gap.

But while Cisco deserves credit, there seem to be fundamental issues with their offering.

• The proposal focuses on malware detection, which overlooks other goals of MITM proxies such as policy violations related to data access, detecting data exfiltration, and validating/authenticating access to management segments and systems.
• As mentioned above, TLS 1.3 and other new protocols are becoming more opaque. Cisco’s approach will likely become less relevant with the reduced clear text available with TLS1.3, the smaller set of supported cipher suites, all making TLS fingerprinting more difficult.
• Cisco’s solution seems to focus on TLS and omit technologies like IPSEC VPNs, Kerberized file transfer protocols, and QUIC.
• In the past, solutions like malware detonation were effective until bad actors learned how to do end runs around the technology. By extension, it seems likely that once hackers see the solution, they will learn to avoid TLS fingerprinting, matching packet sizes and gaps, and other “tells” to appear similar to known applications.
• Cisco’s solution today depends on data collection using Cisco technology collector points.

Despite these issues, Cisco’s proposal adds value. It would help to know what companies like Palo Alto Networks, Bluecoat, F5, and Checkpoint have to say on this topic.

As a final note, back in 2012 an IETF presentation by Cisco Fellow David McGrew pointed out many of the issues with HTTPS interception discussed here. Since that time, the new trends have only pushed it further off the cliff.

The shape of the future

What do the next few years hold, and what are practical solutions for enterprise-wide visibility and policy enforcement?

• The transition to opaque crypto will start in earnest in 2018, but will take years to complete. Mini-industries will spring up to keep legacy protocols alive for the finance industry for another 10 years. But for most enterprises, simple to deploy central policy visibility and enforcement will go away and companies will adapt.
• There might be a rise of non-transparent proxies for client endpoints in the enterprise. But non-transparent proxies are known to be relatively difficult to maintain, so it is unclear if this will gain widespread adoption. Enterprises may increasingly consider client endpoints to be lost causes, and establish a goal to heavily segment them from business applications. To deal with rogue clients, use of frequent rebuild/replace of client software could be more common.
• For applications running on server endpoints in enterprise data centers, there will be an ongoing trend towards applying policy to the application instead of the network. Policy (for example, what entities this application can talk to or receive communication from), will increasingly be a part of a developer’s definition of an application–similar to software dependencies, data models, and API definitions. ¹

The trend towards opaque encryption will produce a stronger model for secure communication. But while the networking industry resets, vendors and enterprises will struggle to find the new normal.

---

1. Skyport Systems was acquired by Cisco Systems in early 2018. ↩︎