DOT and DOH – Bypassing Enterprise DNS Control Plane

The way enterprises, applications on the network, and endpoints are working is changing significantly. 

Security enhancements now being widely deployed, known as DOT (DNS over TLS) and DOH (DNS over HTTPS), are designed to prevent eavesdropping and improve privacy by encrypting DNS traffic that originates from end users. This is part of a wider initiative to encrypt all end-to-end communications over the Internet.

DOT and DOH, the two technologies that have emerged out of a desire to encrypt DNS traffic, take different approaches. DOT is typically implemented at the device layer so that all DNS communication is encrypted, regardless of the application. A DOT-enabled DNS resolver can be deployed internally within the organization, but more commonly they are hosted externally on the Internet so that they can be used by anyone to encrypt all DNS traffic originating from the client as it navigates the Internet. How these DOT resolvers are configured within the operating system is still the subject of much debate, and efforts are underway to define an automated discovery mechanism so that users can take advantage of DOT (or DOH) without having to perform any manual configuration.

DOH is similar but application specific. Commonly DOH is used by web browsers that nominate a trusted resolver to provide a DOH service. Some browsers (such as Firefox) will have these trusted resolvers hard-coded by default (but can be changed) and can result in DNS queries being sent to an unauthorized DOH resolver operated by an external company with whom you have no commercial or legal relationship. Other browsers (such as Chrome) may perform tests to see if the operating system has been configured with any DOH enabled resolvers, and if so then use those. The important thing to remember is that the mechanism for selecting a DOH resolver is determined by the application, not the operating system. As more DOH-enabled applications are installed, it is conceivable that numerous unauthorized DOH resolvers could be utilized, not all of which are necessarily trustworthy.

In both DOT and DOH cases, all DNS traffic is now encrypted all the way through the enterprise network and out to these external DOT or DOH-enabled resolvers. Devices and browsers that use DOT and DOH completely bypass the internal corporate DNS servers, communicating directly with external DNS resolvers that are not under your organization’s control. IT administrators have no visibility into this DNS traffic and any DNS security controls that have been implemented are completely bypassed, providing no protection whatsoever. Attackers can exploit this by developing malware that utilizes these encrypted security channels to bypass the corporate security infrastructure.  An recent example includes a piece of malware called PsiXbot which uses DOH to communicate with a command and control server, completely bypassing all security controls. This attack method can be used to exfiltrate data or deliver additional exploits into the network.

To prevent DOT and DOH protocols from being abused is not as simple as it might seem. DOT can be blocked relatively easily at the perimeter as it uses a well-known port (853), however DOH uses port 443, which is also used for https traffic – blocking it will also block a very large percentage of web traffic. Because DOH DNS queries are embedded inside encrypted https requests, it can quickly get very expensive and troublesome if the perimeter security firewall has to decrypt these https packets, inspect the embedded DOH queries and then remove or substitute them whilst finally re-encrypting the packet and forwarding it onto its intended destination.

Browser vendors are introducing signaling mechanisms to control whether DOH is enabled or not. Firefox introduced the concept of a “canary domain” to control whether it performs DOH enabled queries – if the domain is resolvable and a specific type of response received, then DOH will be disabled. But each application can provide a different signaling mechanism, or none at all, in the case of malicious applications.

DDI vendors in the security ecosystem like Infoblox and BlueCat Networks are beginning to publish lists of public DOH servers that can be used to block queries to specific IP addresses and URLs so that DOH enabled applications cannot contact unauthorized DOH resolvers. This can be a little like “whack-a-mole” as new servers could spring up undetected or potentially change IP address – all it takes is for a new app to be published that people download onto their phones and when they use the company WiFi they could already be using an external DOH-enabled DNS server. Unfortunately, this is not a complete solution and relies on timely discovery of new DOH resolvers and rapid distribution of block lists. However, until a standardized mechanism is agreed that addresses these problems (which will never happen because there will be application authors that do not wish to comply), there isn’t really any other viable solution available, apart from https inspection, which can be very expensive in terms of performance impact and cost.

To learn more, check out the following resources:

  • This is Sara Dickinson’s post that sparked off a lot of discussion in the community: DoH – It’s DNS Jim, but not as we know it!
  • This recent SANS Institute report confirms the unmitigated usage of encrypted DNS, particularly DNS over HTTPS, that could allow attackers and insiders to bypass organizational controls.
  • This half hour video from Infoblox on the impact of DoT & DoH on the enterprise DNS is well worth a watch.
  • Netscout provides another good overview of the topic.
  • Many experts say DNS-over-HTTPs causes more problems than it solves including PowerDNS’ CTO, Bert Hubert and Hackaday’s Maya Posh.
  • ISC (the developer of the popular BIND DNS server) resources include Alan Clegg’s webinar and Vicky Risk’s blog post.
  • Finally, a fantastic overview article describing what is at stake from Wired magazine.

Wrap Up

“RFC 8484 (which adopted DNS-over-HTTPS as a standard ) is a cluster duck for internet security. Sorry to rain on your parade. The inmates have taken over the asylum.”      Paul Vickie – Internet Pioneer, DNS Authority

DNS, one of the most important and overlooked technologies powering the Internet, is experiencing more change with the introduction of DOT and DOH than it has in the last 30 years.   Established enterprise security controls are being bypassed, a situation that can be exploited by attackers. Elegant scalable solutions beyond lists of DOH servers to block queries are not yet available as industry adapts to the new multi-verse world of DNS. 

– Paul Roberts

Verizon’s 2020 DBIR Report: Take Aways & Reaction

Verizon’s 119 page 2020 Data Breach Investigations Report (DBIR) provides a comprehensive analysis of breaches and incidents.  The report’s underlying data set consists of 157,525 incidents gathered between 11/1/18 and 10/31/19 from 81 contributing partners across 16 verticals, 81 countries, and 4 regions.

Twenty percent – or 32,002 incidents – met Verizon’s stringent data quality standards and twelve percent of those – or 3,950 – were confirmed data breaches.  Changing trends can be seen over time against the backdrop of a total of 755,000 incidents Verizon has collected and painstakingly analyzed for 13 years running.

In addition to sections covering industry segments, regional analysis, and organization size (< 1000 employee SMB versus large organizations review), the report analyses a wide variety of factors.   Factors include: Type of Actors (i.e.,  external, internal, organized crime nation states); Actors’ Motives (i.e., financial, espionage, fun/grudge/other etc.); Threat Actions (i.e., Phishing, Use of stolen Creds, Mis-delivery, Misconfiguration, Password Dumper, Trojan, Ransomware, RAM Scraper); Threat Varieties (i.e., Malware, Hacking, Social, Misuse, Physical, Error and Environmental); Data compromised (i.e., Credentials, Payment, Personal, Medical, Internal, Other); and incident classification Patterns (i.e., Crimeware, Cyber-Espionage, Denial of Service, Privilege Misuse, Miscellaneous Errors, Point of Sale, Lost and Stolen Assets, Payment Card Skimmers, Web Applications and Everything Else).  There is a lot to dig into and there is something in the report for every type of reader.

Breach findings include:

  • 86% were financially motivated
    • 43% involved web applications – double last year
    • 37% stole or used credentials
    • 22% involved phishing
    • 17% due to errors
    • 24% cloud assets vs. 70% on premises assets
    • 96% involved IT (information technology) vs. 4% OT (operational technology)
    • 60% were discovered in 1 day or less vs. 26% discovered in 1 month or more

Additional general insights include:

  • Human errors are on the rise with the increase in technical complexity.
    • Credentials are still the favorite attack surface.
    • Security researcher has become the most likely Discovery method for an Error action breach by a significant amount – over six times more likely than it was last year.
    • While patching practices seem to have strengthened, Asset Management problems become vulnerability management problems on the assets organizations do not realize are there.

Effective data sharing is key to fighting bad actors’ evolving tactics.  Verizon is improving how VERIS (Vocabulary for Event Recording and Incident Sharing) data connects and interacts with other existing standards like the Center for Internet Security (CIS)4 Critical Security Controls and the MITRE ATT&CK®5 framework to improve the types of data that can be collected and reported on.  Mapping industry vulnerabilities to top controls – a new feature in this year’s report – help readers translate attack information into positive, constructive actions vital to organizations’ defenses.

Report Presentation: A Labor of Love

Kudos on Graphics 

The quality of and care given to visualization graphics in Verizon’s 2020 DBIR is inspired. The report author’s took care to make sure the abstract work of art image below would not be mistaken for “Butterfly Vomit” by clarifying that “No, a butterfly did not just vomit on your report.” p 32.   The graphic depicts the number of steps involved by threat action type and the associated attribute that was compromised helping to visualize the range of Attack paths. 

In my perfect world, the actual data points would be or become visible when hovered over on bar charts. 

Engaging Headlines and Tone

I felt like I got to know the 5 person Verizon DIBR team (Gabriel Bassett,  C. David Hylender,  Philippe Langlois , Alexandre Pinto , and  Suzanne Widup)  a bit by reading the report and really appreciated the narrative they provided. Some favorites quotes included:

“If you will allow us a mixed metaphor, there is no outrunning the bear in this case, because the bears are all being 3D-printed in bulk and automated to hunt you.  So, carry on my wayward son and keep doing what you’re doing (you know, patching), and perhaps skip over to the “Assets” section to get an inkling of what you might be missing.” p. 23

“Many of the attacks studied in this report fall somewhere between a stickup and the Great Train Robbery in terms of complexity.” p. 36

“All in all, we do like to think that there has been an improvement in detection and response over the past year and that we are not wasting precious years of our life in a completely pointless battle against the encroaching void of hopelessness.”  p. 37

“For whatever reason, these Error types –  Misdelivery (sending data to the incorrect recipient) and Misconfiguration (i.e, forgetting to secure to a storage bucket) – seem to be the peanut-butter-and jelly sandwich of the breach world this year. Perhaps Internal actors are simply too busy trying to perfect their Renegade dance on TikTok these days; we do not know for sure.  Whatever the reason, these errors are found in every industry and region, and in alarmingly large percentages. As mentioned elsewhere in this report, the vector for these errors is almost entirely carelessness on the part of the employee.”  p. 89

Longed for a Discussion of Financial Implications

While Appendix C of the report did share that  “In 2019, the Secret Service prevented $7.1 billion of cybercrime losses and returned over $31 million in stolen assets to victims of fraud” p115 and the Financially Motivated Social Engineering section stated that “median impact cost for incidents reported to the FBI IC3… for business email compromises (BEC), most companies either lost $1,240 or $44,000 with the latter being slightly more frequent” p25, I found myself longing to be grounded in basic macro statistics like – How much gets stolen each year? What do we spend on defending ourselves?  I also wonder about collateral damage and the follow-on impacts of how stolen funds are used.

As I ran at answering these questions for myself, I found there to be a wide range of answers for key statistics. Here are some starting points:

According to IDC, worldwide spending on security-related hardware, software and services will be $106.6 billion in 2019, an increase of 10.7% over 2018.  This amount will reach $151.2 billion in 2023 with a compound annual growth rate (CAR) of 9.4% over the 2019-2023 forecast period.

This is against a backdrop of global spending on IT overall is expected to be 3.9 trillion dollars in 2020 according to Gartner.

According to Cyber Security Ventures, global spending on cybersecurity products and services are predicted to exceeded $1 trillion cumulatively over 5 years, from 2017 to 2021.

According to Statista, 10.6% of IT budgets is spent on IT security.  According to Boston Consulting Group, average spending % ranges 300% depending on source (from 3.7% – 10%).

According to Cybersecurity Ventures, cybercrime damage costs are predicted to hit $6 trillion annual by 2020, up from $3 trillion in 2015.  They state that “this represents the greatest transfer of economic wealth in history, risks the incentives for innovation and investment and will be more profitable than the global trade of all major illegal drugs combined.”

Companies globally could incur US$5.2 trillion in additional costs and lost revenue over the next five years due to cyberattacks.  — Accenture

60% of small business fail within 6 months of having been hit with a cyber breach

It is sobering to contemplate how the stolen funds are being used to to fund ever bigger and badder activities.

Wrap Up

“Cybersecurity is the central challenge of the digital age.”  – Satja Nadella CEO, Microsoft

Verizon’s DIBR is a great report on a terrible situation. I found it well worth diving into the details and was impressed by the graphics and refreshingly engaging and accessible tone of the analysis which lowers the hurdle to raising broader awareness about cyber threats. I did need to begin researching key macro financial implications of the report findings.  Identifying a handful of key impact statistics and tracking them over time could serve as a companion to or possibly become part of the report.  By providing a data based view of security gaps and mapping to controls, Verizon enables organizations to take important steps to shutting down known security gaps – so key to protecting our economy, democracy, and way of life. 

-Katrin Hillner

World Bank Women, Business and the Law 2020 Report: Gender Barriers Persist Globally

Only 8 countries give women equal legal work rights as men. The U.S. is not one yet.

The World Bank’s recent Women, Business and the Law report measured gender discrimination in 190 countries. It found only 8 countries give women equal legal work right as as men, and the US is not one yet. 

Belgium, Canada, Denmark, France, Iceland Latvia, Luxembourg and Sweden scored full marks on eight indicators – from receiving a pension to freedom of movement – influencing economic decisions women make during their careers. The US ranks #38 with an overall index of 91.3 breaking out by indicator as follows:  Mobility 100, Workplace 100, Pay 75, Marriage 100, Parenthood 80, Entrepreneurship 100, Assets 100, Pension 75. 

A typical economy only gives women three-quarters the rights of men in the measured areas. The WBL index measures only formal laws and the regulations which govern a woman’s ability to work or own businesses– a country’s actual norms and practices are not captured.

Clearly, much more work remains as women in many countries have only a fraction of the legal rights of men, holding them back from opportunities for employment and entrepreneurship.

The report also provides summaries of 62 reforms implemented globally over the 2-year period since it was last published. 

Wrap Up

According to the World Bank, discriminatory laws continue to threaten women’s economic security, career growth, and work–life balance.  While progress clearly is being made, there much work to be done.  As a woman business owner with a passion for gender equity, I appreciate the report’s framework and measurement of laws and regulations that restrict women’s economic inclusion, and the important role it plays contributing to research and policy discussions about the state of women’s economic opportunities. 

-Katrin