Are you going to be in Vegas during BlackHat / DEF CON? We’re hosting a mixer, sponsored by Observa! We have limited capacity, so please only register if you can actually come. Location details are in the confirmation email. Tickets will be released in batches, so if you get waitlisted, there’s a good chance you still get in. Looking forward to seeing you in Vegas!
Tickets are available here
We talk about CrowdStrike in this episode, but we know we made some mistakes:
- The sys files may be code in addition to data.
- The bug might be bigger than “just” a null pointer exception.
Luckily, none of that is actually relevant to the main issues we discuss.
Other Links:
- NIST Definitely-Not-A Competition
- David’s blog on PQC size
- David’s CTO Blog Post
- BlackHat USA 2024 Schedule
- Terrapin Attack
- You of all people should know.
More like ClownStrike, amirite?
This rough transcript has not been edited and may have errors.
Deirdre: Hello. Welcome to security cryptography. Whatever. I’m Deirdre.
David: I’m David.
Thomas: My wife made me sick.
Deirdre: Aw. Thomas isn’t gonna talk very much and it’s just us today. We wanted to visit before we all head off to Vegas in a couple of weeks because we have some very important news. We’re having our first in person in the flesh event in Vegas for Black Hat USA this year, or black hat Defcon or whatever it’s going to be Thursday, August 8, 730 to 930 at a place on the strip. We would love to see your friendly faces in the flesh from across the Internet, but we have a capacity limited spot so there will be an eventbrite link in the show notes and please go to that link and fulfill our call to action and actually sign up so you can actually be part of the capacity limited crowd. And that’s where you’ll find out the exact location on the strip. And we are very excited to have a sponsor for our event. Observa is sponsoring our little event.
Deirdre: Thank you Observa.
Thomas: Observa is pretty cool. Observa is Rob Picard and Luca Carotoni and John Villamil, all Matasano people. So observa does security programs for startups. I don’t think you could find three people that you would rather have as your security team for your startup than those three people. So observa, check them out. They’re very cool.
David: Yes, we are all very excited to have the event in Vegas. Thank you to observer for sponsoring the event and please keep an eye on either the episode notes or our social media to get tickets to the event.
Deirdre: Looking forward to seeing you all there. David did all the legwork to make this happen, so if it goes well, he gets all the credit. And if it doesn’t, we will never speak it of it.
David: Moonlighting as a marketing event coordinator has always been my dream.
Thomas: It cannot possibly go worse than the Microsoft gathering where they bust us out into the middle of the desert to wait in an hour long line for one drink and then another hour to get bused back. It was the most amazing meetup I ever met.
David: We can say this event is on the strip. It is easy to get to, yes.
Deirdre: And it is near some frequent locations of the kind of Defcon affiliated hotel.
David: Universes that’s not far from the in n out burger.
Deirdre: Oh good.
Thomas: We will not kidnap you. At no point in this will you wonder whether this was an elaborate scheme to get all the security researchers out of the desert to kill them.
Deirdre: Hopefully it will go better than the event, I don’t remember who hosted it by, but it was at the Hustlers club, and I got to see John McAfee there before he died, and that was fun. Speaking of McAfee and things that are supposed to make your computer safer or.
David: Never a good transition. Speaking of McAfee.
Deirdre: There was a major international information technology event.
Thomas: I think y’all are taking this a little too seriously. This particular event, I woke up to.
David: A text from my mother about this outage.
Deirdre: Excuse me. This affected me personally. I tried to order a Starbucks, and their entire mobile ordering system was broken across the United States, as far as I could tell because of crowdstrike, aka clown strike.
David: Important question. What’s your Starbucks order now?
Deirdre: It has evolved into a Trenta berry summer refresher, which is. It’s just the blue one. They introduced a new flavor. It’s effectively blue raspberry refresher drink with caffeine in it. No bubbles, light ice. And then I usually get the feta cheese wrap with egg in it, or I get an egg white egg bite. Or I’ve discovered that they have a cheese danish, and I’ll get the cheese danish if I don’t want something protein y.
Thomas: Anyway, is Trenta like the big gulp?
Deirdre: Yes, it is. Like 32oz of drink? Yes. And so I get like a. I get a big gulp worth of blue flavored, like vaguely grape drink with 100 milligrams of caffeine in it or something like that, because it’s large. Anyway, so if you were under the rug of most of yesterday, we’re recording on Saturday the 20th, yesterday, Friday, July 19. Crowdstrike, who operates exploit detection and remediation system, is that what EEor stands for?
David: Endpoint detection and response. So endpoint security, corporate antivirus, and sometimes corporate configuration managers.
Deirdre: Yeah, I got one of the words right. They are extremely popular on a whole bunch of operating systems, but apparently they were particularly hit on their Windows systems for everybody on latest and probably latest minus one or something like that. I saw that somewhere. I don’t know if that’s true, but a lot of their deployed Windows fleet, which is companies, organizations across the world, all of them received an update, not to the software, but to a content piece of content that gets parsed by their software. And their parser is in a kernel module for the Windows kernel. And when your parser panics inside your kernel module loaded into the Windows kernel, guess what? Your fucking computer panics and reboots and reboots and reboots until you can get to it. And say, stop that. Please unload any foreign kernel drivers or kernel plugins or whatever, which is called safe mode for Windows, so that you can try to boot into windows, remove the offending files, cause it’s just a file that a existing parser was trying to parse because it got pushed.
Deirdre: You can remove that and actually try to restart windows again safely. For a lot of people, you had to go manually to these machines.
Thomas: As somebody who will never be able to run a deploy again without having a small panic attack, which is a consequence of spending four years now working on a public cloud, I have nothing but sympathy for the crowdstrike people. And also whoever ran that build is my favorite person in the world for a variety of reasons. People are nuts about what happened here and the consequences of this, and nobody can imagine how a company could have let this happen. But this is the natural state of all software. What’s weird is that it hasn’t happened 100 times before.
Deirdre: Yeah, because it does seem like this is a parser bug that’s been there for a length of time, and it was just finally triggered by this, you know, whatever new rule, new signature that they were trying to parse with their parser and it got triggered. And the all zeros content thing that they were trying to parse is one instance of something that can trigger this parser bug, which is like a null pointer dereference in their parser. Well, number one, why is this parser living in a kernel plugin in the first place, a kernel driver? It feels like we’ve learned not to do that sort of thing, especially for less privileged stuff, but that’s exactly what’s happening here.
David: I think generally you’re right, there’s a question. I mean, I’m no expert on what Crowdstrike’s product is specifically trying to achieve, but the issue here is less on automatic updates are bad, and more on why is there so much privileged code running in the kernel if it could have been in user space? How big is this kernel drive generally, and could it be made smaller in this case? I don’t know what the threat model and the trust boundaries are around the crowdstrikes driver. So you would think, oh, just parse those malware definitions, not in the kernel. Why would you parse it in the kernel? But what happens? Is there some way that you can verify that it was parsed correctly? If part of your threat model, was there something running at user privileges? Do you want that thing at user privileges to be able to somehow then alter the output of that file and replace the malware signatures with zeros or just delete them. There is probably a way to accomplish this where the parsing is not done in the kernel and you can still reach that trust boundary. But I’m expert on Windows.
Thomas: But your Crowdstrike, you’re operating at a scale where malware is built specifically to defeat crowdStrike. Most security products, your adversaries, security products in general, things that you pay money for to make you more secure or whatever. No adversary in the world gives a shit about anything that you do. CrowdStrike is the rare exception to this, where it is worth somebody’s time to write a couple hundred lines of code to defeat CrowdStrike. So if there’s a thing you can do where you can jam up a user land Windows Service and thus keep it from getting updates or keep it from detecting or something like that, this is not me vouching for the model of hoisting all of your security stuff into the Windows kernel. I don’t think that’s a great thing to be doing, but you can see why they would do it that way. They actually do have adversaries that are.
David: Yeah, I mean the Windows model, there’s user to admin, lots of people just run as admin. That’s becoming less common on Windows ten and Windows eleven, but more or less. There aren’t really a lot of per process protections and those that exist are still like very hard to reason about in a way that other platforms kind of offer more. But Windows is in a little bit of a bind here because they’re such an old platform and they’re so big. If they say oh, we don’t actually want things to work like this, we’re going to shrink some of what you can do in a kernel module or change it. Oops. Not letting third parties do something is an antitrust. And continuing to let third parties do things is well, Microsoft gets blamed when a driver crashes, so it’s a bit of a rock in the hard place there.
Deirdre: But also, Linux doesn’t have these anti competitive hawks watching it because they’re Linux and not Microsoft Windows.
David: They also don’t have users.
Deirdre: That’s why they don’t have anti competitive hawks. But Crowdstrike also works on they have a Mac OS version, but it doesn’t seem like it’s as capable. And it seems to be slower on Mac OS because I’m assuming because of the way that the Mac OS kernel is architected. But I guess it’s just not the same competitive base as Microsoft and Windows.
David: And maybe it does the same bug and they just didn’t push out a bad definitions file to Mac.
Deirdre: Or maybe it does and it just doesn’t cause a panic. Or maybe the panic is caught, or whatever it is, it doesn’t cause a full system kernel panic or something.
David: If I were a kernel, I would simply not panic.
Deirdre: If I were a null pointer, I would simply not be dereferencable.
David: What do you think, Deirdre? Does rust fix this?
Deirdre: It actually does not, because if you did everything, if you wrote this parser and you’re still, like, pointing at pieces of memory via kernel boundaries and blah, blah, blah, you’re still dereferencing uncontrollable pieces of memory. I don’t think Russ would fix it.
Thomas: Totally would fix this. It has option types. You would never have a null to begin with.
Deirdre: Oh, so it’s not the memory safety, it’s the null ness. We have lots of languages that don’t have the stupidity of null.
Thomas: So, yeah, you would have, like, a none where you expected a sum, and there’s no way to dereference a none.
David: In rust, aside from the specific keyword used for doing that, like, explanation point. Right. Like, what happens when you panic in rust inside the kernel. Oh, is that a kernel panic?
Thomas: It’s a kernel module written in rust, and it’s all unwraps instead of matches.
Deirdre: Yep, yep.
David: Excellent.
Deirdre: Yep. It’s like, oh, we did it. We did the right thing.
David: It’s like, yeah, no, I mean, presumably you would, like, not do that a ton, but, like, I think the point is that a crash in the kernel module is still a crash in the kernel module, regardless if it’s actually a null pointer. Deref.
Deirdre: Yes.
David: Or if it’s just like, oh, the language. The language has saved me by asserting on this bounds check or this null check for me. But, oh, well, once it asserted, like, you still crashed.
Thomas: Yeah, this has been our two minutes of rust humor. You read a lot of stuff about, like, what crowdstrike is and, like, what EVR things are, and it’s like, it’s all very simple, right? This is just what people run in 2024 instead of antivirus. So compliance regimes and IT security practices. If you look at a Fortune 500 company, they’ve had a large scale IT security practice for the last 30 years. Some of this is just logic that is handed down from generation to generation that something has to be running on every endpoint. It used to be mcAfee AV or semantic AV, and now it’s crowdstrike. Honestly, you’re better off now by a lot. You would much, much rather be running crowdstrike, where they every once in a while push out a build that crashes every windows machine on the planet than running av engines that were all written in the late 1980s.
Thomas: There’s so much chatter going on. This is the biggest thing that’s ever happened for it nerd in the last several years. So people can’t shut up about it.
Deirdre: Everyone keeps saying that this is what y two k was gonna be like if y two k actually happened. And I don’t know if that is.
Thomas: Actually so extremely mild.
Deirdre: I don’t know if this is.
David: Quickly, everyone say your age at y two k. Oh.
Deirdre: Unfortunately, maybe the impact, like one time impact is at the large scale that we were afraid of for y two k. But the remediation is quite easy. Like quite easy compared to y two k. Like y two k basically required the most critical pieces getting rewritten, the code getting rewritten versus. This is just like, there is a step by step thing you can do. It’s not fun. You have to go find your host and go into safe mode, remove a file and then reboot it. Ten times, perhaps.
Deirdre: But we know what to do. You have to rewrite all your software. So I don’t know.
Thomas: I have a bone to pick here, and I think David will appreciate this particular bone I have to pick here. I’m again dragging the orange sight into this conversation. Although this particular orange site drama links back to Twitter. So it’s also Twitter drama. But some enterprising souls have uncovered the interesting detail that George Kurtz, the CEO of Crowdstrike, was at one point the CTO of McAfee, the quote, unquote CTO of McAfee during an era where McAfee also pushed out a bum updater that broke everybody that was running McAfee. And my problem here is that no one knows what the hell a CTO does. So there’s like a. I can’t get the picture out of my head that these people are imagining George Kurt sitting there on his computer writing software updaters.
Thomas: So, David, what does a CTO do?
David: I mean, it depends, but I would say CTO is mostly, like, not a real title, and it is actually just like a title that you have when you’re in a different role, but you need to sound cool. Those roles being technical co founder, which is a perfectly valid form of CTO and arguably the most valid form of CTO. It’s like you have someone at the company who has a lot of equity, a lot of institutional knowledge and is kind of driving the technical product direction. That’s your technical co founder. There is. CTO is actually just a VP of engineering in that they’re in charge of Eng execution. I would argue you should get your title to be VP engineering in that case. Again, if you are a VP of engineer, you’re not writing a software updater.
Deirdre: No.
David: Then there’s sufficiently large parts or whatever. Yeah, absolutely. The CTO that is actually a VPN is not writing parsers. Right. They’re looking at Jira tickets and they’re writing okrs. And then the other case is like, you’re a big ish public company, and you kind of need someone at the roughly the C suite level to go around and talk about technology to other people at the C suite level. And that person is usually someone who used to be a VP edge at somewhere big. So, for example, like the CTO of Microsoft, I believe, used to be the VP edge at Salesforce.
Thomas: I have a simpler breakdown of ctos than you do. They’re equally valid breakdowns. They’re both worth hearing. Right. You have, like, the OSi model of CTOs, and I have the IETF model. But, like, and I want to caveat this by saying that none of this applies to the CTO of fly IO. Jerome, who I adore and is amazing, and also spends all day and all night rewriting every line of code that I’ve ever written. So, this is definitely not him.
Thomas: But my general breakdown of ctos is that there’s three types. Um, there is the type where it is the consolation prize that the founder gets when they take away his commit privileges. Finally, when there’s a real engineering team. That’s the first kind, right? And then there’s the kind of CTo you are when you are the CEO of a large company that gets bought by another company where they’re not making you the CEO, they have to give you a see something title. So it’s CTO, which just basically means the CEO of the product line that is now part of this bigger company. And the third kind of CTO is the sucker who took the CTO title instead of pushing for more compensation at a small company that shouldn’t have a CTO at all. Those are the three kinds of ctos there are, right? And, like, if you ask me what the most important job skill of a CTO is, it is identifying themselves as the CTO on a call with a customer prospect. That is it.
Thomas: That is the CTO skill. That is what they do. Very important the idea that a CTO is doing any kind of technical work whatsoever, but people think they’re like. It’s like they’re the lead programmer in the company. They’re like. It’s like chief scientists. Like they’re doing the most science. The same thing, right? It’s one.
Thomas: It’s a title that has absolutely nothing to do with what that, in my experience, has absolutely nothing to do with what the title sounds like. So, no, people, I don’t think that there is much of a George Kurtz connection with the broken updaters.
Deirdre: Now we have to do Sisos and CSo’s.
David: Our advice on that one is never accept that title, even if that’s your role. Just get any other title.
Deirdre: Yeah, it’s the connection between these two and any incident with McAfee and any incident with crowd Sark is coincidental at best.
Thomas: I mean, if you do get the CSO title, just remember the mesoamerican cultures where before they sacrifice you, you get a couple of days of everyone just treating you really awesome, and you get to eat whatever you want. Just make sure you’re getting all the stuff that you can possibly get before the day of sacrifice.
Deirdre: Absolutely. You are there to be fired when there is a security incident that has traditionally been the role. So we did kind of do our little hug ups. Bit about this is the fear of anyone that pushes out updates, whether they are actual code patch updates to your deployments or configuration updates that can have almost as much or equally as much impact because they are executable instructions by the executing software or whatever, which is basically what happened in this case. But this thing, as far as we can tell, the configuration update or signature or whatever it is, got pushed out between 04:00 a.m. and about 05:30 a.m. uTC yesterday. And it was all zeros as far as we can see.
Deirdre: And people have claimed to replicate the panic with content that’s being parsed. It’s not just all zeros. How do we prevent problems like this? Well, the problem is that we don’t have the procedures in place and people in organizations that allowed this problem to occur, not now.
David: There’s a meme of it takes a long drag.
Deirdre: Yeah, it takes a long drag on a cigarette. It’s like, yes, the, the root cause analysis. Your five, why’s, whatever you want to call it, the, you know, blameless post mortem will eventually, like zoom in on there was a pipeline. Procedures, pressures, checks that either were in place and not followed or they were all in place and followed and were insufficient, or there was a bug in the pipeline, in either the CI testing pipeline or the cd deployment and packaging pipeline, and or someone claimed on the Internet, like, oh, this was supposed to only go to the client staging environments, and it overrode that. We don’t know enough. Now there’s a lot of speculation, but having a thing that’s basically all zeros get pushed out feels like something that we should be able to flag. And it’s unclear whether it was built and tested and passed all the tests with all zeros, or if it passed all the tests and then it got built, and after that it turned into all zeros and nothing would catch that or whatever. So I’m very curious.
Deirdre: I want to learn more, and, like, we’ll see if there’s anything that we’ll learn publicly about this. But I’m very interested in sort of like, the pipeline, the procedures of how this gets out into the wild in general, because that’s where I was zero in on and trying to figure out how to prevent stuff like this from happening.
David: The pipeline procedures question is somewhat interesting, I think, and they’ll definitely have to do it for regulatory or legal or even just probably sock price reasons. But I actually don’t think it’s the thing that matters most in the sense that you have this question with any system that has various checks in it of just, oh, if this one thing isn’t working, what is the expected behavior? What a failure rate of some given part of the system. And just because it was caught, he had a failure over in step one, but it gets caught in step three. How bad is that failure in step one? And so, like, there’s shortcomings there and that this failure wasn’t caught. But there’s another question. I’m just like, parsers are the easiest thing to fuzz and test, right? Like, so there’s one side of, like, don’t parse in the kernel if you can avoid it, but there’s another side of, like, if you’re writing a parser, like, we know how to do that well, like, the cryptographers are pretty good at this, like, parsing, like, major browsers still parse x 509 certificates in the browser process because they have to. Cryptographers have got, because, one, they have to, there’s nowhere else to do it. And two, like, the cryptographers have realized, like, how to spend the right amount of effort to make a parser that is, like, up to a standard that is kind of higher than the parser standard might be in other situations, but the reality is, like, of the code that you could write at a, like, very high level of reliability.
David: Parsers are like the most straightforward thing to test and to fuzz and, like, sure, you can go into all of these policies and procedures about the deployment process, but, like, I would almost rather see them skip that and just be like, we have fixed our parser.
Deirdre: Which is why I want to give them the benefit of the doubt that they do. They did take this change of this config, tested it against their parser, and then it passed, and then they built it for deployment. And then something went wrong in the build, the cd process, the packaging process, the release pipeline, not the testing pipeline. But that also means that you have to test twice or you’re testing in the wrong place. Because if you’re testing before build and it passes and you test after build and it doesn’t pass, then you’ve got a problem. But I don’t know that much. But I generally agree with, like, we know how to make. We know how to test and fuzz and write parsers better now.
Deirdre: And it feels like this one is like an extremely non risk tolerant parser. So, yeah, I don’t know. I’m very curious. There’s been debate about, like, why. Why this is written this way anymore. Because in later versions of Windows, they have made safer hooks via the kernel or via that surface available to software like crowdsark. I think this was their Falcon sensor to use in a safer way. And why aren’t they doing that? And it’s like, well, I have a feeling that this is deployed.
Deirdre: This is deployed to hospitals. This is deployed to airline systems. This is deployed to fucking Starbucks. This is deployed a lot across a lot. A lot. A lot of diverse windows instances. I think that there’s a lot of them that aren’t up to speed and making those hooks available. And so they’re doing it the tried and true and available way.
David: Yeah. In terms of diverse deployments, though, I moved recently and I’m trying to get my car title because I need to, like, register at a new place. And so I went to log into my car loan portal, and it was down. I was like, man, all the things to go down because of crab strike. I did not expect Mazda financial to do that.
Deirdre: It wasn’t even like the local municipality, like, you know, secretary of, you know, department of transportation.
David: No, it was not the DMV. It was like, like the car loan portal where I wanted to go to pay off the loan so that they would send me my title because that just felt easier than requesting they send a title to the DMV.
Deirdre: So one last thing before we move on from this is why I thought windows has their own built in av security thing, like windows defender or equivalent for maybe something. I know there is one on Mac OS, and I’m forgetting their name right now. So why don’t we just all use Windows Defender? And why is that? Nothing fine and sufficient. And I don’t. I don’t have a good answer, except that Crowdstrike does a lot more and is a lot nicer to use on the backend as, like, an operator of an EDR solution. And, like, Windows Defender is just not enough. And so there’s a reason that Crowdstrike.
David: Has a market, because this podcast does not endorse any individual antivirus product.
Deirdre: No, I don’t like. No, but, like, there’s a reason that markets have, like, found, like, a solution, and it’s that Windows Defender did not just completely be, like, solved it, just use this for free. It comes with your windows. And, like, what Thomas says, like, you run it. You run Crowdstrike in 2024 instead of, like, back in the day, you would use McAfee antivirus, because that’s what you do. And when the built in stuff into your os is not sufficient as a.
David: Real operator, instead of talking about compliance, let’s talk about literally anything else.
Deirdre: Do you want to talk about post quantum standards compliance?
David: I’d like to talk about how there still aren’t any post quantum standards. It is difficult to comply with compliance guidelines that provide deadlines in terms of when communication needs to be post quantum secured if the algorithms that are post quantum have not even been standardized yet, let alone whether or not they’re suitable for use in the real world.
Deirdre: Yeah, we have a draft. We’ve had a draft for a year. We’re talking about the NIST, the United States National Institute of Standards and Technology, the post quantum cryptography competition that they’ve been, or whatever, they don’t call it a competition, whatever. The thing, the thing that they’ve been running for eight years now, at least the standards, the official FIPS standards for MLChem, which is a new kind of key agreement based on liases. ML DSA, which is a new signature based on lattices, and SLH DSA, which is stateless, hash based signatures, which is not based on lattices. It’s based on, signatures are supposed to drop any day now. And it’s all just like rumors and whispers and just sort of like people sending emails to anyone that might have an in at NIST. I’ve been refreshing their website every fucking day and just like watching and preparing and just, just waiting because they signaled publicly that they might be ready by end of summer slash September.
Deirdre: And then I think someone from Nist was on stage somewhere and they were like, no, ML chem document is done. We finished it and we’ve sent it up the chain and we didn’t hear anything about the other two, but we think they’re all going together in one fell sweep. Falcon is not done yet. They’ve told us that Falcon is another lattice related signature scheme that’s not done yet because it’s harder. I think that’s coming later. But these three, it’s very possible that they will drop any day now. And as soon as they drop, it’s like a starting gun for like lots of people in industry, especially anyone that has to be compliant with like these memos from the United States executive branch about being compliant with the systems that you sell and deploy to the federal government. But also there’s all these other systems of like, if you want to be FIPS compliant, if you want to deploy the software in a national security system, they’ve already outlined, like, you must have these XYZ things and they have to be done by like 2030.
Deirdre: Some of these deadlines are like real soon for the big infrastructure and organizational deployment lift.
David: Yes and no. So the part of the problem here.
Deirdre: Also on the page.
David: Yeah, there’s a number on the page, but the page isn’t, isn’t actually a compliance document. So like the one that people cite a lot is the dodgest that has this document that says this is what the new CNSA guidelines are, the algorithms for computer network security or something like.
Deirdre: That, which is basically national security systems. Yes.
David: Yeah. Which is basically what are you allowed to use to secure top secret data. And they’re like, you know, we are going to switch this to require post quantum, which if you buy the, you know, the post quantum threats, then like, that’s a very reasonable thing to issue a document saying the next version of this 2.0 will look like this and a little bit of guidance on like prioritize, you know, store now to crypt later over signatures. And like, here’s how, like this interacts with like web browsers. And here’s how this interacts with like long term storage and software and firmware signing. And then it provides some, some like suggested dates.
Deirdre: Yep.
David: But like one, this document is non normative. It literally says like, this document is describing what, how we will decide the process. And they’re like, it will be based on standards from NIST, it will be based on these threats and needs, their different priorities and so on. But it has some years in it, including things like 2025 for browsing traffic. And it’s like, well, if NIST hasn’t released a single final version of the standard in mid 2024, you’re not going to, by default, have post quantum signatures anywhere in 2025. Yeah, but notably, the 2025 year is just a recommendation in this document. And then it says 2030 is the deadline. But this document isn’t even the deadline because it’s a non normative document.
David: So all this to say, the only thing that matters for actual migration to post quantum is that NIST standardizes algorithms, and then from there, people can work on solutions, but as long as these algorithms are not standardized, there’s, like, very limited cases where they can be used in anything outside of, like, an experimental context.
Deirdre: Yep. And I will point to yet another document that’s the OMB memo, which does say, goal of mitigating as much of the quantum risk as is feasible by 2035. So that’s, like, different, because that’s for different systems in the us government than these national security systems that the thingy from CNSA two odd is talking about, but that gives it a little more oomph. But it is still, like, trying five years.
David: It’s like there’s a big difference between 2030 and 2035 when it comes to, like, trying to migrate the webpage.
Deirdre: Yeah, yeah, but what you ended on is absolutely true, is like, we can’t. We literally can’t, like, make progress until we have a final standard of, like, the thing that everyone is pointing to, including all these things in the us government, but also countries around the world are watching this process, this NIST PQ process proceed, and they are keeping an eye on it as sort of like, yeah, we’re going to take suggestions from that and maybe we’ll add a little bit more to our stuff. A lot of the governments in other parts of the world are much more comfortable with hybrid solutions for their systems than this NSA CNSA document for national security system, which is basically like, no hybrid, if you can help it. We want it to be pure PQ, which is a big vote of confidence for things like MLChem because they’re saying, like, MLChem 1024 will be, like, by itself sufficient and required to protect key exchange for, you know, top secret, you know, communications or storage or whatever other governments are like, sure, ML, chem and Frodo or MLchem, and, I don’t know, entrue or something like that. I don’t want to spend the whole summer just sort of waiting, waiting, waiting. Especially because I thought we would get it at the end of the summer. And like, no, it’s going to be like, any day now. It’s like, fuck, just do it already.
David: There’s another memo from OMB M 2302, I believe, that says to like. It requests that agencies basically inventory their post quantum risk in terms of, like, what systems would be affected by a quantum computer. And if this memo percolates down to you, if you’re, like, at an agency, deputy CIO or whatever, my advice to you is to ignore the condition and just send back a list of all of your computer systems. I think the practical matter is that it will affect all of it, and nothing is standardized yet. So instead of trying to ask people whose job is not to think about quantum computers in cryptography all day, if you get this memo, just ask for a list of all of your computers.
Deirdre: And then it’s like, cool, we have an inventory, and now what? Now what do we do? And, like, people are not. Are asking that question. They’re sort of like, what? What’s the next step? How do we remediate this thing? That’s a good question that not a lot of people have a straightforward answer to.
David: I think we have talked, I don’t want to say extensively, because I don’t think we’ve done a full, like, episode on this, but I feel like we talk about this every other episode because it’s, like, related to the day jobs of several s here in some form where we’re just like, oh, all of these transitions are hard, and there are no good solutions, and these algorithms are very big.
Deirdre: Yep.
David: But, like, a bunch of that discussion is even just predicated on, like, can we get a final version of the standard?
Deirdre: Yes. Especially because, like, there. There’s the ongoing signature on ramp. So, like, they’re gonna drop MLDSA, which is like a digital signature that one could theoretically replace a bunch of stuff in the webpage with. But it’s so big. The keys are big and the signatures are big. SLH GSA is probably good for signing firmware or things like that. It’s not good for online signatures and putting them on, on the wire, for example, or in your.
Deirdre: Or your, like, end to end encrypted communications or anything like that. ML DSA is only one of these three drafts Falcon might land later, but it’s got some sharp edges that it’s unclear if people will be happy with. Even that is still kind of big. The onramp is supposed to find more signature algorithms that will give different security guarantees. So maybe not based on lattices. It’ll be based on something else. But all of them, except for ski sun in the on ramp, are still too big. They’re still too big.
Deirdre: And, like, that’s gonna take more time. So, like, one, we want these things to land, period. But two, when they do land, we still have a lot of hard work to do to figure out how to make these things work in things like the web PKI, because none of our options are very good. And it’s gonna be some hard architectural design, organizational migration decisions, designs, and planning and coordination to figure out how to make any of this work that doesn’t significantly degrade the experience that users of the secure Internet have come to expect for the past decade.
David: I think. Let’s go back to Vegas for a second and talk about the schedule. Black hat, I think, I don’t know. Maybe you will stick around through Defcon. I don’t believe in Defcon, so.
Deirdre: Okay. Why don’t you believe in Defcons?
David: I like conferences where I can get coffee and water in the hallway, and that’s not really a thing at Defcon. And also, no, I’m on the review board for Black Hat, as is everyone else on this podcast. And so we kind of have a little bit of a connection there. So I tend to go for black hat. If I go at all. I think we’ll all be there for black hat. So let’s. I don’t know, maybe highlight a few things that we saw on the schedule for 2024.
Deirdre: We wanted to talk about this more depth earlier in the year on the podcast, but this terrapin attack on SSH, breaking SSH channel integrity by sequence number manipulation. This is pretty cool. This is very cool.
Thomas: This is the best. This attack is. This is. It’s not drown level. For me. Drown is still my favorite applied crypto attack, but this is still. It’s one of. So, like, when I started, like, when I first started engaging with cryptography stuff, maybe, like, the late nineties or whatever, you run into, like.
Thomas: Like, SSL three and, like, the SSL transcript hash. And I was in my, like, my twenties or whatever, or my early twenties when this was. God, I’m old. But, like, so, like, if you’re, like, a normal programmer and you’re, like, your first experience of dealing with a serious cryptographic protocol. And Paul Kocher designed SSL three. Like, SSL three is terrible, but that’s mostly because we didn’t know as much as we know now. But a serious cryptographer designed most of SSL three, right? Your first experience of it is like, okay, so you’re doing a handshake, but at the same time you have to do a running hash over all of the components of the handshake, which is a giant pain in the ass. That’s like, at the time, it was hard for me to even get my head around the level of complexity of running a hash over multiple.
Thomas: This.
Deirdre: It wasn’t chained. It wasn’t what we have in TL’s 13 now, which is sort of like updated chained hashes.
Thomas: I’m saying these things out loud and like, so, like, I, you know, I can, I can throw together like, a noise implementation in, like, you know, in a couple of hours now, right? But back then, it seemed like just moon science to me. Right? So, terrapin, right? So there’s, you can think of like, two lines of major transport, major encrypted transport protocols, right? Like, there’s the TL’s line and there’s the SSH line of protocols, and sometimes they have the same vulnerabilities, and sometimes they don’t, right? So in the original SSh 2.0 design, the first, seriously, SSL three is the first serious TL’s or the first serious SSL protocol. Nsh two is the first serious SSH protocol. Right? They do not do a transcript hash, like a full transcript hash over all the messages that happen, right? So instead what they do is it’s kind of the same deal where you’re just building up to doing a DH, right? You’re just going to do a diffie Hellman and do a key exchange. So they look at this, I’m putting myself in their heads, but I wasn’t in the room, but I’m pretty sure I’m right about this. They’re running a handshake to do a DH. So they look at this whole thing and they say, well, here are the things that contribute to the DH that we’re going to do. Here’s all the things that matter.
Thomas: So what we’re to do is, instead of hashing every message that happens in the transcript, we are going to just hash the things that could ever hit the key exchange, right, which is like, what could go wrong, right? And it turns out, like, so you can’t influence the key exchange, but what you can do is throw off the sequence numbers now, right? Because they don’t go into the key.
Deirdre: Exchange anymore effectively, which means you can.
Thomas: Like, pad out, like, you can, you can eat messages, basically. You can, like, you can snip out a message from. And like, the only problem with this is that, like, and I haven’t looked at this in a little while, so I could be wrong about this. And maybe they’re going to get up on stage and show like a really, really awesome outcome for this. The only, like, this is like, as cryptographic protocol things go, this is a. I think this is a pretty devastating attack. Like, this is not a great property for a cryptographic protocol to have, but I think in practice and real sh deployments doesn’t matter a whole lot. So sh has after the handshake, like, option messages that you can send to, like, ratchet up security or add restrictions or whatever, I forget what the options are.
Thomas: Right. So you can snip those out, as I remember, but, like, nobody uses them, so it doesn’t matter, I think, is the outcome. But still, that is the thing you could do because they’re only, they’re only hashings. They’re only doing authenticity for the things that matter, for the key exchange, but not for, like, the meta stuff, for that runs the protocol. You can fuck with that stuff. It’s a really neat attack.
Deirdre: And this smells like things that were optimizations in protocol design back in the day, in the nineties, early two thousands, where they’re like, we only try to hash or authenticate or do extra work over the things that absolutely need, quote, need to be done. Such as, like, just the, the diffie Hellman. The things that will influence the value of the diffie Hellman handshake. But the teal, like, especially in TL’s 13, like, they’ve basically been like, fuck it. Like, we. It’s really hard to let some things be manipulable by an attacker and other things not be like, the whole transcript needs to be, like, chained together in a, like, you know, a hash tree, basically. Like, you do, every, every handshake, every piece of the agreement is getting hashed, and then every new thing is also getting hashed and also getting hashed. And, you know, they’ve even gone farther just recently with encrypted client hello.
Deirdre: So that there’s even more information that is not even visible, let alone manipulatable by an adversary when you’re trying to, you know, set up a session in TL’s 13. And this attack smells of, like, we learn these lessons the hard way and fix them in TL’s 13 and other places in TL’s, and they haven’t quite percolated down into SSH.
Thomas: What gets me here is that this is like, just such an obvious difference between the TL’s protocol and ssH. So it’s like this was discovered relatively recently, right? At the end of 20, I think at the end of 2023, is when that pre print came out or whatever, right? But that huge distinction between the two protocols, between SSH and TL’s, has been there for decades, right? And it took until now for somebody to look at this and say, wait a minute, these are not the same protocol. What is the consequence of these not being the same protocol? And the consequence was that the security of the other protocol fell apart. Right. So I just. It makes me happy to be in a world where any moment, something like this might come out of somewhere else.
David: Can I say something slightly embarrassing? Which is that when we were working on the weak diffie Hellman stuff in grad school, like the logjam attack, so we were looking at TL’s. We were looking at, like, diffie Hellman group parameters of other protocols. And I was tasked with figure out what group parameters SSH uses and also go look at the SSH protocol and figure out if it has the same protocol vulnerability that logjam represented in TL’s. And I looked at that handshake for a while, I was like, no, this is a good handshake because unlike in TL’s, it’s got a signature over the full set of key agreement parameters. And I missed, like, what turned into terrapin. I was like, oh, well, it’s got the signature, so, like, we can’t strip the thing that didn’t have the signature in TL’s and then moved on. But I just. I missed it.
David: But I don’t know. I also wasn’t, you know, not to like, come up with excuses, but I was primarily interested in implementing the handshake to get a scanner out of it. Because at the end of the day, as mentioned repeatedly, Thomas is an attacks person, Deirdre is a real cryptographer. And I am just a network engineer who happens to like reliable systems.
Thomas: They should credit you on stage for leaving that bug there for them.
Deirdre: I’m the cryptographer that likes to build the. Build the good thing. I’m still not very good at writing proofs yet, but I’m okay with that for now. Yeah, I mean, like, you were also looking. You were looking at weak DH. And you were looking at these things that influence the actual diffie Hellman value, which is protected. It was the part of this that is protected and the sequence values and these other optional messages were not protected. So like, I don’t blame you at all.
Deirdre: You were looking for something quite specific and it was protected against that. But if you were just sort of looking around at what else you could notice in it, maybe you would have saw something like that. But it’s okay.
David: Our approach, we were, sometimes we collaborated with people who did like actual formal modeling and proofs. But the way that we got into drown, for example, was just one day someone mentioned that SSLV two was still enabled by default on a bunch of stuff. And we were like, that’s probably buggy. And then we just applied the Feynman method to open SSL’s code. We were like, there’s probably got to be something here. We just stared at it. Then we were like, ah, we have found like logic bugs, not cryptography bugs.
Deirdre: Oh, yeah.
David: And then we were like, so then we like contacted OpenSSL and they were like, have you met these other people that found cryptography bugs? I think you might. Your stuff might help them. And we took a quadratic attack down to a linear attack because we just stared at the code and found bugs pointers.
Deirdre: Okay, so you found, did you find a buggy implementation of a good protocol? Of a good.
David: We found a buggy implementation of a bad protocol, SSlV two, with drown.
Deirdre: But.
David: Okay, but we found the bug. We found the bug in the code in the implementation. And then like, Nimrod Avaram found the bug in and his collaborators found the bug in the cryptography. And then when you put them together, it made the attack on cryptography much more efficient.
Deirdre: Oh God, this is the shit that, this is the shit that keeps me up at night because all of this shit is so hard. It’s literally like writing down something that’s non buggy and then translating it into code in a non buggy way. And then you might have a different bug over here and another different bug over here, and you combine them and you get something that is completely just. And I hate it. This is such hard shit and I hate it. It’s all hard. And I don’t.
David: Yeah, I don’t know if you’re familiar with the matteblaze attack on like physical keys, where you target one cut at a time. That is basically the bug that we found in the OpenSSL implementation, which let you target the key one byte at a time. As opposed to all of it at once.
Thomas: Remind me of the mapblaze attack.
David: So, in a master keyed system, if you have access to just an individual key for a single door, this isn’t quite accurate, but more or less, the master key will have one cut, and your key will have another cut, meaning the various points would be at different heights and you know the height of your key. So that means that for every individual cut, you can. You just need to find the cut that’s at the other position. So if you can duplicate. If you have, there’s ten cuts in a key, and you duplicate your key, and you make nine of them the same, and then you try all ten values for the one cut that you’ve changed, you’ll learn, oh, the master. My key is at height four. The other valid cut is at height eight. Therefore, the master key must have height eight.
David: And then you do that for each cut in the key. And so it takes what was like an exponential attack of, like, let’s just guess all the cut heights to becoming linear in the number of cuts.
Thomas: So Tai Duong called that the Hollywood attack. Right. That’s my mental model of most, like, actual cipher attacks, is Tai Duong’s Hollywood thing, which is. It’s Hollywood because if you do printfs while you’re debugging the attack or whatever, you’ll see, like, the string of gibberish, like, the actual decipher text. And at the end, you’ll just see kind of like the rotating letter as you figure out letter by letter with the last. So that’s like. That’s how beast works, for instance, is how the CBC padding oracle works. If you do printf there, it’ll look like it’s working it out one byte at a time, like in the movies, because that’s what you’re doing, right, is bringing the attack down to guessing one out of 256 values instead of one out of two to the 128th, right?
David: Okay, exactly.
Deirdre: That’s very cool. Anything else on black hat schedule we want to highlight?
David: Well, just note, I think the network security track is quite good this year. There’s several interesting things in there. There’s attacks on DNS servers. I don’t know if that involves DNSSEC or not, Thomas, but if it does, I’m sure you’ll be there to laugh. There is exploiting VPN’s, which is always a fun reminder that your security appliance might make you less secure. And there’s stuff about vulnerabilities in RPKI validation, which might actually start to become relevant as RPKI deployment becomes more of a thing. So I think it’s interesting to look at it from that standpoint.
Deirdre: What is RPKI?
David: RPKI is the PKI used to secure BGP router communication so that you can’t spoof routes.
Deirdre: Yeah, that’s. That’s important. And. Yeah. Why. But why is it different? Why is it. Never mind.
David: Well, it’s. I mean, you’re not. You’re not authenticating domain names, so, like, something the authorities and things have to be different, but it’s still just, you know, x 509 certificates, I think, in.
Deirdre: Signatures, it’s different, but not significantly different.
David: It’s another PKI because it’s another use case.
Deirdre: Yeah. Okay. All right. There’s some keynotes. Jen Westerly, who’s running Cisa, is the primary keynote.
David: Jen Easterly, not Jen Westerly.
Deirdre: Perfect. Perfect.
David: Yeah, we’re keeping that one in. That’s an incredible.
Deirdre: That’s perfect. Yeah.
David: The Waluigi to her Luigi.
Deirdre: They’re the primary keynote. They’re like the. Whatever. Main. Whatever. Main stage, whatever you call it.
David: There’s a Michelob ultra arena.
Deirdre: Is it really?
David: Is this at least that’s one of the keynote arenas.
Deirdre: It’s still. Yeah, it’s still Mandalay Bay. It’s all Mandalay Bay. Yeah, the Michelo also.
David: That’s the one that’s like the fight stadium, right? Like, if you think about, like, oceans eleven and the fight night, right? It’s like that.
Deirdre: Something’s there is, um. What’s his fucking face? Is Mike Tyson gonna kill one of the. Those youtubers in a fight there in the near future? They’ll draw.
David: No, but if you go one door over to the Luxor, you can go to the esports arena, and you can also see America’s Got Talent live.
Deirdre: I do have a soft spot for the Luxor just because I like pyramids. Anyway. Not pyramid schemes, just pyramids. There’s another keynote, a fireside chat with Moxie Marlinspike, with Jeff Moss, who’s the guy that runs Defcon and all that stuff. What’s his handle? I forget his handle.
David: Dark tangent.
Deirdre: Yes, dark tangent.
David: I don’t know if he still runs it, but he definitely started both Defcon and then black hat. As, like, the corporate version of Defcon.
Deirdre: Yes. DefCon was absolutely first because they needed a place to send their people with budgets to learn about all the tips and tricks from the hackers at Defcon, and they couldn’t do it via the Defcon name, created Black Hat. And that’s why it’s very different in vibes, even though Def Con is massive now. And, yeah, I guess they’re going to talk about privacy and security and other things like that. And Moxie’s moved on from stuff with signal, and he’s been doing some other cool adventures. So I’m going to, I’m interested to see if he’s going to talk about that sort of stuff. But let’s see.
David: My other advice for black hat is, well, so, like, the first time I went was when I talked back in, like, 2016 about drown, and I was like, oh, I don’t want to go to this vendor floor. That’s for, you know, these. Like, I, like, I do real work. I’m not going to the vendor floor nowadays. I strongly recommend that if you’re there, you do go to the vendor floor. If not, not to buy anything. That’s like a terrible idea to buy something, like, at one of these. But it’s just interesting to see, like, what is going on.
David: Like, how are people marketing things? Like, what are people? What are the things that people are trying to sell? Because, like, we’re a very technical, focused podcast, and we have perhaps a different perspective than, like, you know, if you’re actually responsible for securing an organization. Like, what are, like, the people that are being sold to there. Like, why? How are these being marketed? Like, what are they going after? And, you know, a good chunk of it doesn’t make sense. A chunk of the people in the booth won’t actually know anything about what it is that’s going on, but it’s still very interesting. And also, you can get pictures with f one cars because it’s ridiculous. And so I do strongly recommend that you actually go and walk around the vendor floor at least a little bit. Let’s remember, all those people are, in theory, making money.
Deirdre: That’s true. Oh, God. Yeah. I’ve avoided it like the plague. But that is a very good idea. I should definitely canvas for, like, if people are shilling for PQ migration or just quantum or quantum key distribution. And, like, those things are very, very, very different.
David: So I also recommend that you be careful about if you do go to the vendor floor. Like, consider what you put on your badge, because if you’re not representing your employer on the vendor floor, like in a booth or whatever, but you do want to go talk to other things. You don’t necessarily want to reveal who you work for, especially if you could be considered a competitor, someone that you go talk to, or just because if you have a sufficiently large company on your badge, everyone will try to sell you something and you probably don’t want to buy it or don’t have buying permission for it.
Deirdre: So street smarts. Those are some black hat street smarts. I don’t know. I’m not anything else. I’m going to Defcon. I’m going to Defcon for at least a day and a half. So I’ll see you around Defcon. Probably.
Deirdre: I like to go check in on the crypto and privacy village. And maybe there’s also a new, a newish quantum village that’s been around for one or two years, and they do a lot of post quantum stuff sometimes. So maybe I’ll see you around Defcon. Awesome.
David: All right, well, once again, keep an eye out on our social media or the episode notes for details on the get together in Vegas. Thank you again to observe it for sponsoring it. We’re excited to see everybody first or second week of August, depending on how you count, and have a good, great summer.
Deirdre: Yeah, security cryptography, whatever, is a side project from Deirdre Connolly, Toms Tachek, and David Adrian. Our editor is Nettie Smith. You can find the podcast online at SCW pod and the hosts online at Durhamcrustallum at TQBF and Seeadrian. You can buy merch at Merch Dot security cryptography or whatever.com if you like the pod. Give us a five star review on Apple Podcasts or wherever you rate your favorite podcast. Thank you for listening. See you in Vegas. Bye.
David: Also, we’re on YouTube now.
Deirdre: Yeah, we are on YouTube, but not.
David: With video of us, just audio. We’re on YouTube.
