Not the hero the internet deserves, but the one we need: it’s Ryan Sleevi!
We get into the weeds on becoming a certificate authority, auditing said authorities, DNSSEC, DANE, taking over country code top level domains, Luxembourg, X.509, ASN.1, CBOR, more JSON (!), ACME, Let’s Encrypt, and more, on this extra lorge episode with the web PKI’s Batman.
It’s the history of the internet, which is just messy decisions compounding upon each other that no one can upset.
This rough transcript has not been edited and may have errors.
Deirdre: Hello, welcome to Security Cryptography Whatever. I am Deirdre.
David: I’m David.
Thomas: Hi, I’m Tom. I’m at fly.io.
Deirdre: I am a cryptographic engineer at the Zcash Foundation.
David: I, uh, well, gosh, what is my background?
Thomas: that’s good. It’s good enough for this one.
David: You know, I got a PhD in kind of cryptography from the University of Michigan a few years back. I also started the company Censys and nowadays, I’m at a company called Nametag. I live in Denver, but I did 10 years in Ann Arbor.
Ryan: Hi, I’m Ryan. I’m at Google, but as usual for Googlers, I’m not talking for Google. I’m just here talking.
Thomas: Deirdre, David, we have Ryan Sleevi you on the— I don’t know if you
Deirdre: I am so
Thomas: Ryan Sleevi on. This is amazing.
David: and it’s crazy because we get to see Ryan’s smiling face instead of a picture of his dog.
Thomas: I can’t tell you how you guys, how much time I’ve spent trying to find any picture of Ryan Sleevi anywhere.
David: I first met Ryan at Real World Crypto in 2016. We’d like emailed a bit before then, but I’m pretty sure I talked to you for like two minutes before I realized that you were Ryan Sleevi.
Ryan: Yeah.
Yeah, That sounds about right. I, I I’m surprised. I’m like there’s pictures of me out there.
Deirdre: Yeah, they’re all Karma.
Ryan: Oh wait. No, I took them down, right.
Deirdre: I’m going to stroke, uh, Ryan ego a little bit because, uh, we’re all very excited to have Ryan Sleevi on the podcast because Ryan Sleevi is basically a Batman of the web PKI. Like he does the dirty work in the shadows that you may not see to keep everyone safe and we, he may or may not be a vigilante and the way he goes about it, confirm or deny are you Batman?
Ryan: Uh, you’ve never seen me and Batman at the same time, same
place. So. Yeah.
Thomas: he is the hero we need, if not the hero that we want.
Deirdre: Extremely
Ryan: Oh man, you guys are really
stroking that ego here. Right? All right.
Yeah.
Thomas: Can you like give me a capsule summary of the role that you have, where you’re in every conceivable security standards group I can imagine breathing.
Ryan: Oh no, no, not anymore. Right. I that’s the, uh, the mistake I made when, when first coming to Google was that my, my nuclear project was, Hey, you should get involved in these standards groups. That’s a, that’s a good place for a new employee to start with. And, and you should work on this web crypto API
you know, crypto, um,
Deirdre: thing.
Ryan: So that was, that was, that was some fun starter projects.
So I used to be more involved, uh, and then have mostly backed off. But yeah, my, you know, at Google, I mostly work on certificates at large. Um, you know, I, I came in to the Chrome side, uh, after like two years of doing open source contributions to it. And that was mostly because I couldn’t get Firefox to build at the time.
Firefox has done a lot for their build systems, so credit to them, but, you know, it’s, uh, if you want to contribute to open source, you’ve got to first figure out how to build it. so yeah, I came in working on smart cards at the time and then just sort of branched into a bunch of things because certificates are everywhere.
and yeah, so the standards— you saw me in a lot of standards in the time, because it was that startup starter project of, oh, you should go get involved in standards.
Thomas: Adding to my list of topics for today would now be WebCrypto.
we should, we should talk a little bit about WebCrypto
Deirdre: Oh God.
Ryan: man, the story is I can share in the stories I can’t share, fun.
Thomas: Cool. in this podcast, we’re mostly interested in the stories that you can’t tell. So just like
Ryan: Right. Right. Exactly.
Thomas: no one will hear this. This is a safe place. So
Ryan. Let’s, let’s start here. Um, can you kind of explain succinctly what the web PKI is and what people kind of don’t get about what the web PKI is?
Ryan: So, Yeah.
I mean, I, I I’ll give you the succinct answer, but there’s also a non succinct answer. So I talked a little bit about this at, um, IETS 1 0 9. so there was a presentation in the security area group, that, you know, I was lucky enough to be invited and that’s, that’s sorta talking. And that was my first time trying to really crystallize the definition.
There, there actually been an ETF group that was chartered called like web PKI OPS. And, it probably comes as no surprise that after like two years of discussion still couldn’t come up with a succinct what the web PKI So, you know, here, when I’m invited, I get to make up my own definition. and so the definition sort of that I offered is, the web PKI is the, you know, the PKI that is that it’s interoperable within web browsers that, and I realized that feels like a, a sort of cheap definition, but, um, you know, as, as I talked about it in the SAG talk, there, there’s a lot of different PKIs and, and there’s this notion of say, very early on that this would be like an OSTP service or a system wide service.
Um, and that’s not what we got. Right. What we had was Netscape shipped SSL. They shipped a PKI and IE quickly fast followed, and these were both sort of browser things. and we’ve gone through, there’s been a few decades of excitement, but we’ve really circled around back to it where the web PKI is the, the site of, of CA. the set of policies and the set of implementations that are really tied to sort of web browsers from my browser capabilities. And that’s, you know, the, the obvious players here, you know, safari Chrome, Firefox, um, Microsoft, three, you know, high-end Edge. obviously Apple and Microsoft are at a little weird place cause they have this as like an iOS service, but then they also have stuff that they do in their browsers where they impose extra policies.
And, and so that, that sort of concept of, uh, the set of CAS and the set of policies they have to adhere to, to satisfy browsers sort of fits as the web PKI. and that’s, I guess, sort of similar, talking about standards groups now, like the, what WG exists, which is a bunch of, um, rendering engines working together to develop, the HTML living style. and that, you know, likes what browsers ship, right? Not in an ivory tower, imagination, XML based, you know, markup language, but the reality and the messiness of air handling and edge cases and all, all the weird grotty bits, uh, in HTML, you know, what WG and the HDML living standards. So you can sort of think of like web PKI is like a browser PKI living standard.
It’s that evolving
David: just to clarify, what is, uh,
what is a PKI in the first.
Ryan: A PKI is a great one. Soak up venture capital funds, or at least it used to be. Man. Oh man. PkI was the Bitcoin of the nineties.
Deirdre: Public key infrastructure, right?
Ryan: Do you do
Deirdre: Well, I, you know, I’ll just want to making sure that we’re all agreeing what these acronyms mean.
Ryan: Yes. Yeah, exactly. And so there’s, there’s a messiness rate in that term. cause like you can argue that Kerberos is a form of PKI, at least when you throw in some of the messy bets and I know it’s mostly secrets, but yeah, they, the PKI, when we usually talk about it is, is referencing the certificates and X five and nine In particular.
And usually for almost anyone sensible, uh, with reference to like, uh, peak X, which is RFC 52 80, which is like a subset of, what the telecom industry called PKI into the sensible thing that the internet calls speaking.
Thomas: and like, as you described it, I think you could kind of get the impression that like the governance of the web PKI that people like. So if the web PKI the most important product of the web PKI is the set of certificates that your browser will trust. Right. Like you could get the impression from that description that the web PKI is governed by Google, Apple, Microsoft, Firefox.
Right. But that’s not, is that really the case?
Ryan: you know, that it’s an interesting case, right? You know, I, I think there’s an element there that, that, you know, the set of CAS or certificates that, that any browser trust, right. Is, is really like a product security question. And then that’s, I mean, not to, to wax too political on, on browser discussions, but we see that like in other browser features, you know, we see, uh, you know, Firefox not implementing this feature or Apple raising privacy concerns with this feature.
And we don’t really talk about, you know, Apple governing the web, but we talk about them making a security or privacy decision for their product and for their users. And I think there, there’s an element of that, you know, when we talk about what BKI, but there is also this, this element, which is that any browser, you know, interoperability is, is a huge part of a browser stories.
And when it comes to like these features, if a browser doesn’t implement it, if it’s a good feature, they’ll find that their users want that feature. They’ll ask for that feature. They’ll demand that feature. And I think that’s something similar sort of when we think about what PKI is near the browsers are, are trying to balance the security needs of their product security needs of their users, but also come up with interoperability.
And that’s where I think like groups like the CA/B Forum sort of come into play is to make sure that, you know, when we make changes to our products, that they actually ended up being inter-operable and good
Deirdre: And what’s the cab forum?
Ryan: Yeah. Yeah, exactly. I’m just throwing acronyms out left and Right. Just not expanding, uh, the CA browser forum. Uh, and so the CA browser forum is pretty much just a discussion group of CAS and browsers. Uh, and it has its own storied history that I’ll I’ll save you. but yeah, it’s mostly just a bunch of mailing lists than periodic meetings.
So, similar to the IATF except unlike the ITF, we don’t really develop standards.
It’s
Deirdre: like, is there anything binding in anything that happens on the cab forum? it feels like it’s binding, but it’s, it sounds like just kind of everyone collectively kind of agreeing if someone’s been bad or not complying or not.
Ryan: Oh, not even that, right. The cab forum doesn’t even agree on like badness or goodness. So yeah.
there’s nothing binding the count form, what the count forum does. Um, and, and it exists for some really funny logistical reasons. So the CAB forum produces a set of standards that, the most prominent of which is called the baseline requirements, uh, and that’s for server certs.
And that basically is trying to capture here is the absolute floor for what it means to, if you TLS certificates that are gonna work in web browsers, um, you know, web browsers are of course going to put their own policies for their product based on the security needs. So a good example is different browsers will turn off Shaw one at different rates, right?
And that’s eventually it makes it into the baseline, but at the time it’s, it’s different browsers moving at different paces. Um, so the count form produces the set of standards, but the standard. And I’d say like standards, it really just guidelines. They’re not binding. Um, but what they do is those guidelines then get worked in by either CPA Canada, as in like charter professional accountants and AI CPA, into what’s called web trust. Or they get worked into a different group by Etsy, which is a, uh, European, telecom standards organization. They take those guidelines and turn them into audit guidelines to define how independent auditors can you. So like, there’s this element of the baseline requirements that exists, but they mostly exist to support, the development of these audit requirements.
And the whole reason for that is, if someone’s going to offer an independent and I’ll get into that. Like if they’re going to offer an independent audit, there needs to be some sort of broadly speaking industry agreed term for what the audit criteria are. So that’s what the cab forum produces is this industry agreed set that can then be turned into auditable things by these audit firms.
And I realized like, boy, I just dove into some really messy, weird, like it’s the history of the internet, which is just messy decisions compounding upon each other that no one can upset. Right. And like, or coming with big ideas that then they can never actually implement, like IPV6.
David: To step back from auditing CA’s themselves, which kind of implies that theCAsare a thing that need to be audited. let’s say this podcast is becoming a CA um, why don’t you tell us what we have to do to become a CA and also what is a CA
Thomas: We’re not, by the way, kidding. At the end of this podcast, we’re going to set, set in motion, the effort to create a new CA.
Deirdre: and not including, logging into my Google cloud account and spinning up CA-as-a-service that Google just released.
Ryan: Yeah, yeah. I mean, it’s, it’s perfect. Cause I know you had you on a flat bow on a few episodes ago and he went through that exercise of spinning up a CT log
that
Deirdre: Because it ran behind the
sofa.
Ryan: Exactly. Exactly. And the question is, why don’t you trust this log behind the sofa, which is, is an excellent question.
So, to unpack, you know, David’s thing, which is what is the CA ? CA is, either a certification authority or a certificate authority. And this is one of those like messy terms that who really cares the difference, but you will find religious
David: Yeah. I just saw someone asked this question, like nonchalantly on the internet somewhere. I was just like, oh God.
Ryan: Yeah. Yeah.
David: Let’s talk about tabs versus spaces next
Ryan: yes, it’s right up there. And the end of the day that the CA, whether you expose that as certification or certificate, they produce certificates, you know, and it certificate is this overly complex, terribly designed, very archaic form of binding claims with cryptography. And, you know, you have that fun conversation on JWTs that stick it in Jason and certificates just stick it in ASN one, which was the 1980s way of, passing things around, designed by the telecom firms. And I keep mentioning Telekom Telekom Telekom, because there was an interesting set of constraints, but it’s also not the, not the best technology stack to come out. Like we don’t talk about the OSI model much anymore.
And there’s a reason for that. so, you know, a, a CA is, just someone who means certificates. And in the case of web browsers, really what we’re looking first certificate is to match a domain name with a public key. So that way we know how to, establish an encrypted connection to it using TLS.
Right. So um, You know, hand-waving, over simplify. So, what does it take to become a CA? You know, and that, that answer has boy gone, gone through some fun changes over the past few decades. And I guess, I, I, the question to you is, uh, David, who are you? Why should I trust
you,
David: I’ll sell you a certificate for a thousand dollars. Tom. Can’t watch though, or he has to pay a hundred. Okay.
Ryan: boy? Yeah, that’s, that’s tempting here. I should make you a CA because you’ll sell it for a thousand. Well, I’m sure Deidre would sell it for, you know, nine 50. So why shouldn’t Deidre be the CA?
David: Cause I’ll sue her
Ryan: you
know,
David: I plan on maintaining monopoly.
Deirdre: Mr. Lebowski.
Ryan: All right. So, you know, the, "how do you become a CA?," is, is really the, the core question that, I guess I should jump into a term before I really answered that, which is just we’ve talked about browsers and CAs. Really, we call this collection of CAs and a set of policies, like a root program.
So, you know, there’s a Chrome program, there’s a Mozilla reprogram and it’s the collection of CA’s that they trust. and if we think about CAC organization or CAs, the certificate that definition holds, you know, cause a single organization may have many certificates, but they’re all part of the same organization.
Or they might just have one. but a re program is like the collection of the certificates, the collection of the policy, and generally the implementation of the software that uses the certificates. and so if you want to become a part of a reprogram the question for, pretty much every reprogram is why, what, what are you trying to do?
How does this relate to my use case? why should I, I trust you?
Thomas: let me ask a question real if I like, took the names of all the current CA’s that are trusted by the Google root program and put them in a hat and I stuck my hand in and pull one out at random and I asked, "why?", would you be able to tell me?
Ryan: I could tell you in the abstract, depending on who you picked, I can tell you specifics, but, uh, so there’s in the Google root program. I think it’s, uh, about 57 organizations right now. Um, but a bit more number of certificates, but about 57 organizations. And that number, the reason I don’t have that exact is it, it, it adds and flows with like mergers and acquisitions or new new programs coming in.
Deirdre: Quick question that you don’t have to answer. If you don’t have to: if you get hit by a bus, is that knowledge just gone for Google and their, root CA program? Or is it only partially gone because some of it’s written down somewhere.
Ryan: it’s only partially gone. So, David, you know, uh, worked, as an intern
David: Slavey reviewed all of my code when I was an intern, which is weird. Cause he wasn’t at all in the hierarchy of where I was working.
Ryan: yeah. Yeah. So, so we have a team and in fact, you know, we’re, we’re hiring for the team and we’ve hired, uh, and bringing on board and a number of folks out of our Washington DC office. So I was tweeting about this a few months ago, and I’m really excited because we do have a growing team. but yeah, even at Google, there, there— the bus factor or sleepy wins the lottery and decides to go off somewhere fun and relaxing".
Um, there, there’s definitely some robustness, but, but David also knows that I have a penchant for, for recording useless trivia, like, and just bringing it out at the worst possible time. So there is going to be a loss of, like, the pet industry, but, uh, I think, I think people would call that a win not a loss.
Deirdre: Okay
Thomas: team that you’re talking about right now, you’re talking about the team that manages the Chrome root program, right?
Okay. like we’re, we’re dancing around it a little bit. Right. But like, what we’re essentially talking about is the set of CAs that will be allowed to issue certificates that Chrome will trust, which is like a huge fraction of the internet.
Right. So, um, right there, there’s like an immense amount of power and control over, you know, not just like who who’s allowed to be in business, doing CA stuff, but how they’re going to behave and comport themselves on internet. And we can probably think of like horror stories from the last 15 years years of, you know, people doing just bat shit, crazy things with their CAs, that we all are glad that they don’t
Deirdre: Or negligent things.
Thomas: question I’m super interested in and like an opportunity to sell your role, what’s like the roles that you’re hiring for. It’s like, what’s the day to day of somebody working on that team. Like the people that you’re hiring, what’s the work that’s like, what does that look like?
Like we’re, we’re pretty technical. I mean, we’re, also kind of dancing around the fact that I think most of us know a lot of these terms and we’re just trying to do a better job of explaining them to people that are listening for the first time. But I’m genuinely curious about like when I come in next Thursday, after I take a job in the Chrome root program, ‘cause by the way, that sounds awesome to me, what am I doing?
Ryan: Yeah. So, we have a couple of roles, right? We we’ve got, uh, some folks who are focused, you know, heads down on engineering. So I can talk a little bit about like the engineering that’s happening now and the engineering that that’s coming. So, um, right now we are launching, you know, work I’m saying ‘we’ hear, and I, I now have to correct myself, right?
Cause I,
said, I’m not speaking for Google, but Chrome is, is working on this. And this is all public, right. Chrome’s working on launching a, a certain verifier. and the reason for that is it currently integrates with various S cert verification stacks, which means you get a different behavior
Chrome, on windows versus Chrome on Mac, and you’ll get different errors
and different messages and all sorts of unpredictability, which is terrible as a web developer, because the whole appeal of the web is, write once, run anywhere, it’s, you know, Java, but without Java, I mean, it’s Strava script though.
So,
Thomas: you’re about, cross-platform, taking ownership of how, you know, Chrome related products, validate certificates instead of delegating it?
Ryan: Exactly. Exactly. So that there’s consistent performance and consistent characteristics, whether you’re on mobile or desktop, and then you can debug and understand what’s what’s happening. And so you can make sure that your websites work everywhere, which is great. And this is something right— firefox has always had, because that’s just, they always had it. They never integrated with us, which is why they were loathed by everywhere
David: Firefox uses a NSS. If I remember correctly
Deirdre: Yeah,
David: as their verification stack
Ryan: Yeah. and NSS is strangely enough. I learned part of the Linux standard yeah. base, but like in, in theory? I’ll often district ship it, but yeah, a firefight ships, their own copy of an SS, they ignore the, the LSB version.
They shift that cross-platform and of course, other browsers generally have not concerned themselves with cross-platform behavior like Safari on windows, not so much a thing now you’re not getting as much core foundation on windows as you did in the iTunes
Thomas: and it’s been a while since I, like, I looked at that code and said I liked it. And then a million people dunked on me for saying, I liked NSS, but NSS is basically a competitor to the TLS bits of OpenSSL, right?
Deirdre: Yeah.
Ryan: Yeah, Yeah, exactly. So, so NSS came about, because, at the time of Mozilla’s creation, so right. We had Firefox, from Netscape and then, you know, uh, sorry, Netscape navigator, and then open source and is the open source foundation. We have Firefox that came out of the open source thing of Netscape and, you know, uh, leading a whole bunch of history on that.
and the issue was they could not release the source code for the crypto bits um, because crypto politics was a big part of it at the time. This was if I recall my history correctly around 2001. Um, and so there was actually a discussion of, of Firefox, uh, you know, the, the new Mozilla Firefox using, uh, open SSL, as the basis of its TLS stack and certificate verification stack.
And, uh, eventually Netscape was able to get, uh, those spits open source for the crypto. And that became the Zilla NSS, uh, network security services, as a library, um, which had previously been, part of, navigator. So yeah. It exists more or less as an alternative crypto stack and, you know, open SSL has a TLS library, it has certificate validation as low-level crypto; NSS similarly has, those, concepts.
I will mention though, like, NSS has three different certificate verification, completely independent certificate verification stacks. the, the one that they originally wrote for, uh, Netscape and the way beyond, so what you know is, known as ‘legacy’.
and then they, a bunch of grad students decided to convert Java to C with exception handling using macros. Uh, and this became, what’s known as lid peacocks. And so lippy kicks is, like six layers of sea macros that exist to emulate Java behavior and see, and this is what Chrome used for a long time on the Linux,
Thomas: why NSS is better, it’s because you have options.
Ryan: okay. Right. Right. Exactly. And you know, Java code in C what could go wrong, exception handling through macros, what could go wrong. Um, and now Firefox rewrote their certificate verification stack a few years ago. Um, and they, they call it Mozilla P kicks. And now this is also part of NSF. sorry for that, you know, otherwise boring history thing, but there’s so many ways to verify certificates and each one of them have different behaviors and policies.
Um, but we were talking about root programs. So stepping way, way back there, There’s an element that’s that certainly says, you know, the decisions Chrome makes on, on what CAs to trust or not trust, you know, has an impact on users.
But, uh, certainly over, history, uh, the decisions Mozilla makes has been far more impactful just because of how many open source projects, uh, derive their behavior from, the decisions that Mozilla makes. Um, as well as most Linux distros right? Most Linux distros have something called the CA certificates package that they ship, which more or less as well, Mozilla said, it’s good.
So Mozilla is, uh, the open source golden child and therefore
we don’t
Deirdre: so wild that that’s like, uh, influence of open source that I wouldn’t even have thought of that like, just because this is an open source way to verify certificates. See how you do it. It’s not just sort of like prodding a black box that everyone’s just sort of like, "do what they do because we can see what they do and we can’t see what other people do".
Ryan: Yeah.
And, and you, you hit on a big point there. And one of the reasons that so many folks look to the muzzles reprogram one, there’s the element of the code, but the other part is Mozilla has always been very open. And so their policies on sort of what CAs do we trust and why do they trust CAs? Uh, has always been very open.
I shouldn’t say always.
since 2006, which in tech terms is practically always has been open. and so, you know, the question going back now to the question of what does it take to become a CA right? and the Mozilla model is, is really one to look at. Um, and the, you know, if, if you follow the Mozilla model, you’re, definitely rising to one of the higher bars out there.
And so the miserable model, right, is the first thing you have to do is you have to create your certificate. You know, what is the thing that we’re going to trust? Well, you have to make up your, your Sur. and the thing about, uh, you know, I mentioned the baseline requirements and the cab forum, many root programs have incorporated the baseline requirements as like, this is the, you must be this tall to Uh, so even though the, cap foreign product isn’t binding, products incorporate that into their own requirements and make it binding. So when Mozilla says you must follow the baseline requirements, now the baseline requirements are binding. If the didn’t say that, like you could do it, you can not, it’s up to you, whatever, man, you know, assign yeah. not a cop". you know, going back to creating a certificate, the baseline requirements say you have to get a bunch of auditors in a room to witness you birthing a key from nothingness and you have to birth a key from nothingness onto special hardware. And, you know, the baseline requirements required has to be a FIPs one 40 dash two. Yes. It says, dash do -not dash three, but if it’s one 40 dash two certified device, and, and I say certified, and, and anyone who has worked with Phipps, no, I’m just making up words here. but you birth a key onto hopefully something that is not going to just spit the key out to anyone who comes by and your auditor’s witness you birthing this key because you followed a ceremony with a script. and, and once you’ve done all these things, you know, you have this new key that is born and then you can create a self-signed certificate. And now congrats, you’ve created a routine. and the idea here is from that moment on, you need to play by the rules and the auditors.
You have the auditors come back at a later point to make sure that you’ve been playing by the rules and the idea of why, why, why am I talking about auditors? Can’t you just trust me to play by the rules is because, you, don’t, I don’t trust you. Just Like you said, you did something. Why should I trust you?
The idea of auditors is, get an independent, hopefully neutral third party to come in, uh, look at what you’re doing, what you say you’re doing and look to see if you can provide evidence to actually say that you to actually show that you did what you said. they, they work from the assumption that whatever you say you’re not doing until you provide enough evidence that convinces the auditor, that you’ve probably
done what you said.
Thomas: in, north America, that audit process is basically WebTrust?
Ryan: yeah. So in, in north America, it’s, pretty much a web trust and most of the world it’s it’s web trust. and that’s a standard developed by CPA Canada, AI, CPA that, you know, which are the two— it’s the Canadian and, American, professional associations for charter professional accountants.
but then there’s, uh, an international standard that allows other countries to use those guidelines. And that’s called ISAE 3000 and I can save you a bunch of audit terminology, but the, the idea here is that professional auditors and whatever their appropriate jurisdiction, you know, can produce these reports.
And what that means is like, David, you good? decide, declare yourself a web trust auditor, and like, say I’m just going to make web trusts and reports. and if you want to do that, you have to go get licensed by AICP or CPA, Canada, if you want to use their terms. And so that’s, that’s the sort of catch all is, is web trust provides a licensing for
David: how hard is that licensing to get?
Deirdre: I was going say, who wants to go get
David: I’m licensed as a convenience store in the state of Michigan. That’s actually true, but that pretty
easy
Thomas: this is our new plan. We’ve gone from creating a CA to creating a new WebTrust auditor. If this sounds
David: oh, no, we’re going to create both. That saves a lot of money.
Deirdre: We’re to have David certified that we are a up to code CA.
Ryan: right. Exactly. As long as, you know, David can assure us independence and neutrality, why not?
Um, Yeah. So there are, there are licensing fees that, uh, web trust runs. And this is one of the interesting things that has changed over say the past decade is, it used to be a lot easier just to become a web trust auditor, to, you know, pay the fee.
And if you were one of like the big four firms, you could sort of pay the fee once and then all of your branches and every country can just get covered by that fee. whether or not they necessarily had the professional skills or understanding for what it is they were doing. The, fee was the fee and web trust. uh, you know, CPA Canada has recognized that, uh, oh, Hey, there there’s some brand damage coming to the web trust brand, if just anyone can become a wet trust audit, or we should care a bit more. So you need to be at a professional accountant, but that’s also not the most difficult thing be. And, you know, go through your CPE courses the same as you would for a lot of other professional things.
And then you can become
Thomas: IS web trust, good? I spent a big chunk of my career, quote, unquote, auditing software for So like, I would imagine, like, when you say like a CA gets audited, right? Like a lot of us would have a conception on our head of like, oh, well, they’re sending a team of technical people out to look for vulnerabilities and gaps and things like that.
And, uh, you know, that those people understand cryptography and things that. Is WebTrust that? is there a place where that like comes into play? Are there like serious penetration tests and audits and things like that? Or.
Ryan: Generally, no.
David: Oh,
Ryan: right. I know just, just the
answer
you wanted to
hear. Um, so
David: but also
Ryan: what.
David: given a decent technical standard, there— a large component of this is like non-technical rule policy that be followed, right? Like the fact that only— not anybody can just walk up to the HSM and issue, a new certificate is something that no matter how you swing at is going to have a large organizational or physical component to it.
Ryan: Yeah.
And I guess that the ‘no’ is more to Thomas’ question. I was thinking about it as pen test or security audits. the, the history of web trust. And I gave a presentation to the cab forum about this, I guess a few years ago that actually looked through the whole history of these audit standards.
Where did they come from? if you’ve ever worked for say like a company that has like a SOC audit, right? Yo I’ve got my SOC two on it. which looks at this, this notion of what they call principles and criteria, uh, which are, are very abstract things. Um, that’s why they’re just principles.
Um, you know, do you handle things, uh, with integrity and confidentiality, you know, that’s a principle. What does that actually mean? And the idea here is you write criteria that says like, this is what we specifically do. And then the auditor assesses whether or not you actually do that. And so it’s, it’s a bit more of, um, I don’t want to say like a documentation exercise, but that is a huge part of it, which is saying like, here’s how we meet this
abstract goal and then the, the baseline requirements and the wet trust standards that are derived from the baseline requirements actually do, as David was saying, have a number of very specific sort of technical requirements. And then those, those have to be mapped in, but you’re not necessarily going to see say, um, an audit for I’m doing a pen test on a system.
What they may check though is, you know, what is your patch management strategy? And do you have evidence that you’re actually following that patch management strategy? Um, so rather than doing the pen test to see, you know, are you running the outdated version? They may look at show me the evidence where you applied the fix for this version.
and so it’s, it’s more like those type of audits or an ISO 27 0 1 audit than it is, uh, like a code audit or security assessment.
Thomas: I’m like to like, be the voice of HackerNews in this, uh, this weird podcast we have. Right. Cause I’ve had this discussion so many times there. So like, I think like a burning question that people have about this stuff is like, if you’re a technical person and you’re thinking about CAs, like a big question you have is just like, what are the inadvertent technical mistakes, if make that are going to compromise the security of their CA kind of like, the big thing that we’re worried about here is like misissuance, right?
the idea of like, you issue a certificate for, you know, Google and you’re not Google, right? is there a process by which like the CA software that you’re running your ACME implementation or like how you’re doing DNS lookups for, you know, domain, you know, domain certificates and things like that.
Is there a place where the certifying process for the CA, gives you any kind of confidence that they’re, that people are technically on the ball
Ryan: Oh, man. You know, you get into one of my favorite topics. This is an area where I think a lot of browsers have given feedback and, and, you know, we’ve talked a lot about web dress there’s of course this other audit standard Etsy, and, you know, again like pointing to public comments, but not speaking on behalf of Google, but, you know, the Chrome root program actually is very explicit that it prefers web trust over Etsy and may discontinue Etsy at some point in the future.
Um, and that’s because what you’re getting to is, is how much assurance does the audit process, the auditors, the audit skills, uh, how much does that match with what, you care about as, as the end user, right? As the, as the end user, you’re like, I want to make sure there’s not a google.com certificate issued by to someone who is not google.com.
I want to make sure their HSM is not just blasting, you know, the private key over the airwaves to anyone who is going to listen in, or I want to make sure that they have not stuck the HSM under someone’s desk, which a CA did, in fact, multiple CA’s have done of like, oh yeah, we just put the private key under someone’s desk.
And oops. Like there was
no physical
security.
Deirdre: walk away with
Ryan: no big deal. Yeah.
Yeah.
if you look at the discussions around removing trust in Savannah tech, one of the audit reports, uh, noted that that one of the Symantec subsidies have no real records around the physical security of their private key.
Um, and they had as just as much capability as Symantec for issuance. so, to Thomas question, like, why should you feel confident in the current audits? The answer is yes. look at it with a healthy dose of suspicion. You’re like, I don’t know whether this is gonna make you feel better or worse, but a vast majority of CAS run like EJA BCA, which is the, um, Java CA software that is everything and the kitchen sink all in one, because it is the CA software that you use if you’re running a production line and you want to, you know, stick certificates to show this as a legitimate device or you’re issuing employee certificates, like it is the certificate Swiss army knife, which is both to their credit, and like, if you think about Heartbleed, Heartbleed was taking the kitchen sink of crypto standards and just putting them all in open SSL, EJB CA does have that reputation of, of, they’ll implement anything, if you ask. And particularly if you’re a paying customer.
Thomas: Google we’ll has some kind of hardcore technical audit requirements for some things that they do. Right? Like, I, I can think of at least one of them where there’s like a specific set of fairly hardcore pen testers that are the only people allowed to certify, right? Could you guys like tomorrow come up with a criteria for like, okay.
From, you know, from, from, you know, this date next year, you have to be audited by one of these three firms that know how to audit CA’s. and if you’re not the one to distrust, you could, you guys just do that by fiat, if you want it to?
Ryan: You know, the question, that’s interesting that you get to, you know, I, again, not speaking for Google, when you talk about the, you guys, I know our PR person is going to make
sure I said all these disclaimers,
So, you know, you’re describing what is common at a lot of firms like vendor security audits, right? If and you’ll see this at a lot of tech companies, Right? If you’re going to say, you Slack, is your chat app, you, uh, in fact, there, there may even be, requirements like employers by legislation that you have a vendor security audit process, where you, as the one who’s going to use Slack, you need to do it and assess how well Slack does.
And, you know, Slack will provide you a bunch of standards and say like, oh, look, we comply with these standards, that’s why you can trust us. And then the vendor security auditors will say, Okay, well, you’ve got these standards checklist and you gave answers that did not scare the bejesus out of us to our questions for the things we care about.
And so we’re good with it. and so that?
is, I think, a big element of what. becoming a trusted CA and, and really when we talk about trusted CA and a reprogram it’s similar to, uh— the parallel I draw for browsers is, imagine your browser’s going to ship a third-party extension by default. What would it take to ship that third party extension to every one of your users?
Or if you’re, you’re Microsoft, right? What would it take to bundle a piece of third-party software shift, every one of your users? What are the things that you need to make sure of before you ship that to all your users? Now, users may go install software. They may go install extensions, and they may go and spell it.
Yes. And at that point, like the users made a decision. You you know, you can lead a horse to water, but you can’t make him drink. You can’t really make sure users are going to make the best decision, but you can do your best to help them. but when it comes to the things that you were shipping, pre-installed right,
when you go download Chrome or Firefox or Safari, Brilliant in case, you know, macro S what, what’s it going to behave like out of the box and what’s the step in by default? so you could, you absolutely could. And in fact, that’s what root programs have increasingly shifted to. So Microsoft has announced, for example, that they require a bunch of due diligence on the beneficial ownership of CAS that are in there forever.
So it used to be that, well, you incorporate it somewhere, right? And you filled out some paperwork and you told us you want to be a CA. Now Microsoft requires a bunch of due diligence, and it works carefully with their legal team to actually check through that, that ownership. And you have contracts. Uh, if you want to become part of the Microsoft reprograms shift for windows, you need to execute a contract with Microsoft.
And that goes through, you know, all of the legal contracting process, uh, with that. and that the contract is lovely because it has some lovely teeth and penalties. And what happens if you decide you don’t want to become a CA tomorrow? Well, guess what, there are, Microsoft still has you know, powers for that contract because you don’t just get to unilaterally walk away from the contract.
Deirdre: Apple do that?
Ryan: so Apple’s not yet, and in fact, Microsoft is the only one that currently has contractual requirements and that’s something that, that other root programs have discussed. Um, I guess it doesn’t really come as much as price, right. When you think Microsoft, do you think lawyers, uh, you know, so it, it works and it’s, it’s, really smart for, for them.
You know, if you think about the risk profile for Microsoft as well, it’s smart to have have those contract.
Thomas: It’s Mozilla, Apple, Microsoft, Google, right? Like who the fifth important
root program?
Deirdre: Uh, Amazon,
Thomas: Does Amazon,
Amazon doesn’t have program
Deirdre: they don’t ship. Uh, Hm,
Ryan: So, historically you had Oracle in play because of Java. You had Adobe and play, ‘cause if you’re doing anything with like document signing certificates, but Adobe also, uh, because Acrobat of course can make TLS connections in your PDF. Um, you know, it a TLS root program in there. Um, and, and for Java, of course, you know, um, There that you need a root program for there.
those are the main ones that, stand out and come to mind. You, you do have, smaller programs that depend on, different use cases. So for example, for a while, the Wi-Fi Alliance had a reprogram. If you don’t know the wifi Alliance, they are the ones who, you know, will give you a certified wifi Alliance sticker that says you can implement wifi standards correctly.
Um, but they also worked on standards to allow you like hotspot roaming. Um, so they have this thing called PassPoint, which allows you to roam to from different wifi networks that may require payments. And so they have a root program, for PassPoint that, that talks about like, what are the wifi Alliance CAs that, you know, you’re talking to a Wi-Fi Alliance hotspot, and then you can do this network handoff, or give money to whoever
Thomas: I feel like I have this impression that like, Th there’s a lot of things that you, that Google would do, let’s say for the rest of the conversation, let’s just say that we’re talking, we’re speaking for the Adobe brute program. But there’s, a lot of things that the Adobe, the Adobe root program would do, if they could just do whatever they wanted by fiat, right. But that, there’s a sense in which the CA’s have kind of a veto on things sense, like I’m probably wrong here, right? Like I just follow the mailing list. Right. But like, I feel like there’ve been like discussions or something resembling votes on, you know, new BAR requirements, you know, at cab forum and things like that, where like, you have not reliably gotten your way.
Right.
Deirdre: huh?
Thomas: Which is a tragedy, the way, you like, I I’d happily sign the thing that just gives you control of all this stuff.
Ryan: Yeah. So, there, there was a lengthy discussion about exactly that point in the cab forum around this ballot that was known as Um, and so that’s the 31 was called the browser line and ballot. And what it tried to do was take existing requirements. You know, some that were decades old from different root programs and bring them into the BRS. Because if you think about it, right, if you’re a CA and you want to issue certificates that are going to work in, in web browsers, you don’t design just for Chrome or you don’t design just from Microsoft. You really want them to work everywhere because that’s your whole selling point to customers is our certificates work everywhere you want to be, right.
these will come after you. If you. say exactly that, but close enough. And so the, the goal of 31 was to actually bring those requirements into the baseline requirements. And it turned out to be rather controversial because there were some things that CAS didn’t like the root programs were already imposing.
Uh, but that CAS felt that that were unnecessary. And so there was this lengthy discussion about like, what, what is the role of the cab forum here? And at the end of the day is, you know, if, if the cab forum say were to, to veto a bunch of changes to the baseline requirements, it’s relevance decreases, and that’s true for any organization, right?
If your goal is to produce voluntary standards and your voluntary standards are useless, everyone’s going to ignore you and go somewhere else or, you know, do the other thing. So there is that. But equally, I think, I think all browsers are concerned on wanting to make sure that one, they’re doing something interoperable and two, that they’re doing something achievable, right?
Like if a browser says, you know, you must jump a hundred feet every time we say jump, like that’s really hard for a human to do. It’s just naturally unassisted. You have to think about like what’s my hundred foot jump system, you know, you’d have to start buying trampolines or bungee cords to really make sure that you can get that, that full hundred feet.
And that takes time. Uh, and I think browsers and reprograms are aware of that. Right. And so there’s always this balance here of finding what is the data to figure out what is reasonable and what is the balance for security.
Thomas: Do you feel like generally, like the root program itself, like the Adobe root program, anything it wants, it’s going to be able to get like, just through the whole system, through all the layers of this, right? Like if, if I tomorrow walking at my job at the Adobe root program and say, you know, from now on, nobody gets to run EJBCA anymore.
Right? Like, could I do that or is there like a, is there an organizational reason why I couldn’t like reasonably, add a requirement like that?
Ryan: I, it’s the game of, asking the Adobe root program manager, uh, what’s going to happen if no one listens to you. right. are you prepared for the consequences of that? Right. And that’s always the challenge for making any, any changes in an ecosystem, especially a multi-stakeholder ecosystem is. If you’re going to say this certificate is not going to work in your product anymore, are you prepared for the consequences of that? Right. You know, and, and you know, the consequences, I like users can’t access the site. Our user’s going to start using other software. Is that, is that something that you care about or if it’s a paying customer, right.
Is your, your paying customer going to get upset? So there is this, this element of, you know, and this is true though, with any software change, right? Like if you decide, you know, Fly is, is going to, massively, you know, change how its API is behave and suddenly every one of your customers is going to be broken.
You’re going to have a bunch of cheese off customers and probably won’t be customers for very long. And so that’s, that’s the, you know, tension is you have to figure out how to balance that. And for a browser, the concern of course is always, am I going to break something? And so the balance, like how do you balance that?
Well, you introduce requirements that have like a phase in date, say like new certificates issued after Tuesday. You know, assuming Tuesday’s a reasonable day for change, new certificates issued after Tuesday have to behave like this. And that way, you know, a hundred percent of the certificates out there don’t matter.
Like they may not comply, but that’s Okay.
They’ll continue to work, but come Tuesday, everything needs to work. And so how do you pick Tuesday? You pick Tuesday on how well you think It will be achieved. And that a hundred percent of the new certificates on Tuesday, we’ll be meeting that new request. and you see like Apple does this with, with most major OSTP releases, right?
they made major changes to their reprogram and, uh, iOS 10, 13, and then further in OSTP 10, 15. And they said, you know, here’s, in the case of Apple, the big controversial one in this se 31 with lifetime, they said, certificates need to be valid for 398 days or less. And here’s the definition of what 398 days. Right. Here’s the number of seconds in a day. And here’s how you deal with leap seconds.
And this is, this is, what time means, which are all
Deirdre: Matters you’re negotiating your TLS connection and your certificate and your local host has clock skew. And then like your certificate is according to your skewed local clock, uh, too old. And it does not verify. And then you can’t connect to, you know, google.com or whatever. Google never has this problem, obviously, but there have been plenty of other websites have had this problem.
David: Or imagine if you’re a CA and you put in all that work to comply with the rules, and then it turns out you issued your certs for 398 days and one second.
Ryan: Okay. Ah, yes. with apples approach, right? They, they have the policy requirement and then they also have the technology requirement because they expected and unsurprisingly as happened, CA’s fail to implement the policy change correctly. And so they still had to take steps on the software side.
Sometimes that works for policy changes, that doesn’t always work. Um, you know, and so that’s the balance, but you know, to the Thomas your question like fiat, that’s, that’s a big part. I think, of a root program management and, figuring out which is, how to make changes that are going to improve the security of your users, be aligned with your product needs your security goals, because these are ultimately, you know, the set of CA’s that you’re bundling in your product are CAS that, Exists to help you achieve a product security goal and you want CA’s that are going to be aligned with that. And how much time do they need and how much time is a reasonable.
Um, and that’s a big part of the cab
Thomas: you’re not saying this and you’re not going to say this, but I’ll, I’ll just express my opinion here that with one with probably just one, one exception. The CA’s are a force for evil, right? that they’re there I mean, th they have a business goal of issuing certificates as cheaply for them as possible at the highest amount that they can issue certificates for.
Right. and that generally they’re pretty obstinate about, you know, imposing new requirements that would, you know, make it safer and make it less likely that they’re going to misissue. And that, you know, there’s a pretty widespread mistrust of the web PKI kind of as a result of this belief that, the certificate authorities are entirely mercenary, you know, and entirely resistant to— mean.
We also, we just, we know a lot more stuff about securing, you know, networks, applications now than we did, you know, 10 years ago. But I don’t think there’s a widespread perception that CA’s have kept up with that. And part of that is also like, I think Symantec Symantec was Verisign’s CA right? Like that was the largest,
Ryan: it
was many CAS.
Thomas: brand name that think everyone thinks about para, or maybe just cause I’m old.
Right. But like Verisign, used to be /the/ CA right. there were just where you got certificates from and like,
Deirdre: You put your little badge in your webpage because that fucking matters.
Ryan: Yeah, you add Verisign and you add thought, and guess what Symantec was both of them. They had acquired both Verisign and thought you had to the two Oh, GCAS right. And the context, right. Is there assigned, was spun off from RSA, has the original sort of holder of the patent, and in conjunction with, uh, several companies, Lisa was one of the companies.
I think Microsoft was, was another as to spin out Verisign and create this, this group. Encourage and promote public key cryptography. If you wanted to license the patent for RSA, you had to include various signs, public key. That was the catch. and then thought existed because uh, you know, mark Shuttleworth was like, Hey, there’s crypto export laws.
And it’s hard to get strong crypto outside of the U S so we will create a non U S entity that sort of ignores the U S restrictions and so thought existed to provide strong crypto outside of the U S and Symantec acquired both of them. So, yeah, like, don’t know that I would agree with your assertion that CA’s
are necessarily mercenary or evil.
Well, I think you are overlooking the theory. I’ve been a lot of really positive contributions from CAS, right. And that, um, I think there are CAs that are doing a, lot to try to find the right balance because you have certificates, rating and certificates are, are pretty much this, it’s fundable it’s commodity.
any certificate from any CA is just as good as any other certificate from any other CA because they’re all held to the same standard. And they all provide the same assurance, which is a binding of a domain name to a key. And you’ve had a lot of, it’s certainly true that there have been CA’s that have tried to distinguish themselves by saying, we provide something more than that.
Right. And you have this notion of like EBV certificates or extra strong validation. and the end of the day, it’s still a domain name and a key. And that’s what matters to every piece of automated software. And so your only value proposition is to the non-automated stuff. And that’s really hard to quantify it even exists in the first place.
Deirdre: let’s encrypt pioneered quote, unquote pioneered at least seems to pioneer. If you’re just like a general developer, trying to get a cert for your web application so that you can serve stuff over a TLS, does ACME, so that you can, automagically be like, here I am hosting something on this DNS and ACME will verify that you own and control this DNS location here.
You get a cert for 90 days and it’s very automated and it feels more like it’s not just a bunch of humans just being like, yeah, it looks good to me. Like they can get social engineered into issuing a certificate for, you know, my domain.com or whatever. Is that accurate or is that sort of being like, well, no, they’re actually CAs who are doing stuff like that under the hood.
You just didn’t really hear about it until let’s encrypt made it explicitly, like, this is the whole point we’re trying to automate this, make it cheap, free, and this is how we’re doing it.
Ryan: Yeah.
yeah. You had a lot. I think of let’s encrypt as sort of pioneering the combination of a lot of things and really executing very well on engineering side, but a number of these weren’t very new ideas, right? So you had a StarComm, which was an Israeli CA uh, and they ran the CA you know, start SSL, which offered free certs.
And you could get a free cert for one to three years, you know, depending on sort of, uh, the types and the cert was fully free. it wasn’t necessarily automated. It did some weird stuff with clients and eventually, uh, start comm sold themselves and a very sort of, um, infamous debacle that later led to the distrust of, of the acquiring CA whoa sign. But, um, Starcom, you know, was one of the free CAS. Um, there were several other free CAS, um, Komodo, uh, which has now rebranded the CA business into sec Tigo as a new ended. they spun it out. So Komodo has gone long, lists actio. At the day it was Komodo. They also had free certificates out there, you know, that, that predated it.
And for 90 days, and there were, there were several other CA’s that would do 90 days starts now for some of these 90 day starts. They wouldn’t let you get new ones or they wouldn’t let you automate. So that was their hook is like it’s 90 days and it’s extra painful. So you want to get and pay for the foster. Um, on the automation front you had, CA’s like global sign and digital cert and pretty much all the major USDA’s and interest, um, had automation portals. And in fact, the automation portals would handle a lot of the domain validation, automation and plugged in with a variety of absolutely terrible protocols like scab or their own hand-rolled protocol.
But that works like if you were a window shop, say running active directory certificate services that speaks all these protocols, you could glue these two together and, you know, you could get a certificate from GlobalSign from your HTCs domain controller, you know, and everything just sort of worked and it was magical.
So they had like a big focus on automation. But they treated it more as like a service contract. And, and, you know, if you looked at let’s encrypt, let’s encrypt combined them. And of course, even now we have several other free CAs that have done that, uh, bypass Norwegian, CA now offers, uh, free certificates and sports acne. I think they’re six months, zero SSL, which is really just a rebranded or, or white label brand of set Tigo also offers free certificates. DigiCert supports acne for some of their, uh, other certificate types. not necessarily even the free certificates. So you can use ACME plus paid for certificate.
If you’re looking, say for Digitor to provide support for those certificates and, you know, support for certificates is a, like thing. Let’s encrypt will not support you at all, which is why they brought a lot of software to hopefully manage. But, you know, if you’re trying to support say older devices and let’s encrypt cert may not work, you need to figure out from another CA and then you have to of course, work with that CA to figure out what configuration and magic incantation is it gonna take.
And, CloudFlare released a tool, uh, CF SSL, which exists to help you understand that, hopefully to reduce the support cost, but, you know, you plug in a certificate and they designed it for CloudFlare of, you know, you give us a certificate, we’ll have to figure out what the configuration needs to be to actually make this certificate work.
Uh, historically that was where a CA support team existed to, to provide you that handheld holding personal guidance for whatever software you do dealing with. so Yeah.
let’s encrypt was, was useful, uh, for a lot of things and combining these concepts, and of course, ACME is. it’s a standard that tries to do It’s simple. Um, you know, for some value of simple, but compared to every other certificate issuance protocol is just this horribly Baroque and complex thing. It’s, it’s so much easier. And, so, you know, on the topic of certificate management protocols, when the IDTF took on certificates, they always assumed that they would have one true certificate management protocol.
Like the idea was, Oh yeah, this is easy. Well, one to find a spectra certificates and two we’ll define a management protocol and it turned out like the management protocol took, uh, you know, two decades more and a whole bunch of infighting later and became tremendously complex. So
at least they
Thomas: format that, that certificates use only the actual, you know, the bits and bytes format of certificates is X.509. right? Is there any defense for X.509?
Ryan: Is there any defense for Jason? Is there any defense for C or,
Thomas: To both, of those? Yes. I probably give like a, you know, a one
sentence
Ryan: I I’m intrigued.
Thomas: Jason’s really easy to parse, um, as universally supported it, it’s, you know, it’s a borrowed notation from a very popular language. it has a bunch of basic features that are like a good, like kind of, common denominator for what people need for storing you know, bags of attributes. I don’t love JSON, but like, I can see why JSON kind of took off. Right. And CBOR kind of responds directly to, okay, we have JSON and it’s really verbose. Right. It’s uh, you know, JSON messages are really big and CBOR is a straightforward encoding of the same concept. So it would be like how I would… X.509 is a lot more than that. Right.
Ryan: Yeah.
So, so X five and nine, right, is if you think of it, it is the schema, but then there’s like a schema of language that actually defines like how you, how you go from a schema to bits on the wire. So, you know, if you think of like Jason schema and Jason, right there, there’s X five and nine, which is the schema.
And then there is, uh, ASN one, which is the, sort of how you turn this into bits on the wire. And there’s specifically this notion called distinguished and coding rules, which is how you turn a particular schema into a hopefully unambiguous series of bits and bytes. To, to the thing of like, what makes this valuable.
One of the arguments against race on right is it’s untapped, right? Uh, it could be a string. It’s a number, a number, maybe it’s a, you know, two to the 56, maybe it’s students 64, maybe it’s somewhere in between. Who are you, to say? Well, you know, maybe it’s two to 53 of this implementation, this string, maybe it’s UTF eight. Maybe it’s ASCII it’s
I
asked you, you don’t
Thomas: Can’t give you, I can’t give you an argument for JSON certificates. I would never do that. Right. but for the
existence of Right. But like there are other types represented, like even in the 1990s, we had XDR, which was a much simpler form than, you know, X.509. If he’s like that,
Ryan: Yeah. Yeah. So, so the context here of, of that is, I don’t know if I can give you a full-throated defense on why didn’t we choose anything else I can give you the other explanation of how we end up with this mess in the first place, which was X 5 0 9, wasn’t meant to be a schema itself. It was actually just meant to be a part of an authentication protocol to "The Directory", and "The Directory" was this X 500 notion of the true source of all names on the, you know, that you could ever want to express. And we would have one ontology for naming things. And then "The Directory" you could authenticate using an X five or nine Certificate. And this was as an player, different password. So eczema was really just, it was a pluggable authentication model, that got completely bastardized and ripped out from "The Directory" and, you know, became what we know as certificates today because of work that was going on in the IATF on like how to encrypt email. Um, so. so Uh, "The Directory" used because it was designed a lot by telecom and what, they were optimizing for was streaming processing.
And that was their whole goal is we want to be able to take bits as they come in. We may not have all the bits we need, do something with it, you know, in the buffer, and then get those bits out of the buffer because we’ve done the thing we’ve needed. And so, uh, ASN one in particular, bear has this indefinite length and coding and all of these weird debts.
And they’re all designed around this, this notion of what telecom needed, which was to be able to do structure where processing of data, working on data flows without having to wait for entire messages to be delivered before they could act on it because they were constrained in memory or, you know, you don’t want to have to buffer everything cause that has lag and latency.
and so
that’s
how we’ve got this mess is, you know, even at the time, this was the best
solution
David: say, I think, uh, it’s important to like separate, like the X.509 schema’s from the ASN.1 schemas from the DER encoding, DER encoding is really not bad at all. It’s just like length type value, basically with a little bit of bit twittling.
Ryan: yeah.
It’s, it’s like proto buff, uh, you know, or thrift or, or any of your other, you know, TLV or PDU, depending on what you want. It’s you know, which is as scary as dealing with DNS on the wire, right? DNS is one of these wonderfully length prefects. And uh, with most like length prefixed protocols, if you can’t do math, then you can’t do math and bad things happen.
If you, you know, if the inner number is greater than the outer number, bad things happen and you run off the end of your buffer. Uh, so I mean, it’s terrible in that regard, but you’re absolutely right that, that, you know, the subset of ASM one that X, Y benign uses is, is somewhat sensible. And, you know, it’s, the issues there are, uh, how do I, you know, represent this concept? What are the different ways of expressing it and do pick one? And a lot of bad
David: common names that get what are those and why are they like untyped compared
Deirdre: And the subject alternative names go into an extension,
David: which are like X.509 generalized names. It can be anything, DNS names are IPS, but also just like an email address. Or I have a certificate somewhere that has a picture of Sleevi’s dog embedded in, uh, the subject alternate names.
I was going to try and see if I could get a, real, a certificate authority to issue.
Um
Deirdre: show notes.
David: I think I’ve since lost that certificate. I’m
pretty sure it was on my Google intern, computer. Which as we learned in the previous episode since been co-opted to be used as a foster fuzzer
Ryan: That’s right. ABIs checks, Uh, grabbing all the intern machines. Yes. so I think, you know, going back here, cause we’ve talked a little bit about Ru programs and since I mentioned "The Directory", this is like one of those ones, weird things that we’re stuck with similar to IPV four, similar to BGP, like oops, mistakes were made, but they’re impossible to change because they’re just so widespread.
You know, when, when all this technology was being developed, the idea is that there was going to be this "Directory" protocol, which would be allow you to bring different networks together and exchange information about the networks. And everyone can have your own little enclave and you are the king of your castle and, you know, you have control over your dominion.
And so this notion of CAS, the idea here was, you know, Thomas, if, if you want to say, what certificate is for your name, the idea is I would go to the directory and I would have some uniquely unambiguous way to identify, what it is. Like if I want to talk to Thomas’ machine Fu I would look it up in the directory.
When I went to talk to Fu I would ask you in, in the directory, Thomas, who is your CA of choice, or what CA do you use do issue certificates before you were things and you would be the sole arbitrary truth, where that broke down, right, is the directory is really an alternative name, these game to DNS.
You know, they, they are completely different naming schemes because DNS is, there is the root zone and there is calm and net. And then, you know, the need promise that there’s google.com or sleepy.net, there’s all these things. and so these concepts don’t really map anymore. And so this whole concept model of X 5 0 9 really fell apart and Netscape did their best to sort of put it back together by saying we’ll look, we’ll just have cacas. We’ll pick the CAS that work in our software. and these will be able to issue for any name and their, their choice of CAS was really what CA’s are out right. That users may want to have their certificates work with. you know, so the selection of CA was really oriented around who has certificates out there that we want to work in our product versus who’s certificates, you know, who do we want to trust to issue certificates on behalf of our users? And so there’s like this, this weird shift that happens. And, my IAT I’ve taught goes into this, you know, in the history, but this is where I think when you talk about CAs being evil or, or talk about this, I think really what we’re seeing is there’s been this, this shift of how we think about CAS, how we think about root programs from how this 1980s model was to where we are now, different groups have got it, I have understood this shift at different rates or they’ve understood parts of it. And they’re all trying to adjust. Everyone is trying to address the ecosystem to figure out like, how does this all play? What does this look like? And so. As with a lot of computer security where we are now in 2021 is vastly different than where we were in 2001.
You know, Microsoft got the SDL and open source turns out to have one, but in a very weird way than I think how anyone thought open source was going to win, you know, and there’s been a lot of change for how we think. And so now our program is more an extension of product security than what it was originally, which was we would have "The Directory" and everyone would have their own little enclave and authority zone, and everyone would be able to exchange, you know, this is whoever, whoever, whatever, CA’s David trust issue for, you know, computers in his network and whatever CAS, dangerous stress issue, you know, for computers and her network.
Now we have this notion of, well, these are CA’s trusted issue on the internet and, and that’s the shift. So I don’t, yeah, that’s, that’s where, like, of course I’m not going to say CA’s are evil, but I, I do think that there is Th there’s been this change and not every party in the ecosystem has really understood or fully grasp all the changes that are happening.
and this creates challenges. So like right now, the EU is proposing that the EU should be the source of truth for all trusted CAS for software anywhere. And so they they’ve got this law that they’ve got this uh, and they’re revising this law. And part of this is trying to say that, if you do TLS, you must accept these European CAS. And when I say these European CAS, it means any member state, or anyone that the EU terms with, treaty with, can unilaterally declare, this "CA should be trusted for the internet". And then, uh, anything doing TLS needs to trust that. And, they cannot reject certificates if they comply with the European standard.
So you can’t say this certificate is not trusted. You can’t remove trust in those CAS.
Thomas: This is like a way of saying Luxembourg could issue a Gmail certificate.
So Luxenberg can we all just Luxembourg should not be able to issue a Gmail certificate, right? Like we agree on that.
Ryan: Or if you look at the countries, if you look at the countries that you use looking at treaty with, right, they’re having discussions with the UAE. They’re having discussions with Thailand, with Vietnam, with Japan countries that, uh, South Korea, um, Brazil, countries that have various, uh, interpretations on, on sort of how to approach and cryptic communications.
And the idea is if you’re talking about identity, this makes sense. If you’re talking about PLS and domains, this is perhaps a bit more questionable, and this is something that is actively being debated right now. The revisions that introduced this requirement, uh, are being, uh, are being proposed as law.
And so they’re going through like the. European parliament and European council of member states, to be discussed and marked up, before becoming final law. So this, this will be, we live in interesting times, but a lot of that model is based on that early 1980s assumption of every country would be sovereign for their country.
And then every, you know, in the U S every state would be sovereign for their state, and then every state could license corporations. And so each corporation is authoritative for their corporation, and there’s like, RSCs on like us naming scheme for north America for how you can address the Tide House, which is a, you know, popular Silicon valley bar, or at least used to be, they shut down on like, here’s the one true name of the tied house and the directory.
And so if you want to get certificates for tide house, you know, this is
how you
look them up in the
Deirdre: as it, as a Irish citizen by birth. I am not in favor of this EU measure and I’m going to write to my representative. That’s how that works. Right.
David: What, what are you talking about? Our podcast CA could become a fine European CA, that’s that’s what I’m hearing. We’re premium!
Ryan: you need to go in corporate and Luxembourg. And, and so, you know, on that topic, sec, Tigo I mentioned, right, now has a, uh, European army in Spain. And They have now become, because I said to you, I should say, as mentioned is, uh, based in the UK, And so, uh, things like different now for the EU when the UK.
Thomas: I’m sitting here Googling whether the Vatican is an EU state right now
Deirdre: Oh, no, don’t think they are. I
Thomas: Um, well I’m Catholic. I’m going to go talk to my priest to see if I can get
David: can get holy certificates.
Deirdre: Go ask the cardinal.
Thomas: do it. You could do away with all of this nonsense, right? It just store things in the DNS, like, and just kind of a global hierarchical CA like the internet as a source of truth.
And no one will be able to charge for it. Cause I guess the DNS is free. Like that’s the DNSSEC kind, right? DNSSEC and DANE, which is like— DANE is like, the CA is the DNSSEC, you know, hierarchy or whatever, right? Like it may come as a surprise to some people here, but I’m not a fan of DNSSEC, but like
Deirdre: have they from RSA-512?
Thomas: they are, they’re on RSA-1024
Deirdre: Oh,
Ryan: nominally short-lived so, you
Thomas: I’ve been kind of living in a blissful state of thinking that, um, you know, Chrome at one point dabbled with, uh, with DNSSEC, and Mozilla dabbled with DANE at some point. And like there was DNSSEC code at Apple for while, and then all of that code went away. And so my kind of, my general line on this is, uh, oh, it was tried and it went away and it’s never going to happen.
You could reassure me right now that it’s never going to happen. Right. So I can just keep telling people, stick a fork in it. DNSSEC is dead?
Ryan: Boy, I wish I could. I, no, no, DNSSEC will not die, in it, in fact, like if you think about it, we got this whole mess with CAs and Netscape was very clear when they proposed this and this was, um, Kip Heckman was the main engineer on the SSL 2.0 spec. and so there’s, there’s an old cypherpunks mailing list that was, you know, very active at the time.
And Kip really was, was not afraid to get his hands dirty in this list and in discussing this, and it, was very clear that if we had DNSSEC, like Netscape would have used DNSSEC, but we didn’t have DNSSEC. So, they, they came up with this, this, you know, CA choice. Chrome certainly played with what was called DNSSEC stapling. And the idea here is you would have a shortlist certificate that would have a full DNSSEC, chain as an extension of the certificate. And if you have that, then you could talk for whatever name was sort of asserted for it. and this was, you know, Adam Langley’s, project, and, uh, you’ve ever seen Adam Langley.
You’re seeing this name one of the most impressive engineers that I’ve ever worked with and just incredibly, um, productive and proficient. So, um, he worked with us. There were I don’t think I would say that the challenges here were necessarily with DNSSEC, it’s just, uh, it, turns out short lived certificates are really hard to get, right.
As let’s encrypt has demonstrated, it’s taken quite a bit to actually get off the ground. I think Let’s Encrypt was six years in the making, uh, something to that effect, you know, from, from the crew responsible for it and so it was one of those, if it’s too hard to use today, why do we care care around this code?
I think as we look to look at say some of the balkanization going on, where we see some of these, uh, interesting legislative challenges coming on on, on saying like you must trust our CA for the internet. Uh, you know, DNSSEC has a lot of appeal because it is, based on the source of truth and DNS, right?
Like if you’re wanting to talk about sleevi.com, you you’re going to hit DNS. Anyone else claiming to be sleevi.com you don’t want to trust. And so it makes sense, to get the records, the big challenges with DNSSEC, beyond, as, as you know, Deirdre mentioned that there’s the crypto, which okay. Has been weak, but is getting better, has just been that consumer what’s known as CPE, right? Consumer premises, equipment, or otherwise known as home rootrs, is just absolutely terrible, and use really crappy DNS stacks that are, you know, decades old, because of course it’s IOT. You’re not a true IOT vendor if you’re not shipping something that’s already a decade out of date, you know, and, and you’re not providing updates for it. Um, they would do things like just crash, uh, on seeing a DNSSEC record Um, we had, uh, on the Chrome side again, like I keep saying we, but, uh, Chrome had this fun bug where, uh, the VP of Chrome’s home rootr would crash. If Chrome tried to make more than 60 DNS requests at the same time, like the whole network would crash.
And like, so of course we sent multiple senior Chrome engineers there right, we’re getting network traces and packet captures, trying to figure out why is that when Chrome sends seven requests, this thing flops over, but six requests is fine and it was some old AT&T equipment. And
Deirdre: it had like max buffer of DNS record like that and they was like, maxing it out or something like that.
Ryan: it had a buffer overflow in the DNS code that. they patched, it was DNS mask with, you know, a bug. And so, Yeah. Yeah.
so I mean, I I say this to, to not reassure I, think that things like, DANE and DNSSEC actually, are going to be putting important because, DNS went through a lot of this geopolitical upheaval with the ICANN transition when it transitioned from being under the commerce department to being under ICANN and IANA.
And there was just a tremendous amount of, political, hand-wringing around that from, you know, we don’t trust the U S government to operate this, but do we trust a California nonprofit, ICANN, to operate this. But it’s mostly been settled? So there is some appeal of like well, maybe DNS is the better place to put these keys for some of these problems.
Thomas: I’d say there, but like,
but
Ryan: yeah, I know
Thomas: two things I would like, I would zero in on. Right. And the first is like, it seems like from a governance perspective, DNSSEC has even more, or DNS in general is even more tied to, you know, local governments.
Right? Like, you know, Finland is the authority for, you know, what, what records are going to be under the, you know, the .fi TLD. Right. But the bigger question I have, and maybe like, you think that this is like a realistic thing that could happen, that I assume you have like a, must have an answer to this, right.
Is how do you deal with revocation? Right? Like if, you know, some random CA right now, issues a gmail.com certificate, my mental model is that you guys will, through a series of different mechanisms, probably detect that, and then probably nuke that CA from orbit, but you can’t do that with the DNS hierarchy, right. There’s only one of them.
Ryan: So, yeah, I’m going to give you things that are gonna keep you up at night, right? I’m going to give you the spooky scaries. So like the example of Finland, right? Controlling .Fi. Because of the control of .fi, they can get a cert for anything under .fi today.
Right? Nothing prevents that, they are the source of, truth for .fi. they can get a cert for any of that. so it doesn’t really change much from the status quo of what we have with, with, you know, CA’s today is, um, if a CA issued a cert to, and we’ll just, I don’t know who the .fi operators, but the Finnish government sees .fi as sovereign territory.
So whoever they picked to run fi let’s say, you know, they, they issue a cert, the CA didn’t do anything wrong for issuing that cert. And in fact, like there’s no reason to revoke that cert at least by all of the standards today
Thomas: imagine a scenario right? Like, um, you could have like a CAA records or something locking our particular, like major property down to, you know, a particular CA and that major property could have an arrangement with that CA saying, you know, we’re only going to issue certificates under this process.
Right. Like it’s not simply domain validation. We’re doing something else. Right?
Ryan: yeah. but .fi could change how that behaves, right? Like you said, all those on your name server, the .fi would just say, oh yeah, you know, this, this lovely domain you have here, you know, thomas.fi, we’re just going to send it to our name server that doesn’t have these CAA records and then everything’s groovy.
Thomas: Okay.
Ryan: So, Okay. So that was that like, .fi case, the, the revocation case I think is, is one because, um, you know, I, I recently was tweeting out about, uh, w what I called the fast fed and TLS security, which looked at, uh, this new protocol that, that combines open ID connect, and several other Federation protocols and tries to provide, you know, one protocol to rule them all, is, revocation, mostly doesn’t work, especially though in non-browser sort of scenarios. So it really depends on, on your threat model here. If there is a gmail.fi or, I think the interesting one to really think about is something like a GCP certificate. Cause that’s, that’s the one that would really give you the spooky scaries, is how often do you access Google Cloud, say in your browser versus say using cURL, or your server to server, you know, stuff? that revocation is largely not going to work. cURL is not going to know to nuke that CA from orbit and your, your Linux distro may not nuke the CA from orbit. historically like the Linux distros would ship a year or two year outdated copy of the Mozilla store.
So even if Mozilla moved, the Linux distro may not move.
Or, my favorite case, was like Linux distros that added back CA’s just unilaterally. So, um, there was a, this led to a fun discussion by Mozilla recently where Mozilla had, you know, the Symantec CA’s that were in their program and they, they eventually removed, they had code in their software in Mozilla NSS to not trust those certificates for an eco certificate issued after particular day.
But they finally got around to removing it from their root program and several Linux distros where re-added the Symantec certs back to their root program. And the reason was because Microsoft’s nuget, which is a .NET, like, package management library was using these certificates for code signing and was using the TLS root store for code signing.
And so then they decided, well, this is going to break our nuget packages on Ubuntu. So we’re going to add Symantec back to Ubuntu for all users. so, you know, in your, your scenario here of like a CA going rogue, like the revocation that we might believe we have, doesn’t really work as reliably as you’d want.
and in fact, if you did have DNSSEC or DANE, you would have a much more reliable signal for that, because you would look to see the fresh DNSSEC record. You still have TTLs to deal with, but that’s, a different problem. Right. And then a lot of resolvers will cap how long they’ll cache and when I say like how long we’re talking, like on the order of O{days}, not O{weeks or months}
as
Deirdre: I jump in here and ask about pinning for applications that pin certs and keys.
Ryan: Oh Yeah.
Thomas: 20 minutes dunking on everything I believe it.
Ryan: I mean, Yeah.
So, pinning is particularly terrible. there are ways to do it right, but generally the way to doing it wrong is to say, I’m going to pin, I’m going to use, say an OS root store. And then I’m going to restrict my CA’s , or I’m going to pin. I’m going to use a CA and I want that CA to work in a bunch of different software, and I’m going to pin my software to it. And so like, this is an example of say, you have a mobile app and you have api.mysite.com. And in your app, you pin that API end point to a particular CA but you have a bunch of other clients like set top boxes or browsers or TV. If I don’t know, that all want to access that site.
It’s a terrible idea because you’re trying to say that this CA will work for all of my use cases in perpetuity. And if you’re not the one who is actually making those decisions, the, the parallel that I draw is, um, it’s like, depending on an internal data structure of an opaque pointer, like, oh, yeah, offset 16 is always the reference count.
I can always just add, you know, take this void pointer, add 16 and then use that to mutate the refcount, like, no, it’s an implementation detail. It’s not part of an API. You’re not guaranteed any of that. Uh, and pinning is a lot like that where it’s it’s reaching it into an implementation detail.
In this case, the implementation detail is a root program. And you’re trying to say that this program will always include this particular CA and you’re not guaranteed that, and the root program in particular doesn’t guarantee that, you know, the root programs have never, and if anything, the root programs of the past few decades have been trying to shift their clients away to understand that look, any of these CA’s, you can get a certificate for, we will do our best to make sure that the certificate works for as long as it’s valid, but we can’t guarantee you that.
We can’t make any guarantees whatsoever even after that. so yeah, pinning, pinning is terrible.
Deirdre: like an abstraction violation. you shouldn’t even be thinking this hard about this certificate. You should just be like trusting your root CA’s to deliver you good certificates. And if they deliver you good certificates that validate to the root, then you’re good.
you shouldn’t try harder to unwrap this onion and pin a particular cert and be in charge of it and all that sort of stuff. Yeah.
Ryan: Or, you just pinned to a you control and tha,t, you know, you exclusively control and make the decision. So, you know, this is where for like mobile apps, if you really want that. And let’s be honest for most mobile apps, they’re not doing it for security, uh, from network, uh, adversaries. They’re doing it to protect the device owner from themselves, which I think is, is, um, a bit of a dubious claim, right?
Like, I can understand why Snapchat would do pinning. Yes, I get that threat model that Snapchat is, but I think it’s a little dubious to try to, you know, have an application, um, that prevents the device owner from inspect. And that’s a whole other conversation, but if you are going to do pinning, the idea here is you pin to a CA that you control, that you specifically control and you’re responsible as well for dealing with all the inter-op, um, for that.
So, you know, it’s not going to work in browsers because it’s a CA you control and that’s, that’s an acceptable trade-off. So maybe you have one API end point for browsers and one API for you’re pinned endpoints. With the Symantec deprecation, this was something that a number of large sites, uh, went through and you can sort of see publicly, like, um, if you look at Netflix now versus where Netflix was at, they had a lot of, They had one endpoint, their a web and TV and set top boxes, all used a common endpoint, a common set of API gateways.
And so when browser started removing trust in Symantec, they had a bunch of set top boxes that only trusted Symantec. And so the solution there is you need different hosts, right? Here’s the host for the website that give web friendly certificates for web browsers. Here is the set of a hosts for a set top boxes, the gifts set top boxes, here’s those for TV.
and in fact like, if you really take that idea to its logical conclusion, you start having a host per TLS set of capabilities. So, you know, this device only has support for TLS 1.2, and these CAs, like it gets its host name, and this other device supports TLS 1.3, give it a new host name, and they’re all to the same backend, but
their, their
Deirdre: at all these different ones that have different capabilities.
Ryan: Exactly exactly your configuration can match what your client capability. So when you think about pinning, pinning is, you know, the same as conceptually using a self-signed CA, and if you can accept that you’re willing to use a self-signed CA for that case, and you’re not going to get a much complaints then yeah.
Application pinning makes sense that it has to be a CA that, that you’re in control of, and you’re managing the trust for, and of course, how you manage that trust. there’s, there’s been some great research by, uh, another former Chrome intern and, uh, Sasha Fau, who’s looked at a lot of Android code and, terrible, pinning on Android.
Dan Boneh has, you know, this, this, uh, paper, the most dangerous code in the world, uh, which is another, you know, just great work, looking at how, how many ways you can screw up pinning and both iOS and Android have now offered API APIs that allow you wto do that a bit more safely, but you should only pin to CA’s you control.
David: But so overall to kind of cap off, do you think that things have gotten better over, the last. 10 20 years? Like, as, we talked a lot about the world changing and switching from the directory, or excuse me, "the directory" , to now. and there’s been a lot of improvements in the TLS space. I think
Ryan: Yep. Yeah. So, I mean, I think things have improved a lot. I th ink, you know what I think of like some of the major achievements, the fact. In the old days and by the old days, I mean, really anything before 2015, which is not that old, uh, you had just hundreds of organizations, each in independent control of CA is like an issue for the whole internet.
as browser started requiring that the baseline requirements and the audits and you, uh, actually checking the paperwork from CAS. A lot of that has gone away, you know, um, digicert was really famous when they. Acquired the old Verizon CAS, because Verizon had something like 379 organizations that all had used the internet, if you will, that were all unaudited and, and, uh, very dubious and quality, um, and did a certain managed to sunset all of those by generally moving those CAS onto infrastructure that DigiCert controlled and manage made sure was compliant and audited.
So, you know, we have seen a lot of really substantial changes. We’ve seen the end for some of the crappier crypto. Um, we’ve seen a lot of improvements around how the domain validation goes that I’d like to believe that that CAS CA’s are still a weak link. Right? We, we started off talking about audits and how David, you can become an auditor yourself, you know, just some, a little bit of CPE.
And, uh, I think it’s like somewhere between 10 to a hundred thousand for CPA, Canada and
boom, Bob’s your uncle. Yeah, then you can start auditing CAS and then Deidre can start her own CA and then the world becomes a much more interesting and exciting place. But, you know, we have done a lot of that.
I think where a lot of the work coming forward is going to be in improving these audits because there’s a lot of, a lot of interesting stuff going on in the cloud audit space that I think is applicable to CAS. Um, one of the, one of the things like for CA’s is, um, the current report is, is, uh, similar to what’s known as, you know, a SOC 3 type report where it’s just like a two pager.
Oh yeah. You know, David’s an honest person, Deidre’s an honest person. They, she did what she said, but there’s no. examination of how you actually reach that conclusion. Like what did you look at, David? What did you test? in cloud there’s, there’s generally more of a SOC two type audit, um, which. Detail, not only how Deidre’s infrastructure runs, but what you looked at, what you tested, how Deirdre claims she achieves particular goals, and then what you examined and whether or not Deirdre passed or failed those claims.
And she failed some of them, but, you know, overall, she still met the goal. It’s just not the, not as good as you you’d want to see, but you can’t argue that she’s not there. those are things that I think, uh, are, are very relevant to what we’ll see for, for TLS. there still, obviously big gaps, uh, BGP. Is still just a nightmare scenario for, for dealing with that.
And there’s not a lot of visibility into the space. you know, there, there are BGP monitors, but figuring out, how many certificates are affected. I’m not aware of folks actively monitoring that to say, like this BGP hijack, these domains were pointed to IPs in this network range. And now we’ve seen interesting certificates and certificate transparency.
The tools are there, the information is there, but I’m not sure that anyone’s looking at that. so that’s like an area and of course, DNS registrar security is, is still, uh, not ideal. for DNSSEC, that is not as important, right? The registry security matters. Uh, the registrar matters because sure.
You can point the name servers. As I was telling Thomas, you can link the name servers to anyone and defeat a lot of these controls. So like, registrar, security matters. and there hasn’t been as much improvement there. It’s if you look on say the Mozilla forums, you’ll see some .cc TLD registry is being hacked, you’ll see registrar and account takeovers.
I recently saw a write-up from Ian Caroll who used to poke around the TLS space and is now poking at other, other things was looking at, um, domains hosted by mark monitor where, you know, he was able to potentially take over. And he’s very careful in how he frames this, uh, because they were all posted at S unclaimed S3 buckets.
And so all he had to do was claim the S3
Deirdre: can host something
can be control this domain now. Like give me, oh my God.
Ryan: be the CA. Yep. Yeah. Yeah. So we, we have substantially improved. there there’s fruit there. I low-hanging fruit there. I think that’s still to be worked on for CA’s the current requirements are, A bit confusing, especially if you’re not like enmeshed in, in this. And so there’s still a lot of work to make the requirements clear.
And that’s something that I am excited for. Uh, that’s going on in the cab forum of actually specifying, like, this is what the certificate looks like. We’re not going to describe prosaically. We’re actually going to show you like bits and bytes and tables and make your certificate match this. But yeah, do think things, things have improved, but, I think if anything, it’s going to be more and more introspection on Why exactly do we trust the auditor?
Why do we trust the CA? and to be fair, auditors hate this just as much as CAs and the reason auditors hate this is they’re worried about the reputational harm of being found out that, they’re doing a bad job. so, there I, there are audit firms. Who’ve made very clear that if there are greater requirements on transparency, they will probably stop providing audits to CAS because they don’t want the reputational risk of the CA doing something bad or being challenged by the rest of the community who say like, hold on a second.
How did you reach this conclusion
based on this evidence?
David: So, things are getting better they’re getting better in x.509 and / CAs than they are in the DNSSEC world, which I think will make Thomas happy.
Thomas: I’m just sitting here moping.
Ryan: Yeah. RPKI is, is happening slowly. you know, uh, job Snyders or, you know, yoga deciders, I’m terrible at pronouncing names. Um, someone active in the RPKI space. Uh, it turns out that RPKI implementations are terrible in their own. Right. and RPKI like tries to give a lot of specificity, which is to their credit.
Like they cut off all the bells and whistles and knobs. It just turns out that implementations aren’t actually enforcing that. So it wouldn’t be hold your back to square one, but RPKI is happening and that will do a lot for some of these BGP hijacks and some of this nonsense. and hopefully ICANN, we’ll continue to improve registrar accreditation program, uh, and look at strengthening security there. Cause that remains a weak link.
Thomas: Do you have, do you have comparable influence over registrar on registry security then that you do over programs?
Ryan: Uh, you know, again, there’s the you guys question, which um, ICANN is, an interesting organization. I think there, there are a lot of, truly influential and respected people who do have, a lot of influence at ICANN, if you look at, um, the security, stability and advisory committee, the SSAC, you’ll find a lot of tech luminaries there.
Uh, Warren Kamari, you know, is on that group. You’ll find a number of other people. so I think there is some influence, but, I would say that there is perhaps not as— ICANN, of course has the influence. there. I probably shouldn’t get into a whole lot of politics on ICANN. Other than there are some fascinating reading, like on the politics of that ICANN, from some of the, the, uh, MIT press publications, like I can provide some links for, but Yeah. the, uh, the registrar security, I don’t think will improve as quickly as say the web PKI has improved unless ICANN really, really cracks down.
However, that’s also not going to help for ccTLDs. So I think if, if you’re a site, you know, certainly you should be incredibly wary of the use of CC TLDs because a CCTLD, these are treated as basically sovereign territory of whatever that country is. And so they don’t apply. They don’t have to follow anyone’s rules there.
They’re doing their thing on the internet it’s sovereign territory. and so you find a lot of the ccTLDs having very poor security standards, or just doing things that are say actively harmful to stability. So be careful about your .Or
David: Are you telling me domain I own secure.af might not actually be that secure?
Ryan: It might not be that secure.
David: it’s Afghanistan, political issues right now. Like that, and this is very low down on the issues Afghanistan right now, but we’ll see what happens to their DNS registrar.
Ryan: Yeah,
yeah, yeah. As you said, there’s a lot going on there. yeah.
So, sorry. I mean, I, realize we’ve ranted about a lot of things are, I shouldn’t say we’ve, I’ve ranted about a lot of things. I’m optimistic, I’m optimistic for the future though. And, and there are a lot of great people and, you know, we, we’ve lost a tremendous influence, Gerv Markham.
You know, who used to be at Mozilla. Was just a profound influence in, in truly improving things and shaping things up, as a very modern voice. And he passed away a few years ago. but I am encouraged to see that there are a lot more people getting involved. And, uh, this is where I get the plug, which is, you know, if any of this is exciting, you know, hanging out on Mozilla’s dev security policy list, which they run as the public list and have, have done since they introduced their root program to have these discussions, you know, the questions that like, what does it take for David to be an auditor? this was something that was discussed, You know, back in 2008, cause it used to be that WebTrust wasn’t the only audit standard is there were a lot of audit standards. And there were a number of folks who did not use any of WebTrust or ETSI who were incredibly competent and proficient at auditing and, times change, risk models change, and they became WebTrust accredited, because that’s what you do.
but there there’s a lot of, of welcome feedback. Andrew Ayre is another person who’s very prolific on that forum and truly kicking butt, at, taking a detailed look at sort of what the claim CA’s are making, whether it’s being their incident report or their sort of policy statements and saying, do you actually do this?
You say you do this, but do you, and there’s, you know, Huge value in that. And we’ve seen like, I guess where I was going with this is over the past few years, there, there has been an increase in participation, which is really, uh, encouraging. You’re going back to, you know, Deirdre’s bus question is that’s the best way to help me, you know, win my scratchers is, start hanging out on these forums and, and start getting involved because you can become the next me or better than me or something. I’m not sure what the pitch there is.
David: You can contribute to ZLint!
Ryan: Yeah. there’s a lot of opportunities, yeah. Well, this was, was fun
times
Deirdre: much, Ryan.
David: and just one last question to close out with how many airline miles does your dog have?
Ryan: Hers has expired. she used to pull like
60 to 80,000 miles a year with me, but, uh, you know, pandemics being pandemics that’s uh, no more, no more frequent pup status,
Deirdre: of
Ryan: right?
Deirdre: two legged people.
Ryan: Yeah. Jet blue used to have a program where, uh, your pet could earn miles on every
flight. 300 miles. And it was the jet paws program.
Deirdre: Ryan Sleevi, thank you so much
for
David: you.
Ryan: you.
Good luck editing
this I am sorry for the
length fun times.
![Campaign Security with [REDACTED]](/images/voting-booth.jpg)