Incidents like the Log4j incident and new governmental regulations have forced tech leaders to examine the security of their software supply chain. Understanding the complexities of this is challenging; how can CIOs determine their exposure and prioritize their vulnerabilities? In this conversation, Yadin sits down with Chip Childers, VP Security, Compliance, Open-Source & Privacy Engineering & Chief Open Source Officer at VMware and Jim Mercer, Research Vice President - DevOps & DevSecOps at IDC, to discuss the software supply chain and how CIOs should think about it, in depth.
Incidents like the Log4j incident and new governmental regulations have forced tech leaders to examine the security of their software supply chain. Understanding the complexities of this is challenging; how can CIOs determine their exposure and prioritize their vulnerabilities? In this conversation, Yadin sits down with Chip Childers, VP Security, Compliance, Open-Source & Privacy Engineering & Chief Open Source Officer at VMware and Jim Mercer, Research Vice President - DevOps & DevSecOps at IDC, to discuss the software supply chain and how CIOs should think about it, in depth. They look at how we became so reliant on the open-source community and the impact of generative AI.
“When you talk about the idea of having to have development resources to do patching, it's those transitive dependencies, honestly, that you may not be able to patch because you're relying on other people's work. That's why understanding this complexity really matters.” - Chip
“I don't think a lot of organizations realize how dependent they are on this open source community as we've started to kind of grow out, develop applications and rely so heavily on open source.”- Jim
---------
Timestamps:
(01:15) Why are we concerned about the software supply chain?
(05:25) Building complex systems on top of other complex systems
(08:15) Realizations from the Log4j incident
(11:22) Resulting shifts from new compliance and regulations
(16:21) Creative chaos in the software industry
(18:48) Reliance on the open-source community
(19:23) How can you identify where code is coming from?
(20:17) Prioritizing vulnerabilities
(23:08) The snowball effect in the supply chain
(25:00) How do you understand your exposure?
(33:15) The impact of generative AI
(37:27) Where should CIOs start heading into board level conversations?
--------
Chip Childers on LinkedIn
Jim Mercer on LinkedIn
Yadin Porter de León on Twitter
[Subscribe to the Podcast]
On Apple Podcast
For more podcasts, video and in-depth research go to https://www.vmware.com/cio
0:00:00.8 Jim Mercer: The challenge gets deeper than just knowing about the open source that I'm putting in my product first party. There's this long list of transitive dependencies.
0:00:12.5 Yadin Porter De León: Welcome to the CIO Exchange podcast, where we talk about what's working, what's not, and what's next. Yadin Porter de León. Securing the software supply chain is top of mind in the wake of incidents like log4j and incoming governmental regulations around the world that will require a better understanding of where code comes from and how it was created. Understanding your software supply chain and where you may be vulnerable is incredibly complicated.
0:00:36.7 Yadin Porter De León: In this conversation, I sit down with Chip Childers, VP security compliance, open source and privacy engineering, and chief open source officer at VMware, and Jim Mercer, research Vice President, DevOps and DevSecOps at IDC. In this episode, we dive into the creative chaos that has created our way of life and resulted in significant vulnerabilities and complexities throughout the software supply chain. Both Chip and Jim have spent years looking at the challenges and vulnerabilities and have recently been delving into the intricacies of the new regulation in the space. The first voice you'll hear is Jim's followed by Chip.
[music]
0:01:17.6 Yadin Porter De León: So Jim Chip, why are we concerned with securing the software supply chain? What's different now? I mean, there's always been security. Everyone's always talked about secure code. There's all these different things that have happened ever since the very beginning of the Microsoft secure coding framework came out years and years ago. I had a great conversation with one of the creators of that. What is really different when we're talking about now securing the software supply chain?
0:01:41.8 Jim Mercer: We kind of identify here at IDC anyways, doing our research here, we kind of identify the software supply chain is everything and kind of every one that touches in organizations code across the software development lifecycle. So, that's a pretty broad scope, right? And so in the past, a lot of this has, to me, largely been left up to developers to an extent, right? Around the software supply chain, right? A little bit of self security there. And it's traditionally been somewhat not well protected. I mean, I think there's a lot of angles to this. There's certainly the supply chain itself. And then I think there's the open source angle, which is a huge part of that supply chain, right? And obviously we've seen the growth of open source and the adoption of open source to kind of drive more awareness of the challenges of vulnerabilities in open source.
0:02:32.8 Jim Mercer: Along the lines of if you were like an ISV, you may have been using open source for some time, but as organizations kind of became more software companies, if you will, right? You know, I'm Home Depot, I'm whoever, right? I now am a producer of software more so than I was maybe 20 years ago, where I maybe relied upon my vendors. And they started to adopt open source as well. There's a lot of benefits to open source, around the ability to help me go faster, right? I don't want to have to reinvent the wheel, if somebody's already created that code. And why can't I just use that great code and save myself some time and I can focus on kind of my business logic that's important to me. But I think as those organizations started to go down that path, I think that's where they started to realize like, there's more to managing this whole thing than we thought.
0:03:24.7 Chip Childers: I think a lot of the buildup happened in a bunch of parallel streams as the technology industry evolved. But if we just kind of snapshot right now and say, why is the concept of software supply chain and then the measurement that the software supply chain is not as secure as it should be important? You know, fundamentally it revolves around a number of successful and damaging attacks that occurred as well as some very challenging to remediate vulnerabilities that had been identified in very commonly used libraries. If you look at any software, whether that's the software on my phone, whether that's software that's running in a data center, whether that's software that is just some website that I go hit to get the news, the amount of participants in the process of creating that application is multiple orders of magnitude larger than what a typical, let's say VP of engineering would tell you.
0:04:22.9 Chip Childers: A CIO thinks about how many people have I staffed to that app, right? I'm gonna build a marketing website. Great. Maybe we hire one to two developers that are gonna help build a marketing website. I'm not minimizing marketing websites, but just think of real simple few page site. It is likely that there are hundreds if not thousands of software engineers that have participated in the creation of the code that's required to render that page in your browser. So I think one of the bigger challenges for CIOs and in fact members of board of directors for public companies, for private companies, it's coming to a understanding of exactly how deep and exactly how complex the chain of individuals, the chain of projects, the chain of components that are being reused really is to produce what should seem like a simple thing. And then when you compare that with what should be a very complex bit of software, you start to see the depth and magnitude of the way we are very interconnected in this global supply chain.
0:05:27.7 Yadin Porter De León: And I think an interesting concept for me too is that how developers are building on top of other complex systems or creating new complex systems on top of other complex systems or more code on top of existing code, like you said, you don't wanna be reinventing the wheel. So that complexity built on top of other complexity is, I think, one of the things that you're saying too, is one of the driving forces and why there's some concern around we need to understand all the different layers, all the different components, everything that's going into this. And how do we know if it's really secure? I mean, just tackle maybe too a little bit about that idea of trust. How are you trusting the different code and where it's coming from that are component and all the different layers? And is that concern that's floating up now to the CIO and the board, like you mentioned, Chip?
0:06:05.5 Jim Mercer: Yeah. One of the interesting things there, and I think Chip, you were kind of alluding this to an extent, was you've got Fortune 500 companies that are reliant upon software that's being maintained by somebody who's volunteering their time. And it could be one or two maintainers.
0:06:20.0 Chip Childers: Yeah, it's good. It's good to note that, sometimes in the news of like, hey, look, it was only one or two people who are actually maintaining this thing.
0:06:27.0 Jim Mercer: And what's incenting them to fix it, right? So even look at log4j, which we all talked about a year or so ago, right? I mean, that had just a couple of maintainers and you saw that the whole world, if you will, was tipped upside down looking for log4j, trying to resolve this vulnerability. And I talked to customers, it was eight months after it had been announced and they'd say, we're still looking for log4j on all our code and trying to find all the places that it lives and how we can patch it and take care of it. So, I mean, it just tells you that there's this kind of, I don't know if a house of cards is too aggressive, but there's this dependence upon the community that I don't think a lot of organizations realize how dependent they are on this open source community as we've started to kind of grow out and develop applications and rely so heavily on open source.
0:07:22.9 Jim Mercer: And the interesting thing about log4j is a lot of the open source we talk about today is open source that I maybe recently took into my application and maybe I'm tracking that in some way using a software composition analysis or something along those lines. But log4j has been around for like years. It's like in almost every Java program ever written, right? It's just something that was out there and you just always thought it worked like turn on the water in your faucet, right? You just expect it to work. So you have all these older applications that aren't getting regular development attention and now you're gonna go back and look through and say, oh, where are we using log4j? We might have picked this up years ago.
0:08:03.3 Yadin Porter De León: And no one's been tracking all that pull in and all the different pieces too. And I think Chip, you're gonna jump in with something as well.
0:08:09.2 Chip Childers: Yeah, I was actually gonna say one of the other reasons that topic is germane, frankly, I would argue is due to leadership. So the log4j incident actually was one of a set of those vulnerabilities that I talked about earlier that has triggered a realization across industry. And when I say across industry, I actually should use a different term there, right? It's triggered a realization across the global technology ecosystem. If you just scope the conversation to the United States, that particular incident is what caused the National Security Council to begin to work the powers of the executive branch to drive improvements in at least the federal government's software supply chain. I actually just came from a conversation down in DC yesterday where we got a number of people together.
0:08:58.6 Chip Childers: There was those of us that represented commercial software vendors. There were people there from various open source foundations, kind of the stewards of some of these projects as well as members of the US government, predominantly on the civilian technology side. But a few folks also from the defense side. And the challenge of thinking about these complex dependencies and they're both open source and commercial dependencies, right? Because open source is just a particular licensing method when it comes to sharing information. And because it's generally without dollars, the volume of it is really high. And the likelihood that developers are gonna pull an open source components is extraordinarily high as in almost guaranteed, but we still have these commercial dependencies as well.
0:09:37.4 Chip Childers: Every commercial vendor relates to, or pulls in commercial code from other commercial vendors, or we might rely on their software in sort of the proverbial stack. So kind of going down in layers of infrastructure or going up in layers of infrastructure. And so you have this multidimensional dependency tree that requires thinking about all these crazy dynamics. So one of the things that we've been reasoning about is a trade. Those of us that deal with software, we've been reasoning about this idea of the power of sharing software is incredibly important. In fact, we would not be recording this podcast today 'cause we wouldn't have the podcast recording platform, nor would we have the web browser, nor would we have the operating system that we're using to build this, you know, those things that we...
0:10:22.8 Yadin Porter De León: Yeah, it's good to recognize that too. The things we take for granted, they just like, we turn, like Jim, like you said, you turn the water on and it just comes out.
0:10:29.3 Chip Childers: That's right. And so these things are built layer upon layer. Jim talked about this, right? We're building up layers of complexity, but that's since time immemorial, we abstract complexity, we make it easier, and that opens up a new wave of innovation, a new wave of capabilities. So this core concept of sharing is really important. We've just reached a bit of an existential moment for the trade or for the craft of software where we've realized that collectively, we all have work to do to make sure that we're baking in the concept of security, but also understanding how much trust we can place in each dependency link that we have in our chains.
0:11:10.4 Yadin Porter De León: Yeah. And what's, like you said too, both of you talked about some of the vulnerabilities have bubbled these up to the surface. And then also what has then come out of that sort of bubbling up the surface is... And this is where you get into the CIO and the board conversation is, well now we have new compliance and regulations that is driving some of the new processes, some of the new workflows that need to come into place in order to be able to have that regulatory compliance. Can you guys talk a little bit about how that's shifting the way that investments are being made, the way that processes are being put together, the way that software is actually being done?
0:11:40.0 Chip Childers: Well, I think you need to broaden it beyond just software supply chain. It's just, this is an area where our understanding of the threat profile is emerging and growing. But I think a lot of the realization at sort of the board level or in the c-suites, it's coming down to the fact that this is one of multiple cybersecurity related threats. The regulatory environment is increasing the chances of liability if appropriate care and diligence isn't applied to your enterprise and what you provide to your customers. And so we're just in an environment where expectations from all corners, our customers, our regulators, the policymakers, everyone is expecting a level of diligence to just increase around cybersecurity. So software supply chain is one of those areas that we must continue to improve upon.
0:12:33.3 Jim Mercer: It's Interesting, just the other day, or at least earlier this month, CISA put out a open source software security roadmap that actually talked about the importance of open source to the government and to the private sector as well, but also implying that CISA was actually gonna get evolved to help the open source community in making open source more secure. So I think that just kinda speaks to the importance of open source software. I think, as you talked about compliance, there's been this trickle effect that I think it all kind of, to me it started with SolarWinds and out of the executive order, right? And there's been this rollercoaster of things, but out the executive order came out from this SSDF or secure software development framework that the government is looking for their own agencies to kind of enforce, and the suppliers that they work with to attest to the fact that they're actually delivering or building their software in a secure manner.
0:13:34.0 Jim Mercer: So it's interesting this kind of awareness of the overall supply chain and some of the challenges there. It's a good point. I mean, this goes... I mean, we hear a lot about open source and there's a lot certainly going on in that area, but I mean, this covers a lot of different aspects of the software supply chain, right? Down to how are we securing the tools in our supply chain, right? Or how are we... What are our build standards? How are we building, how are we testing, how are we deploying? You know, are we looking at our infrastructure? You know, how are we patching? I mean, there's all kinds of aspects to this. There's a lot of tentacles to kind of go into the supply chain. And I think trying to wrap your arms around that can be challenging 'cause there's a lot of different pieces there.
0:14:19.5 Jim Mercer: But I think it's important to, with the supply chain becoming a target for bad actors, I think it's important to think about how we do that. You know, I think back to... I worked kind of in the development area and oftentimes what what would happen is the developers would be the administrator of their own tools, right? Because hey, the development tools, security really didn't have good visibility in there. Development probably wasn't really opening it up anyways. But somebody would be the administrator who's also a developer of certain tools, somebody would hit some sort of a security problem and they'd go to that person and say, hey, can you give me authority to do something? And that person might say, well, I'll just copy your account to look like mine, right?
0:15:02.2 Chip Childers: [laughter] There's some bad habits there. Yeah.
0:15:06.0 Jim Mercer: That's right. That's right. What the heck? Let's all have it. Right? So, you've got that lack of security due diligence that's traditionally, I think, been around some of these pipelines. That can be a challenge.
0:15:17.8 Chip Childers: Yeah, it can absolutely be a challenge. The funny thing is that announcement from CISA about the US federal government's roadmap around both supply chain security, but also open source engagement, two days ago is when it came out from the time we're recording this. And as I said, I actually just got back from DC, I was in a room and one of the authors happened to be talking about how it was eminent and it hit people's inboxes. And he swore that that was not actually timed to have occurred to when he was speaking. But I will say it's, if I put on my c-suite hat for a second and say, okay, what are the things that I need to know about how these government activities, whether it's in the United States or whether frankly it's in the European Union or whether it's in Australia or whether it's in the United Kingdom?
0:16:03.3 Chip Childers: Nation, states, and the governments, whether they are legislatively or whether they are through executive action, they're starting to demand more from the software ecosystem. That generally is a good thing. I'm not gonna necessarily get into parsing what I like and what I don't like about some of the approaches, but the well-balanced approaches that know how to respect, the interesting duality of our modern technology ecosystem would not exist without a creative chaos of everything from individual developers sharing code 'cause they just felt like sharing an idea, all the way through to large scale trade, group style, open source efforts. There's a creative chaos there that's actually...
0:16:44.1 Yadin Porter De León: I love that. I love that concept. The creative chaos. It's fabulous. A lot of... So much great things come outta that creative chaos. But also there's some of the complexities that you're talking about.
0:16:52.5 Chip Childers: That's right. That's right. And so the way that this is best navigated is by learning to balance the freedom that a chaotic environment needs in order for us to harvest useful innovation from it, at the same time then providing the appropriate frameworks, the appropriate tooling and the appropriate support to ensure that we can think about measuring the trustworthiness of the output from that creative churn. We can think about assuring that that trustworthiness level is either measurable or sustainable. And it just helps kind of lift the entire ecosystem up as we balance those two things correctly. It's gonna be hard to do. And it's also gonna be an evolution. One of the areas that you can almost think of as a public good would be what are called public package repositories, right?
0:17:40.0 Chip Childers: If you've got Java developers, I guarantee your Java developers are relying on Maven Central, that's provided by a single company to the entire global ecosystem. If you've got Node.js developers, they're relying almost virtually guaranteed on the node package manager, NPM. Again, managed by a single company. If you go over to the Python ecosystem, really important for the machine learning surge in interest and adoption. Python is like the preeminent language used in that part of the engineering space, the Python package index. Every single person that's talking about their great new machine learning project or great new generative AI project, relies on the Python package index. And that's run by a not-for-profit foundation. That needs resources, it needs time, it needs help, it needs money to support this thing that's almost a public good.
0:18:33.5 Yadin Porter De León: It's almost like the open source is almost like one of the commons now, and everyone's leveraging the commons. And if something bad happens to the commons, it affects everyone in some of the negative ways you've already mentioned.
0:18:41.0 Chip Childers: Yeah, that's exactly right. And yet you can create none of the software without it. That is the hard reality right now.
0:18:48.4 Yadin Porter De León: That brings up that complexity, that reliance on that open source community and those who are maintaining certain open source repositories. It brings up that, well, as you said, all the different teams are relying all these different pieces too. How hard is that problem now when you have, when you're a technology leader and you now have to basically create like a nutrition label on the back of your software where it says, hey, look it, I've got 10 grams of log4j or I have five grams of this type of thing. When you have to be able to then tag and say, these are all the different locations where the code is coming from. So I can be able to say, look it, I know where this code is generated from. I know where the vulnerabilities can potentially be, or at least I can know when I have to address something. How daunting is that challenge now to be able to identify where all the code is coming from?
0:19:29.1 Jim Mercer: I mean, there's tools such as software composition analysis and creating what's called a software bill of materials. Right? And the challenge gets deeper than just knowing about the open source that I'm putting in my product first party, but there's this long list of transit of dependencies. Because if I'm using open source or I'm building open source, I'm an open source developer. So what do I use? I use open source. If I have open source component A, it may include B, C, D, E, F, what have you. And if there's a component along that chain that has a vulnerability, theoretically, then I am picking up that vulnerability in that code. So building this software bill of materials is helpful. I can see what's going on. I can even identify where I might have vulnerabilities, but it gets harder than that because there's vulnerabilities that I may have that they're not code reachable or I don't exercise that component in that way that I might be exposed to that.
0:20:32.7 Jim Mercer: And then there's the whole aspect of, okay, I have all these vulnerabilities that I've been able to identify. Now what do I do? Which ones do I go after first? Which ones are the most important? You know, given my application, given the likelihood that I might be exposed in some way, right? And understanding how to prioritize those vulnerabilities can be very challenging. I mean, all vulnerabilities that are recorded come with a score, CVSS score, which is a good... Which is a good starting point for a vulnerability and in understanding of what that prioritization is. But if I have something with a CVSS score that's maybe not a top highest score, may still be important to me because it's in my customer facing application where I take all my orders or it's internet facing, right? It's more exposed.
0:21:25.5 Yadin Porter De León: There's multiple dimensions to that prioritization.
0:21:28.4 Jim Mercer: Right. You're gonna think what is the tier of the application, right? What's the likelihood it's gonna be exposed? And the challenge is that it does take teams time to patch these vulnerabilities, right? Sometimes there's mitigating controls where they can go ahead and attack in that way, but it does take teams time to try and address those vulnerabilities. Matter of fact, some people have said it's somewhere in their 90 days or more to patch vulnerability that's been identified. That's a long runway, right? And then I have these vulnerabilities. I've talked to organizations that say, hey, I have a backlog of hundreds or thousands of vulnerabilities that I need to patch. I don't have the development resources to go through and patch all those vulnerabilities. So I need to figure out which ones I really need to patch, right? Which ones, if I couldn't do anything, what's the most important then kind of next after that, right?
0:22:17.1 Yadin Porter De León: Yeah. That's not too different than when we're just trying to figure out how to fix things around our house. Like what's, when I do so the roof doesn't fall off? Let's start there. Let's make sure that water actually runs into the house. And those are the kind of things, 'cause like you said, there are hundreds, like you said, sometimes, and there's not the resources and the time for all of them. So that prioritization that's... Yeah.
0:22:35.4 Jim Mercer: Around the house, I can just ask my wife 'cause she's always right.
0:22:38.4 Yadin Porter De León: There you go. See.
0:22:41.5 Chip Childers: Well, so I'm gonna build off this fixing something in the house thing, right? So the reality is that when I do a project, it's a huge win if I didn't have to go to the hardware store at least four times.
0:22:49.4 Yadin Porter De León: That's fabulous.
0:22:51.7 Chip Childers: Why am I saying that? Right? The relationship though is Jim, when you talk about the idea of having to have development resources to do patching, it's those transitive dependencies, honestly, that you may not be able to patch it because you're relying on other people's work. And that's why understanding this complexity really matters. Let me kinda give an example that perhaps I overuse, but I think it can help a manager with a non-development background understand how quickly this can snowball. So you want to do, I'll go back to this marketing site idea, right?
0:23:19.0 Chip Childers: You wanna build a very simple marketing site. You hire a node.js developer, just one should be just quick little project for them. Node.js developer says, great, I'm gonna use the React framework. Super popular. You can do stuff for mobile apps, you can do stuff for websites. And they go to the documentation and grab one of the example React apps. It's about 500 lines long, sort of replicates the calculator that you have on an iPhone. Very simple, right? Like 500 lines of code, no big deal. They get going, they start modifying it to make it do what you wanted to do. I don't know, maybe it's a mortgage calculator or who knows? Well, that 500 line project, when you actually install it and have it start up, it pulls in over 1400 distinct packages.
0:24:00.6 Yadin Porter De León: See, that's critical.
0:24:05.4 Chip Childers: That's how quick that happens. And this simple bootstrap example application that really doesn't do all that much is a really good lens through which you can look at how complex that dependency tree is. Because those direct dependencies are nowhere near as numerous. It's the transitive or the next layer down. And in fact, the graph of those dependencies, right? So the same dependency can show up multiple times and kind of the tree, it's over 3000 nodes in the graph, right? So 1400 total dependencies, multiple instances of a bunch of them. And if you start with that sample, this is where we come back to the idea of vulnerabilities and fixing them. A number of those dependencies that are by default pulled into this bootstrap software, bootstrap project have critical or high vulnerabilities in them. Now, when you start digging into it, this is where the challenge really comes in.
0:24:55.9 Yadin Porter De León: This is exponential. This is, I mean, how do you... You're on a thread though, Chip, but how do you just begin to wrap your arms around what your exposure is?
0:25:03.6 Chip Childers: Well, so here's the trick. You're not actually exposed to all those vulnerabilities. And so this is one of the big problems that we need to, as a industry, whether you're an open source developer, whether you're a commercial entity, whether you're the government, we need to figure out how to communicate the impact along with severity, right? So for example, if you look at those vulnerabilities as they're used, as those packages are used in this example application, those vulnerabilities are not, those code paths are not exercised, which means that the app itself that you hired that contractor to build isn't actually vulnerable. So there's this idea that builds off of a software bill of materials, right? The bill of materials would be the set of ingredients, the dependencies and transitive dependencies called vulnerability exchange. And this is gonna be really, really important 'cause it's kind of the next step, right?
0:25:58.0 Chip Childers: So if I ship, Jim, if I ship you my bill of materials, you're gonna look at it, it doesn't matter what the software is, you're gonna look at it and say, there are vulnerabilities in here. And immediately reply to the email and say, when are these gonna be fixed? The way for me to help express whether there's impact and what that impact is, or whether we're still triaging it, is through a document called a VEX document, right? And there's different formats that are being developed around this. They're gonna be really critical because the depth [0:26:25.8] ____ that we talked about and the exponential potential there, it does matter, right? Because you do wanna see these vulnerability counts stay down, but what matters more is understanding whether they do, whether they represent an exercisable vulnerability.
0:26:41.2 Yadin Porter De León: No, that...
0:26:42.8 Chip Childers: Does that make sense?
0:26:44.0 Yadin Porter De León: No. That's critical. I feel like that actually understanding the reason for why you want to do something is because it's a priority. And how do you prioritize that is you have to actually know what's the impact gonna be. Great, there's a vulnerability, but it may be inconsequential or it may be huge, or it may be consequential on something that's not very important application or may be incredibly consequential on something that's a very, very important application. And there's that due diligence that has to be done. It sounds like there's a lot of process, a lot of rigor that has to go into that. Is this a new motion or has this been something that's already been a muscle in most organizations or some organizations finding they're having to develop this muscle for the first time?
0:27:21.5 Chip Childers: What do you hear, Jim? I mean, you talk to so many different vendors and customers. Is this something that just reasoning about vulnerabilities like this, is this something that organizations are doing?
0:27:29.3 Jim Mercer: It's something that we started hearing about back last year. And what we're seeing, well, when I talk to end users, most people don't even know what it is, right? I mean, it's relatively new. But when I speak to vendors, there's a number of vendors in this space, in the SBOM space or SBOM management or SCA space, Software Composition Analysis, right? They're starting to create capabilities to build a VEX document to ride along with your SBOM to be able to do that.
0:27:58.9 Yadin Porter De León: And VEX, that's an acronym. What does the VEX stand for?
0:28:04.2 Jim Mercer: The Vulnerability Exploitability eXchange. Did I get that right, Chip?
0:28:06.1 Chip Childers: Yeah.
0:28:06.7 Jim Mercer: Well, I knew it was something like that. But anyways, yeah. So basically to the point Chip made earlier is there's almost no chance that I'm gonna give you an application or a component or a service if we're partners, that if I give you an SBOM that you won't find some kind of vulnerability in there, somewhere along those long list of transitive dependencies, right? And then it's, rather than you calling me or saying, no, I need clean software, this gives me a way to be able to identify vulnerabilities that come up in my SBOM and why they're not applicable. And I think there's three categories, if I remember correctly, to define what it is, right? Maybe it's not code reachable, there's a couple of others. But basically give me the avenue to define that. Now, to me, I think it makes good sense and we're seeing a lot of activity in the community. Certainly vendors are starting to come alongside this, right?
0:29:04.4 Jim Mercer: But it can be challenging sometimes to actually produced that VEX document. In other words, there's not many vendors, or I can't think of a vendor off the top of my head who can actually reliably produce me a VEX document. There's vendors out there who will try and determine whether something might be code reachable and maybe give you some evidence around that and can help you do that. But a lot of times to me, this comes down to a developer's time to actually validate that. We don't call it in this way, we don't use that component in that way. Right? Once it's done, theoretically it's done. Right? Now I understand how we use that component. I've got good documentation, but there's still a fair amount of work to produce that. But it does help with this exchange of software and we're seeing more requirements around providing some sort of software bill of materials around my software, whether we're partners or I have some sort of a relationship with you, or I'm using some level of open source.
0:30:04.9 Jim Mercer: There's this increasing need to share SBOMs. Matter of fact, I think it's starting to show up in some of the compliance regulations around things like PCI, I think FDA, I think there's... I heard it's in NERC, things like that where there's a requirement where I need to provide you with a software bill of materials is part of my compliance with that regulation. So as doing that, vulnerabilities are gonna come up. There needs to be a way to kind of explain whether this is something that needs to be fixed or it's something that we don't use that component in this way, or we've mitigated in some way, right? Sometimes there's mitigating controls I can do, even with log4j, right? There was some mitigating controls that I can easily kind of turn that off if I understood I had log4j in that particular application. So sometimes there's a way we can do that through configuration to make sure that we don't exploit it.
0:31:03.0 Chip Childers: Yeah. So one of the things that I think is really important to reiterate is that this is a global community journey. The whole ecosystem's going on a journey of maturation here. So vendors of related tooling, those of us that offer commercial software, open source communities, and most importantly the consuming organizations, all of those participants in the ecosystem are maturing the processes that they use. They're maturing the idea of standardization around some of these practices. But let's just take vulnerability exchange as an example. If we rewind 10 years, every single commercial software vendor was essentially doing this analysis. It was a bit bespoke. It might have been based on a request, not your own vulnerability management process, but the communication to the end customers about whether or not something was something that somebody had to worry about. It may have been a little bit less than formal, but this happened all the time, right? Vendors would look at something and say, yep, we're using this library. This library has some vulnerabilities.
0:32:02.0 Chip Childers: A customer found it via static composition analysis of the software, I shipped them. But we're ready to respond with, don't worry about it. That's actually not exposed. The vulnerability is not possible to exploit or yep, actually, here's the patch version that you need to pick up and upgrade your environment with. So these practices have been there, but the reason why some of these emergent, both standards and VEX is actually more of a codification of a concept. There's a couple of specific file formats that are different standards for how to express this. But as we mature, what we're trying to do is actually tackle collectively that complexity, whereas previously we haven't really solved the problem in as scalable a way and as automatable a way and as much of a global way as we really need to. So that's where we're headed as an industry. Tons more work to do without a doubt. But it's a super critical thing for every enterprise, for every government, for every consumer even.
0:33:05.2 Yadin Porter De León: Yeah. And in that complexity too, I wanted to touch on really quick 'cause it would be tough for us to have this conversation without talking about generative AI, and the code that's being generated there, and how that's affecting this complexity and the process and the maturing of this process that you're talking about.
0:33:21.6 Chip Childers: Alright, I'm gonna throw out one of my favorite examples that doesn't have to do with generative AI but generative AI makes it worse. So, here's the example. It's called being stack overflowed. So I love Stack Overflow. It's a wonderful resource. Developers around the world use it all the time to answer each other's questions. People go to it, right? It's sort of like you don't pull up out a book anymore to figure out how to go solve a software problem, right? A coding problem. Often you'll just search Google, right? And most of the time it's like a stack overflow result or whatever. The reality though is that there are code samples that keep getting copied and pasted into software that are fundamentally vulnerable bits of code, right? So we have this propagation of poor software just because of the nature of humans, right?
0:34:06.3 Chip Childers: Like, humans are lazy. We're gonna go try to find an easy answer and copy it in. Okay? So now you imagine that's like a one at a time, each individual developer's kind of deciding and somehow group think and thumbs up on the answer, on an answer site. These things start to get copied over. We have bad code that makes its way into all kinds of products, all kinds of open source projects. So generative AI, doesn't matter which example we're using. The notion of generative AI is just simply the fundamentals of machine learning with some additional research that's evolved the machine learning technologies in a way that allows it to actually create new things. But you're still training on a corpus of data, right? So you're still training on some set of information, or whether it's an image-based generative AI, it would be a bunch of images you train on.
0:34:53.4 Chip Childers: If it's a large language model with a bunch of the things that are open AI and other companies, it's trained on large texts of data. So if a large language model is answering questions from a developer about coding, the question is, what's the quality of the training data? Because when it starts to construct something, is it constructing it using best practices? Is it constructing it based on poor input? Even worse, there's some really interesting research going on right now to think about how a person who has their own website or just simply uses GitHub can begin to poison the training set for these models in such a way as they can actually force vulnerable answers. So this is an emergent area of risk, but also lots of opportunity in the technology itself.
0:35:43.6 Yadin Porter De León: That's fascinating. Just it's hacking the open source dataset or the large learning model dataset to be able to create vulnerabilities or propagate vulnerabilities for those who are using things like that.
0:35:54.3 Chip Childers: But that's the thing, it's not even hacking, it's just simply publishing, right? Just publishing bad code, right? I mean, even with a read me maybe at the top of the top of the project that says this is stuff not to do.
0:36:05.0 Yadin Porter De León: Yeah. Some of the most effective hacks are the simplest, silliest things. They're not really like someone like, they always like... Everyone talks about a person in the dark room and they type away and they're like, I'm in like. No, mostly it's just getting someone to click on something in an email. That's the... That's social engineering. It's publishing something on GitHub now. It's the simple things.
0:36:22.1 Jim Mercer: And the stack overflow example that Chip gave us is a good one. That propagates it all over, it blows up the blast radius of whatever that code is and whatever the vulnerability might have been. And I totally agree. I think the learning models of gen AI are so important. There needs to be some level of, if you will, explainability to the results I get. Right? Where did this result come from? Can you tell me something about the back of that? The other thing about Gen AI, we always hear about hallucinations.
0:36:51.2 Yadin Porter De León: Exactly.
0:36:52.1 Jim Mercer: It's always gonna give you an answer, right? It's kinda like my father, when we used to get lost in the family car before Google Maps, right? [0:36:56.0] ____ I'm dating myself, right? He wouldn't stop and ask for directions, right? 'Cause he just knew he knew how, right?
0:37:03.9 Yadin Porter De León: He's gonna go for it.
0:37:05.0 Jim Mercer: No matter...
0:37:05.9 Yadin Porter De León: I like Marc Andreessen likes to describe it as that Gen AI is a puppy, it just wants to make you happy. It's like, hey, go do something. Yeah, I'll do it. No, I just wanna make you happy. I'm just gonna do this. So what would be... What's like... So, we've covered a lot of topics here. What would be the call to action? So technology leaders who've gotta wrap their heads around all this stuff. They have to have like, let's say a board level conversation or e-staff conversation. What's that call to action? What should they start with first? What should they do next?
0:37:31.4 Jim Mercer: To me, part of it is education and awareness and start to take some action on your own supply chains. Understand what your supply chains are, right? Understand the risk, it's all about business risk at the end of the day, right? Understand the security of the supply chain and kind of fortifying that supply chain, they really should become part of your corporate GRC blueprint for managing your IT and software risks across the business. And then maybe think about how are you gonna handle these types of issues when they come up? Things like you might typically do around any risk, right? An incident response plan. How do you handle that? Right?
0:38:06.0 Jim Mercer: How do you understand what it is you you need to do? Making sure you're doing your due diligence. I always say a lot of times, certainly vendors, right? They'll try and throw a new tool at it. And there's place for innovation. There's a bunch of new vendors and capabilities that have come out all around software supply chain security, and there's some really interesting and great stuff there. But just look at what you have, look at what you're doing. Look at your own pipelines. Make your teams accountable for doing basic things. Least privilege, you know, our back, role-based access control, right?
0:38:41.7 Yadin Porter De León: Basic blocking, tackling.
0:38:43.1 Jim Mercer: Yeah, blocking tackle, right? Taking secrets out of the code. I mean, there's so many, so many things that you can do that are, if you will... They cost time, they cost some level of diligence. But if you go after them and do them, you're gonna help secure your supply chain exponentially. Even what we've been talking around open source, right? Understand what your team's doing about open source. Maybe have an open source program office, right? Where you can... A lot of organizations do that, right? To talk about how do we want, how do we want to interact with open source as an enterprise. I mean, I was talking earlier about fortune 500 dependent upon some guy in his basement with a laptop, right?
0:39:20.8 Yadin Porter De León: Yeah. That's where we're at right now.
0:39:22.9 Jim Mercer: Exactly. Right. So, maybe if that project is so important to your organization, maybe you need to take a real thought about that and try and mitigate some of that risk and say, you know what? We're gonna get involved with that project. We're gonna... Maybe we'll pay them as a maintainer. Maybe we'll offer to contribute. Or maybe we'll be a part of that, right? Think about all the kind of dimensions of your software supply chains and get an understanding of what's going on. And then think about how am I going to kind of attack making sure I secure those without just saying I'm just gonna spend money. But look at the due diligence you can do and obviously look at where my greatest risk is by the application itself, right? If it's just some sort of departmental app, right? Obviously we wanna secure everything, but we wanna start with that customer facing app that's internet facing or...
0:40:07.6 Yadin Porter De León: Yeah, the mission critical items.
0:40:09.7 Jim Mercer: The mission critical items. Exactly.
0:40:13.4 Yadin Porter De León: Yeah. And Chip, I think we'll end on you too. What are those board level questions that should be asking, or the board should be asking of executives or execs should be asking themselves?
0:40:20.7 Chip Childers: Yeah, I mean, I think everything starts with ensuring that you can answer the question accurately. What components are in the software that you rely on? And if you can answer that question accurately, then you can begin to do appropriate risk management. And that risk management might be just simply focusing on the vulnerability aspects. But what I would actually suggest, and this sort of gets to some of the risk that Jim was pointing out, there is a notion of assessment for these external components that moves beyond vulnerabilities to things like how many contributors are contributing to a project? What is the release frequency or has it gone dormant? So each one of these, there's a number of different attributes that can help you get a good picture.
0:41:04.9 Chip Childers: One thing that I would suggest, and maybe this is below the boardroom, but if I'm in the boardroom, I would say, do you know what's in your software? And do you have some way to measure the amount of trust you place in these ingredients or the amount of risk management you do around these ingredients. And then the implementation of that. I would point to things like the open SSF scorecard. It's a great automated tool. It can measure a project against all of these slightly intangible aspects. Looking at things like code freshness and frequency of commits and number of contributors. And that will begin to give you a quantitative way that you can handle that enormous list of components that you're going to actually get when you say what is included in all the software I rely on.
0:41:49.7 Yadin Porter De León: No, that makes sense. Chip, Jim, I know we could continue this conversation, go deep into so many of the pieces that you've covered, but we're Kind of at the end here. So I wanted to give you each a chance to talk about where can people learn more about what you're doing, what you're talking about, where can people find you online?
0:42:05.2 Jim Mercer: Sure. So the majority of our research is at idc.com. We've actually published a number of documents specifically around the software supply chain as of late. And we did a bit of research on software bill of materials. I did that with one of my peers, Katie Norton. I wrote a piece back in March that kind of tries to identify, what are all the kind of aspects of the software supply chain. Going back to open source is a big part of it, but there's a lot of other components that you need to be concerned about as well. Your infrastructure, how your builds are set up and so forth. And taking a look at that.
0:42:41.6 Jim Mercer: Then we also looked across there, we did what you might call an IDC market glance, and we looked and said who are the innovators in this space and what are the types of things they're doing? Which I think is useful to know, right? Even though I'm an advocate to that, don't just go out and run on buy a tool, right? But you need to understand what kind of innovations going on this space and what are some of the things that you can do. So we've been kind of diving into this space to just get a broader understanding of what's going on and trying to keep abreast of what's happening out there in space, both in the vendor landscape and users, and to an extent what's going on with the governmental agencies.
0:43:19.8 Yadin Porter De León: Thanks, Jim. And Chip, where can people find Out more about you online?
0:43:25.6 Chip Childers: Yeah. I would actually say the thing to do is to find out more about what my team and what VMware's been up to. Go to the VMware office, the CTO blog. There you're gonna find a number of posts that we're publishing around the software supply chain problem space, how industry is working collectively with open source maintainers as well as federal agencies and governments to try to improve things, right? So the office of the CTO blog is a great place to go. And then the other thing I would say is for those of you listening that begin to engage perhaps in this rather important conversation with your peers, with those of us that are on the vendor side, you're going to run into VMware employees. Those employees are working in some of the standards groups that are defining things like how to present a bill of materials, how to deal with vulnerability exchange. We have engineers working in upstream ecosystems and upstream communities to try to help protect those public good package repositories. So say hi to them for me, or tell your team to say hi to them.
0:44:23.7 Yadin Porter De León: Yes, buy them a beer, slice of pizza.
0:44:28.3 Chip Childers: Absolutely.
0:44:29.0 Yadin Porter De León: Excellent. Well, Chip, Jim, great conversation. Thank you so much for coming on the CIO Exchange podcast.
0:44:34.0 Chip Childers: Absolutely. Jim, thank you for all the time. We've appreciate it.
0:44:36.7 Jim Mercer: Great. Thanks Chip. Thanks Yadin. Really appreciate it.
0:44:38.7 Yadin Porter De León: Thank you for listening to this latest episode. Please consider subscribing to the show on Apple Podcasts, Spotify, or wherever you get your podcasts. And for more insights from technology leaders as well as global research on key topics, visit vmware.com/cio.
[music]