Renewable Energy SmartPod
Renewable Energy SmartPod
How to Keep Data Centers Secure - with Bill Kleyman from Apolo.us
Sponsored by EDF power solutions
You've seen the headlines about the massive amounts of money that is being poured into AI data centers. You've heard concerns about how AI data centers are going to be powered -- and who is going to pay for it. But you probably haven't heard much about how all those data centers are going to be secured. After all, it sure would be a shame if you spent billions on a data center only to have it knocked offline by a physical or cyber attack!
Bill Kleyman, the co-founder and CEO of Apolo.us, joins the show to discuss the array of threats data centers are facing and some of the countermeasures that are being deployed. From ransomware and other cyberthreats to drone attacks and physical incursions, Bill outlines all the ways malicious actors can wreak havoc at a data center.
Bill also delves into some of the findings of the latest "State of Data Center" report. (Note: You can DM Bill on LinkedIn for a copy of the survey results.)
More resources from this episode
How Hackers Attack AI Data Centers with Wireless Devices
How the Internet May Be Taken Down
More resources from EDF power solutions
(Note: This transcript was created us AI. It has not been edited verbatim.)
Advertisement 00:00
This episode of the Renewable Energy SmartPod is brought to you exclusively by EDF power solutions. Looking for an EPC partner for your next community project? EDF power solutions’ DSP team is your trusted partner for turnkey EPC services with safety efficiency and cost effectiveness. Connect with the team from EDF power solutions, and let's build your future to learn more. Visit EDF-re.com or just click on the link in today's show notes.
Sean McMahon 00:42
What's up everyone And welcome to the Renewable Energy Smart pod. I'm your host, Sean McMahon, and today we're going to be talking, once again, about AI data centers.
You've already heard about the unfathomable amounts of money that's being invested in AI. And if you work in the energy industry, you're probably familiar with some of the concerns that are starting to bubble up about whether or not the grid will be able to power this tremendous growth in data centers. The whole issue started to become a political hot potato, because voters are starting to feel the pinch from increased electricity bills. But one of the things that people haven't been talking about enough when it comes to the build out of all these massive data centers is how we're going to keep them secure.
To tackle that sensitive topic, I'm going to be joined in a minute by Bill Kleyman, the co founder and CEO of Apollo.us. Bill is an expert when it comes to data center security, so we're going to talk about not only things like cyber threats, but also physical security. What can be done to keep malicious actors or drones with bad intentions away from all these data centers that are quickly becoming a key component in how we live our daily lives.
But before we bring in Bill, just two quick things. First, an apology from me. I've been getting over a nasty cold, so I apologize if my voice is raspy throughout this episode. And second, just a quick reminder that if you want a daily dose of renewable energy news delivered directly to your inbox, head on over to SmartBrief.com and sign up for the Renewable Energy SmartBrief or just click on the link in today's show notes.
So without further ado, please welcome to the show. Bill Kleyman, co-founder and CEO of apollo.us. Bill, how are you doing today?
Bill Kleyman 02:32
I'm doing fantastic. And I think, I think I actually need to tone down this energy level, because we Sean, we're gonna have a really important topic, a very sort of uncomfortable topic. I think there's gonna be a lot of butts shifting in seats, because they might be slightly uncomfortable. But here's the thing, get off your Amazon shopping list or your chat gpts or whatever you're doing right now. This is one of those conversations that you definitely want to listen to. So Sean, very long winded answer, I'm doing well, and I'm excited for the conversation, and thank you for having me here.
Sean McMahon 02:59
It's great to have you here, and I couldn't agree with you more. It's an important conversation. Important conversation and probably uncomfortable. As I mentioned in the open, there's a talk of hundreds of billions of dollars going to all these AI data centers. How we're going to build them, will they get to the grid fast enough? And I'm not hearing enough conversations about how we're going to protect them once they're online. So that's what we're going to talk about.
Bill Kleyman
Can't wait.
Sean McMahon
I want to tackle this in a couple of different ways. Obviously, cybersecurity and network attack is one way to hit these data centers. But I think it's more fun to kind of talk about the physical security, right? I'm a big fan of spy and espionage movies and things like that, and you know, so I guess I'm gonna ask the first question like this, if you were Ethan Hunt and the Mission Impossible team, and you were tasked with a mission that challenged you to penetrate an AI data center and take it offline, what are some of the tricks you'd have in your bag, and how would you get it done?
Bill Kleyman 03:54
Wow, I think that I you know, here's the kicker. Everybody listening. I got a set of questions from Sean, and this was, this was not highlighted, and really, literally in my, in my brain right now, I'm envisioning, like, like, an Ethan Hunt with, like, zip lining down a ceiling in between racks, and then stopping before the race tile floor to go unplug a UPS device. Probably a whole lot of effort for not a whole lot of impact. Okay, I think we should preface this by letting everyone know that I I've had a chance to work in a lot of different data centers environments. I've walked through many data center halls. So I don't think it would be fair for me to tell you break into a facility to steal stuff. But if I was, if I was, you know, a malicious actor, if I was a bad guy, there, there's definitely a few lessons learned that we've we've experienced now. There's, there's two parts to this question. Sean and I really kind of want everyone to hear this out. Yes, it's kind of funny to think about the whole mission, impossible thing about how we get into a facility. But for. For everyone listening, data centers have been around for a while, right? The challenge that we've gone from these nondescript warehouse type buildings built in the middle of nowhere to, in case you haven't noticed, this industry is having a moment, right? We're no longer buildings full of servers. We are literally a part of the national critical infrastructure now, you said billions. I just got back from the Schneider Electric Innovation Summit, and they put up a statistic and trend that by 2030 global infrastructure, I mean that this is not just data centers, but all the stuff that supports it. We're talking like water and cooling and power systems is going to reach $106 trillion globally, which is just a ridiculous number in terms of how much we're going to be putting into these things, right? So we're seeing how these facilities, they're moving trillions of dollars in transactions. They run hospitals, they control logistics, the power of AI engines that all of you right now are talking about, right? And so when we start to take a look at data center security, you know, I love what you said, right? It's, it's truly a layered approach, right? Physical security, logical security, working with supply chains, we'll probably talk about. But the threat landscape has shifted from a teenager with a hoodie, which is actually ironic, because I'm not a teenager, but I'm totally wearing a hoodie to nation states, organized crimes, insider risk, and even someone with like a drone and a grudge, right? We're going to talk about that, you know, or someone as motivated as Ethan Hunt trying to zip line through a roof, right? The good news is that the industry has a lot gotten way more serious, and we don't really talk about it, but the stakes are much higher. So when we start to take a look at physical security, first of all, I promise, Sean, I'm going to answer this question you asked, Bill, what are my What am I? You know, you know, tools in the back. So to say, you don't hear about this very often, right? And that's because operators and their customers, if there's a successful data center attack, are going to be very quiet about it. They're not, they're not going to talk about it, right? And we can, we can look at patterns. So let's, let's talk about, maybe not necessarily Ethan Hunt, but Ethan Hunt like situations where this has happened, right? So we've seen break ins happen at like, telecom huts, small edge facilities. They're targeting like networking gear, batteries and to be honest with you, copper, we've seen the theft of servers from smaller hosting providers where you don't need much of a tool bag. You just walk in, you know, open a door, there's, there's a funny saying, you walk around with a clipboard and a yellow vest and no one's gonna ask you questions. That's literally somebody who can walk in this this is this happen they walk in and you can just grab the stuff and and walk out because, like, access controls are monitoring or weak, or sometimes even not existent. And in some regions, operators have to deal with things like cable cuts, vandalism or, you know, protesters trying to block access roads. I think the big lesson here is is the closer that you are to critical infrastructure and valuable data, the more layered defense you need to need to go after. So if I'm a malicious actor, I'm looking at things like fencing and bollards and cameras and mantraps and biometrics and armed or well trained guards, right? Many malicious, strict visitor management, mature operators, the ones that they you know you they don't care what you have in your tool bag you are not getting in. Already designed under the assumption, under the assumption that someone's going to try the door, they're going to rattle the fence, they're going to test the procedures at some point. So if I'm a malicious actor and just kind of backing up i I'm contributing editor to dark reading, which is an information week publication. So I've had a chance to do some diving, everything from like the dark web into physical security, and I know we're going to talk about this, but if I'm the malicious actor, and this is for everybody listening, think about your target points, how would a malicious actor get in? What would they need to get in right? And here's how wild it gets. I just spoke recently on a podcast with a company called Bastille, B, A, S, T, I, L, L, E, and they focus on drum roll physical security, and this is a situation where, all right, I'm gonna, I'm gonna try and break into this data center. And what I want to do is I want to exfiltrate some data. Well, what I'm going to do is I'm going to take a drone, not just for reconnaissance, but to scan and map RF emissions, Wi Fi environments, maybe Bluetooth beacons that were properly secured. Maybe look for wireless access points that have been updated and they have security flaws. I'm going to hover. I'm going to land on there. I'm going to potentially take some some recon. I know, I know we're going to talk about this, but that's how I would potentially get in. Now, if it's a physical break in, there's going to be some, some serious surveillance that has to be done when I was, when I was a younger like network security engineer, we did dumpster diving. We went and found, for example, how some companies didn't do proper shredding of their documentation. And I was able to, by dumpster diving, find out what a company's IT organization was, and I was able to give him a call and say, Hey, this is Bill Klayman with your IT company. I'm doing a Microsoft Update right now. Could you go ahead and give me a username and password and we're able to get in right now. In those situations, if I'm a malicious actor, my tool in the bag for me, at least, is my gregarious personality to be able to get in. Otherwise, there's RF frequencies. There are, you know, different ways. Is poking at a defensive architecture, but in in an environment where you are truly, truly, truly, truly mature, right? And you've got those defenses, the the fencing, the biometrics, the locked doors, the security guards, it's not it's not easy to break in, and that kind of environment, and number one, once you're in, getting out could be just as difficult, right? So you have to have a very, very targeted approach and a reason to break into a data center facility. And also, what's your goal? Are you trying to get data? Are you trying to get a server? Are you trying to disrupt something, right? Are you what kind of a malicious actor? Are you? So I think, I think that my tool bag in this is horrible for me to say depends on the malicious activity that I not mean obviously, but someone is trying to do, but what? What a great start. I think the challenge is that our our bag of tools, has evolved so much. Not our bag, I think the malicious actors, bag of tools has evolved so much that, you know, when we look at infrastructure, physical infrastructure security again, not just somebody trying to do a ransomware attack, which we'll talk about cybersecurity, but actually inflict some kind of a physical damage. It's much harder today than it was ever before, but it's certainly not impossible because of things like, obviously drones and firearms and other like physical security threats against these facilities.
Sean McMahon 11:18
So we talked a lot about drones right there. So yes, you know, I feel like that technology as a weapon, yeah, has advanced quite a bit. You know, the war in Ukraine has taken drone warfare from something we used to like imagine 10 years ago to it's real and it's advanced quite a bit. So yes, that's the first thing that came to mind when I was thinking about how you would just physically attack a data center. Would be a weaponized drone. So what kind of systems are in place for that? Are we jamming these things? We're not shooting them out of the sky. You know? I mean, has some of the first of all back up, like have such attacks happen where they're just trying to inflict physical damage, not scan and pull network information, but just hit it. Any instances of that happening? And what kind of defenses are companies using to prevent that.
Bill Kleyman 12:01
It's fascinating that you're mentioning this. Yes, it's happened. An armed drone was flown into a tech facility, unfortunately, in Ukraine, and then just that's I think we should also note, for the sake of transparency, I'm Ukrainian. I was born out there, so you know, we've got friends and family out so some of this information is certainly firsthand, but that that happened, but that's also an active war zone, right? Have we seen something like that happen in the United States? Not so much right now and again, this is going back to this really, really fun webinar that I did with the security company, and then this is, this is specifically what they focus on. Here's the reality, the situation, and this is, we're gonna get a little bit uncomfortable, and I'd really like for everybody listening, whether you're an enterprise facility, whether you're working with a co location, or you've got a partner. You know, maybe you don't care at all about a data center, but you should, because your data is probably in a data center somewhere. Right? Here's the future landscape of where drone warfare is such a strong term, right? Where drones can be used in terms of a malicious activity. But so yes for reconnaissance, mapping cameras, guard routes, capturing license plates, even trying to drop devices on roofs or near intake vents to take out things like cooling architecture. On the more advanced side, you can imagine drones being used to map RF so radio frequency emissions or Wi Fi environments around a facility, and that's why you're seeing more operators implement things like drone detection, geofencing alerts, close quartered coordination with local law enforcement in even aviation authorities around restricted airspace. While you're seeing some of these really critical facilities actually have restricted airspace around their facilities, right? And so there's, there's some ways that you can effectively do drone detection, working with, like I said, local law enforcement, right? Getting really good geofencing alerts, having some rudimentary radar systems to see, like, hey, there's, there's an incoming thing that's not a bird. Those are, those are absolutely means and methods of security, but, but the reality situation is, I'm worried about this. Certainly physical security is a big one. Actually want to make sure that we note on this. So a part of what I do in this industry is I get a chance to write the afcom state of the data center report, and I get a chance to present at that conference and event. And so I've written this report for the past nine years, and in the 2025 for the ninth consecutive year, we'll talk about ransomware. It literally was the top concern for many of these data center operators, but the number two concern, and it's honestly the first time we saw it hit like the top three, was human threats, internal or external. So 57% of our respondents came back and said that we are concerned that some human is going to do a physical there's gonna be a physical threat against our facility, whether it's someone doing it internally, or someone coming in from coming in from the external, that is a rise of 17% from last year's survey, which was at 40% right? So people clearly are asking the exact same questions, Sean, that you are asking of me, and hopefully what everyone else is thinking about you. Yes, we've seen drone warfare come quite a long way. You don't have to drop an explosive device. You could literally fly a drone into an HVAC unit or some kind of a critical piece of infrastructure to cause some serious damage. There's, there are, like I said, some means and methods to protect against infrastructure attacks in this way, but it's certainly an emerging threat? I mean, that's a great question.
Sean McMahon 15:22
Yeah, I'm not trying to get all “Ghost Fleet” on you on that, but I just think it's important.
Bill Kleyman 15:25
It's we look I'll be I can count on half of my hand how many meaningful physical security conversations we've had around the data center. Or we call out some of these real world threats. And you're absolutely right. The evolution of these kinds of threats that targets and the motivation behind these malicious actors, it is truly evolved folks these facilities that we're talking about. I firmly believe that every data center is going to become an AI data center eventually. How they get there, that's entirely their own journey. But please, please, please, understand one really important thing, that these facilities aren't hosting like email servers anymore, or just a sequel box, or just like, you know, an as 400 or whatever it is, right? Or a Dell server that costs $5,000 if it breaks, they're now HPC, high performance compute, these AI driven types of facilities, right? If you lose a Dell server and it's $5,000 Oh, no, shoot, let's go get a different one. If you lose a DGX node, that's one of those from Nvidia's with 8h 100 cards in there, you've just sacrificed a Lamborghini. And I'm going to teach everyone a new technology acronym now that we have enough of them. It's called an R GE. That's called a resume generating event, okay, something we don't want to no one wants to experience that one. But when we, when we start taking a look at sort of these, these kinds of environments and these physical infrastructure, it's really important to note the vast difference in the types of things that these are hosting, and the fact that every single person listening to this podcast, every single one of you uses a data center in some form or fashion, and most of all is now a user of generative AI as well, where these facilities are hosting this kind of architecture.
Sean McMahon 17:10
We'll be right back.
Just a quick reminder that today's episode is brought to you by EDF power solutions.
Break ground with confidence. EDF power solutions’ Distribution sScale Power team specializes in EPC, community solar and environmentally sensitive sites. From development to construction to O&M, EDF power solutions is your trusted partner from 50 to 150 megawatts. Connect with the team from EDF power solutions to explore your 2026 build. To learn more. Visit EDF-re.com or just click on the link in today's show notes.
And now back to my conversation with Bill Kleyman.
So now you brought up the AFCOM report that you said you've you've played a hand in the last nine years. And I want to dive into that. What are some of the trends we're seeing there? And again, maybe I should flip the question. Instead of being Ethan Hunt trying to get in, pretend you're the person trying to protect you from the mission. Impossible team.
Bill Kleyman 18:17
Yeah, holy cow. I spent about four years at this little data center company called Switch data centers out in Las Vegas, Nevada. They're not little. I'm being facetious. I was really fortunate. I was their executive vice president reported to rob the CEO. And I was so, so fortunate to be on a team that delivered world class architecture, right? But on top of that, there was also world class security for anyone. Security. For anyone familiar with the switch facilities, security is paramount to what they do. The folks that work in those facilities are, you know, very well trained. Most of them are our former law enforcement, military, special operations folks. And the one thing that switched it differently is they didn't contract that out, right? And that's not to say that it's a bad thing. There's, there's really, really wonderful organizations out there, Overwatch, for example, full of military people that are helping support data center operations, for example, or salute mission critical, another really great organization that takes former military personnel and assigns them into into the data center industry. If I can take my expertise, and to be perfectly honest with you, switch data centers, set my bar in terms of what it looks like for security. It goes into two very important aspects. One, obviously, you have to have really good physical security parameters, all the standard stuff, everything from biometrics to good fencing to good walls to good camera systems using AI detection systems, even things like autonomous systems, like drones and robotics to scour your your parking lot to make sure that there's not cars that there aren't supposed to be there. But all of these physical systems, Sean and everybody listening, all these physical systems, are useless unless you train the person as well shared responsibility and security. Is going to be one of the most important, important things that you could ever imagine. So I remember very specifically being trained bill. When you pull out of the gated area, you stop right in front of the gate, you wait for that gate to close, and then you proceed. Now that effectively prevents somebody behind you from bumping into you and going in through an open gate. That's an example. Even things like, like going through the the man door traps, right? You open the first door, you go into the man door trap. You can't open the second door. You don't open the second door until the first one closes. First one closes. Great, now you can open the second door, right? And there's, there's all, all of these, obviously, no pictures inside the facilities. There's so many different ways to be able to understand and clearly identify a threat. If it doesn't feel right, how do you report it? The person is is huge, huge, huge part of the security training, security awareness training, right? And this is we're still only focusing on the physical part of it. We can talk about the ransomware. You know, if you see a phishing gaming email, you know your CEO does not need an apple gift card. Don't respond to that. That's probably a phishing email. But the physical security part of it, again, going back to some of the biggest threats out there, it's a lot of understanding, you know, physical hardening, understanding what critical components are there. You know, if it was me and I'm sitting there trying to prevent the Ethan Hunt, there are some really important countermeasures that I would incorporate against let's call them individual actors, right the physical hardening of the critical environment. So like elevating or shielding transformers, protecting fuel tanks, one of our partners in Omaha, Nebraska, has what's called an envelope design. Everything is surrounded within a concrete wall barrier. I'm talking like everything from 20,000 gallons of diesel fuel to the actual eight generators. They're all inside the facility, inside of concrete walls and a concrete barrier that goes as far as far up as it does down. I mean literally, it's in the ground and all of their telecom and fiber lines, are you ready for this? They are encased in concrete eight feet underground. So, like, like, there's ways that you can do this right, access, control and segmentation. So even folks, even if someone gets in the perimeter, they need to hit multiple layers before they can even reach a customer, equipment, or, like, a critical plant, so badges, biometrics, pins, escorts, separate paths for staffs, visitors and contractors. You know, not everyone needs to be following the yellow brick road. There's going to be different paths for different people that are coming into your environment. And honestly, right now, your ability to do detection and response is unlike anything. Please understand it's unlike anything before. These video cameras that we've got right now, high resolution video and analytics that can detect loitering, literally the AI agents like, Hey, I think this guy is just literally loitering by the edge of that fence. You should go check it out right now. This Is that unusual activity or unusual movement that we can now use AI for monitoring alarm alarms on doors, gates, critical rooms, plus an integration with 24 by seven security operations and local police. Look man, when I was at switch, they did, they did practice with SWAT response teams, local law enforcement like, literally practice sessions, just in case there was ever a situation. And again, I'm gonna harp on this one more time, Sean, before we move on, people in process, you could have the coolest gate in the world, but if no one knows how to lock it, what use is it? I know that's funny. Obviously we all know how to lock a gate, but regular drills, clear playbooks, training garden staff to treat anomalies very, very seriously, because now we're getting much better at finding what those anomalies are, and the fastest way to fail is to have all the technology in place, and people who are completely numb to the alerts like that's that's the easiest, fastest way to fail. And, yeah, don't do that.
Sean McMahon 23:55
So one question that's popped to mind, you know, while we've been talking here and amid this huge construction boom of AI data centers is, How early do all these countermeasures need to be considered? You know, because, literally, on a construction site, you know, could someone get in there and plant something or, you know, put something on the site that is buried maybe, and just wait for the construction to finish, wait for the AI data center operator to come in and put their stuff in. And then kind of either flip it on or whatever, like, how secure are the construction sites before these centers are even completed and up and running?
Bill Kleyman 24:30
It's a great question Sean and I love, I love how you're thinking, you know, kind of, kind of like the bad guy, right? Where does the where does the flow of malicious activity start, ultimately, right? It was it once it's built, or before? Yes, but that's also why you're seeing much, much of what we talked about right now being incorporated into physical design and architecture and construction from the ground up, right? So physical hardening of critical components, access control, detection and response. You will see many of these construction sites with really. Enhanced cameras, right? And also people in process. There are 1000s of folks coming in and out of these facilities. There's multiple points of validation, but there's also a chance that some one, malicious dude is going to do something. I'm going to give you an example, a real world example. This happened. Not going to say who it is, but this really did happen, a facility was coming online, sizable, not, not small, I think maybe 50-60, megawatts. And it was almost done. It was almost almost done, right? And inside of the environment, there are these, these overhead busway systems that have a lot of the power infrastructure running through them, right? And we're talking like 48 hours before it was energized. 48 hours before energized a facility, individual whose job was to clean right, clean the racks and all that stuff, comes in and starts cleaning right, and starts cleaning the overhead busway. And as he's cleaning it, between two junction points of where there's just the massive amount of voltage was being connected. They found paper clips, paper clips, paper clips like jammed into electrical equipment, right, right, and had that person who was cleaning when they were cleaning it up there, not they saw one fall out. They're like, whoa. Why is there a paper clip up there? Called obviously security, called maintenance. They came and they reviewed the entire infrastructure. They didn't find anything else. It was literally but that one junction point had that become energized, I don't know what could have happened. It could have been an arc, Flash. It could have been because someone could have been killed, really, or injured horribly, right? You could have taken down the damn facility. You could have caused a fire. That's That's legit, right? And so this construction company that was putting the stuff together actually had to change how they deployed high voltage electrical systems, even like the final inspection site where those junction was happening to make sure everything was clear. So here's your example of exactly what you talked about. This facility was being built. It was not energized yet, and someone did something because they were I I can't get into the detail of why this person did it. It was actually just like a grudge that they had against their employer. That was their malicious intent, unfortunately. But that's what this person did and and it was, it was bad, right? I've heard of other use cases, but this was legit one where pre deployment of a facility, there could have been a very serious incident. So to answer your question, someone's driven enough, someone wants to they certainly can do some of this stuff. But that is why process and protocol are just paramount to validating to make sure that everything is safe, that everything can be turned on, that you don't have any of these little scary easter eggs that you're going to find a little bit find a little bit later. But, you know, to everybody listening as you build some of these facilities, and you need to do final walk throughs, final inspections, just, you know, Sean, you You did it best. You've asked me to wear the hat of the malicious person with ill will. Sometimes you need to do that as well with critical infrastructure. So, I mean, that's a great question. Yes, when you're building these things, you need to make sure that they're secure. They're secure across the board.
Sean McMahon 28:05
Well, I think we've, you've done a great job of kind of covering the physical and the human threats and how those can be, you know, mitigated. But now let's talk about the cyber and get into that. So what are you seeing there? What kind of trends? Obviously, you can't talk about some of it, some of it's still in the dark. But how are data centers protecting themselves from that kind of threat?
Bill Kleyman 28:25
There's, there's five aspects that I really want to make sure that we talk about, especially when we come to cyber security. There's, you know, what keeps data centers up at night on the cyber security side, at least, is, is persistently changing, and historically, the concern was someone literally breaking into a customer workload. Right now they're going to take some data whatnot. Ransomware is a big one, and I'm going to implore everybody on this call. I wrote an article called the data center ransomware attack that cost you everything. And when I say that, I literally mean everything. It took down an entire data center a cloud service provider out in Europe, because a malicious actor was able to access a virtual machine of management, machine that was in a virtualization environment that was then moved VM, migrated into a production system, and then impacted everything right? And if you take a look at the article, the sad letter that the the CEO and owner had to send to everybody is like, there's nothing to recover. There's nothing. We have nothing to recover. I'm so sorry. Right, all gone, right. We're not talking like an Amazon level, but it was a real world cloud provider. So again, the data center ransomware attack that cost you everything. It's on data center knowledge. I do implore everyone to give this a read, because it's a wake up call. The number one, the number one threat for the ninth year in a row, which has been the existence of the afcom state of the data center report, has been ransomware. And it's 60% of the respondents that came back and said we were really worried about this, up from 54% not that that was an insignificant number to begin with. And you know, we talked about the physical security part of it, right? But ran. Answer. More specifically, these, these attacks are becoming so incredibly devastating and absolutely massive in like, in like the terabit, terabit, basically space that they are just, they're just hitting these environments so hard, right? You almost have no recovery. So you have to have ways that you can offload that kind of attack. But when we start to take a look at cybersecurity, there's, there's five aspects that I want everyone to focus on. Number one, obviously, is ransomware, right targeting shared environment, so hitting management domains, virtualization layers. That's a big one, great target backups that support multiple customers, that can escalate from a customer incident to a facility wide crisis, which is exactly what I talked about in my article, the other, the other part of the conversation here. So much. So much of our systems in our facilities now are becoming connected. So you may have heard of the whole it ot operational technologies, those things are getting IP addresses. So BMS, building management systems, dsim, data center infrastructure management or OT operational technology systems. If someone can manipulate cooling a power plant or monitoring, they can create real, physical damage, right? You know, if you, if you figure out a way to turn off a cooling plant, congratulations, you've done it right. And we've seen, for example, here in Chicago a data center, right? You know, talking about, I don't want to say who it was you everyone knows how to use your favorite search engine went down because of a cooling issue. Their cooling plant went down. And all of a sudden, the Chicago Mercantile Exchange stopped running. Holy cow. Now, that was a mechanical failure that wasn't any sort of, you know, malicious attack, but if that was you just successfully took down on a major financial trading operation, right? That is, that is a massive, massive thing to think about. So one of the big things I want to make sure everyone talk about this is where it and OT convergence can get dangerous if it's not properly secured, or if it's not designed correctly. Big piece of advice. See, what are your environment as an IP address? How connected is it? When was last time you updated some of those systems? Right? Are there? Are there malicious holes that a malicious actor could get into? The other one that I want to make sure that we talk about our vendor and supply chain access so remote connections from OEMs integrators that support things like UPS is and chillers and generators and even building controls. Remote Support tunnels is also a potential attack path. And the more support tunnels we create, the more roadways that we effectively, you know, position for malicious actor to get, to get through. So you need to be, you need to be conscious about that. And interestingly enough, when, when a supply chain fails and there's an actual issue with it. It can be it can be pretty substantial. I think we recently heard from that same afcom state of the data center report that let me see if I can find that metric here, upwards of about 20% of respondents experience an outage, a system outage because of a failure in supply chain. Right now, that's just something breaking apart not being available. Now, if that was a real like incident where supply chain completely broke, you could be experiencing some pretty substantial issues within a facility. And obviously, the other big thing that we talked about are going to be insider threats. You know, we mentioned this a little bit earlier in terms of security. Again, that's like the second biggest challenge right now for the lot of these facilities. So contractors, service providers, staff who have legitimate access and either make a mistake or act maliciously with higher privileged accounts, one bad action is like a tidal wave of bad things happening. Okay? And now that these facilities, there's more people moving through them. There's more people getting accessing into these architectures. Really make sure you monitor, manage and monitor who and what is accessing those kinds of systems. And finally, finally, the most important one, one that I I'm not going to ignore, AI in automation, blind spots as operators begin to introduce more AI for monitoring, troubleshooting, optimization, those models, data pipelines, API integrations, those are a part of your attack surface. And if you trust a model's output blindly, you can be manipulated. That's the kicker, right? So in Apollo and everyone, you're welcome to take a look at what we do. We build agentic AI and AI apps, right? But all the stuff that we've created, none of that is an elephant on a unicycle. Looks great, but it doesn't actually nothing do for your business. But in that kind of architecture, you must be very, very careful in terms of how you design these things and keep the human in the loop. Architecture always in place right? Make sure it's validated. Never trust it blindly. Always trust your gut and make sure you validate what's coming out there. Those are, those are the biggest from a cyber security right now. So ransomware compromise of BMS, DM, OT, systems, vendor supply chain issues, insider, or we're called near insider threats. And. Obviously things like AI and automation blind spots, those are, those could be some of the biggest cyber security challenges right now in the data center industry.
Sean McMahon 35:05
Okay, so we've spent, you know, last half hour or so talking about all these threats, and you've done an excellent job outlining, you know, physical cyber, you know, every kind of attack point that could be vulnerable. So someone like you who's lived this industry, written about it. You know, quite influential. How would you rate the preparedness of the data center industry as a whole to some of these threats that you've outlined here today?
Bill Kleyman 35:31
All right, let's, let's talk about the physical side of it. First, we as an industry have been thrust into the spotlight faster than anything ever before. I'm really fortunate. I've been in the industry for 20 plus years. I graduated with a network engineering and telecommunications undergraduate. So I'm a geriatric millennial. I don't know why you have to call me. That's completely unnecessary. Just call me older millennial. But when I graduated in 2004 with my bachelor's degree, I went and started working with like in network closets and data rooms before they even call were called data centers, right? And I've seen this evolution. And what has happened in the past 24 months has eclipsed my two decades in this industry in terms of the rapid pace of expansion in innovation. So to answer your question, truly, you know holistically, you need to understand that there's different types of facilities that are out there. There really are, right? There's everything from tier one, these new tier zero architectures to tier four, plus, like heavily secure kind of environments, government facilities, government rated facilities, for example, again, you can be incredibly secure. For example, one of my partners is Scott data center in Omaha, Nebraska, originally sanctioned by General Cartwright for DOD operations, right? So you better believe that they've got some crazy levels of security. Another great example was my time at switch data centers. Holy cow that they take security seriously. Now I'm not going to mention some of these other ones by name, because I don't want to get a very mean phone call. I've been in walk through facilities where there is no fence. You just got to open a glass door, and you just walk down a hallway, and you could just literally get into a facility by accessing one more door, right? And in theory, could I go in through a back way into this sort of, like shared commercial space that has maybe, like 20, 30,000 square feet of data center space? Could I find a way there? Probably. And then, could I do something for like five minutes before some security guard tackles me. Yikes. Remarkably even right now I can say yes, I can, like, in some really critical infrastructure types of places where there are housing, some really critical infrastructure types of things, I could do that right now. I could find a backdoor, I could probably get in, and if I had nothing even on an agenda of what I would do maliciously. Could I run around and unplug stuff and press a whole bunch of buttons for five minutes before again, I get tackled? There are some facilities that are out there they can do that. I think, on a whole, how prepared are we as an industry for physical threats? Six and a half? I would give it a six and a half. Because, you know, what really, really concerns me is, you know, copycat risks that are similar to what we saw for like electrical substation attacks, right? That's scary man, where someone wants to damage external infrastructure like a transformer or fuel tank or switch gear from a distance, that's scary man, it's, that's, we have all that stuff supporting data centers. What prevents somebody who is, you know, NIMBY, not in my backyard, don't build facilities out here from from taking out their anger and unfortunately, choosing violence instead of civil discourse, that's that's scary to me, right? And so a long distance pot shot. How are we ready for that? We might not be. We might not be ready for someone to do something really extraordinary malicious and violent against one of our facility, but I think overall, from a physical security environment perspective, six and a half, obviously, with some smaller, less funded facilities on the lower side, and some, you know, other facilities, like, like a switch, for example, or my partners at Scott setting that, setting the bar at like, like a good eight or a nine, nine and a half, And somewhere in the middle. Now, from a cyber security perspective, I honestly believe that we're maybe a little bit more like a like an eight, like a seven and a half or an eight, and really, we're never going to get more than that, just because the malicious actors are usually, like, six months ahead of us. That's just, that's just how we play in this game, right? We're always trying to play catch up. But when we look at cyber security attacks, they're just persistently creative. And there's a whole lot of valuable targets that continue to emerge. And that's that's kind of the scary part about it, is that the more digitized we become, and the more data that we use and share and create in this world, the more targets there are, and every thing and everybody. Is a target, whether it's a massive organization, or your grandma trying to get her old grandchildren picks back, or something like that. Those are all legitimate threats. And if you think that a malicious actor isn't going to go after Grandma, and you know ransom, where all of her pictures and give her, step by step instructions of how to go get bitcoin and send it to them. Have faith they're gonna do that. In fact, that's literally a use case that I heard, unfortunately and read about, that this grandma paid, I think, $2,000 in Bitcoin to get her pictures back of her grandchildren. From a cyber security perspective, I would say we're probably like a seven and a half or an eight, and only because the ramifications of an outage right now and the cost of a downtime, both from like a market perception, but also the damage that it can cause. I definitely believe that preparedness for everything that we're experiencing, the best way I can put it is uneven. Right, the top of the market is building fortresses with backup fortresses. That's what it feels like. But there's still so much tail that has work to do, right the other part of this market, but the encouraging part, the encouraging part, awareness is up, ransomware headlines, critical infrastructure incidents, the AI growth have forced, forced boards, investors and regulators, to pay attention, including like this conversation that we're having, I firmly believe over The next few years, it's going to be about timing and about turning the attention into consistent, measurable security outcomes, not just tools and slide where, but incorporating really powerful tools and the people that can that can enable all of that. So just just, just a little positive sprinkle before, before we sign off here, because I genuinely believe that we have so much more awareness and conscious thought about this that at least we're making it hard for the malicious actors.
Sean McMahon 41:47
All right. Well, I appreciate that ratings, you know, scale of one to 10 for both kind of buckets of this data center industry and the preparedness. So Bill, listen, you're a found of knowledge on this. I appreciate your time today. So thank you very much for joining me and sharing your insights.
Bill Kleyman 42:01
Sean, it's, it's an absolute, absolute pleasure,
Sean McMahon 42:05
Alrighty, well, that's our show for today. But before we get out here, I want to say one final thank you to the exclusive sponsor of today's episode, EDF power solutions.
If you like this show, please share it with your friends and colleagues, and of course, be sure to follow us on Apple, Spotify, YouTube or wherever you get your podcasts. The Renewable Energy Smart pod is a production of SmartBrief, a Future company you.