Episode Categories:
Resources:
Join the discussion: Ask Jordan and Alistair questions on Microsoft AI
Upcoming Episodes: Check out the upcoming Everyday AI Livestream lineup
Connect with Jordan Wilson: LinkedIn Profile
Try Our Free AI Prompting Course: Register for our free Prime, Prompt, and Polish AI Course!
Microsoft Azure Developments from Ignite Conference
The ever-evolving field of cloud computing continues to amaze with forward-thinking advancements and solutions. One needs to look no further than recent announcements from the Microsoft Ignite Conference for glimpses into the future of AI and cloud-based solutions.
Discovery of DPUs: The Future of Processing
With the constant evolution in processing units, the emergence of Data Processing Units (DPUs) marks a significant leap forward. These specialized chips streamline storage operations, replacing the traditional roles of CPUs and network cards. Their main function is to efficiently read data from hard drives and distribute it via the network.
High-Speed, Low-Power Consumption
The innovative design of DPUs enables them to deliver impressive performance with power efficiency. The DPUs draw approximately 100 watts per server, significantly less than the 400 watts drawn by traditional data center servers. This approximately 4X reduction in power consumptions paves the way for more energy-efficient and cost-effective operations.
Increased Security with HSMs
Data security in the cloud takes a formidable stand with the introduction of Hardware Security Modules (HSMs). Earlier, HSMs were standalone units designed mainly for generating encryption and decryption keys. Embedding this technology inside servers ushers in a new level of security, allowing encryption and decryption tasks to be performed much faster.
Addressing Environmental Concerns
The pursuit of more efficient ICT solutions is guided not only by performance goals but also with an eye on environmental sustainability. The integration of quantum cryptography in technology infrastructure is part of this forward-thinking approach. This approach ensures not just secure data handling, but also contributes to environmental sustainability, with less impact on energy grids.
Special Considerations in Construction
Moving beyond the software and hardware innovation, the discussion also covers material considerations within data center constructions. For example, cross-laminated timber is emerging as a preferred material due to its lightweight nature, ease of use, and lower carbon footprint as compared to traditional construction materials like steel and concrete.
Looking Ahead
As businesses around the world continue to rely on cloud-based solutions for a plethora of tasks, it’s clear that the underlying technology will need to keep pace. The developments announced at the Microsoft Ignite Conference represents a forward-thinking strategy to meet these evolving demands. With each improvement in areas of energy efficiency, processing power, and security, these advancements are a significant stride in making AI technologies more accessible and beneficial to businesses around the globe.
Topics Covered in This Episode
1. New AI Tech by Microsoft at Ignite
2. Detailed explanation about DPU
3. Chip specialization in the tech industry
4. Role of the HSM in encryption
5. Future developments and challenges
Podcast Transcript
Jordan Wilson [00:00:15]:
As soon as you think you knew what was happening with GPUs, then there's something called NPUs. And today at Microsoft Ignite here in Chicago, a lot of us got our first taste of DPUs. It's like you can hardly keep up. But here's the thing, with what Microsoft just announced today at the Microsoft Ignite conference, I think things are gonna be getting probably cheaper, faster, and more secure. It's like that triangle of 3 things we all want, but you can never really get. Don't worry. I brought in someone much smarter than me today to help us talk about that. So, please help me welcome Alistair Speirs, the Senior Director of Azure Infrastructure at Microsoft.
Jordan Wilson [00:00:59]:
Thank you so much for joining us.
Alistair Speirs [00:01:00]:
Thanks, Jordan. Alright. Cool.
Jordan Wilson [00:01:02]:
So can you tell us a little bit first about what you do at Microsoft? What is your role?
Alistair Speirs [00:01:05]:
Yeah. So I work on the global infrastructure team, and our job is really to manage the data centers, growing capacity, the worldwide footprint of our cloud infrastructure, from a macro lens of, like, where to build, the power we need, the the market kind of forces that are driving, like, the relationships with our Intel's, AMD's, NVIDIA's, and building up that infrastructure. And then, of course, what goes inside the server as well. So that is the server architectures and some of our own first party silicon development work that we do as well.
Jordan Wilson [00:01:37]:
Yeah. Can you tell us a little bit what was announced today? Because there was a lot you you know, there was chips, there was HSMs, DPUs. There was so much announced today. I think they said it was, like, more than 90 different, like, AI announcements, and I was trying to keep up. But, at least on your side, what was actually announced, you know, with these, new DPU chips and the HSM chips? Yeah. I think, you know and Satya said
Alistair Speirs [00:02:00]:
it really interestingly. It's the middle innings of this era of AI. Right? And so it was almost a progress report and all the work that we're doing to optimize this infrastructure and optimize the capabilities to deliver AI to more and more people. So if I think of, like, optimization, you know, every layer of the stack, it's not just what we're doing in software. It's not just what we're doing in, like, data center facilities, but the hardware and then the specific types of hardware that make up this data center kind of platform. We talked about optimizing specific ones there as well. So first, the the Azure Boost DPU. So DPU is a special type of processor or what we call an an ASIC or application specific integrated circuit.
Alistair Speirs [00:02:42]:
And this chip is designed to do one thing and one thing well, and that's storage operations. So you can think of it as taking the place of a CPU, a network card, other infrastructure that goes on in a traditional server, replaces it with one card that just has one job, which is to kind of read data from from hard drives and push it out over the network. And by doing it and by integrating that into this really single purpose chip, we're able to get a lot more efficiencies out of that chip, use less power to run that operation as well.
Jordan Wilson [00:03:17]:
So could you kind of help, put the last couple of years into perspective? Right? Because I think when the, you know, the chat GBT wave, you know, of 2022 came and, you know, then we've had, you know, Copilot now generally available for more than a year. But was it kind of like you just had chips doing all different jobs and maybe they weren't too efficient, and that's kind of why and what's led us to now having these NPUs for edge AI and and the DPUs? Is that kind
Alistair Speirs [00:03:45]:
of what's happening? Essentially, you're seeing a lot more efficiencies out of this. You know, the early models will run, you know, brute force infrastructure with CPUs, GPUs, whatever we can get our hands on. Like, let's let's run these models and try and train and infer these AI models on that infrastructure. But, you know, over time, as we optimize, like, the models a little bit more and get better at the software, as new chips become available and new capabilities become available, we'll start to fine tune the infrastructure to really focus on this kind of emerging category of AI type apps. And that's really what you're seeing here with new new GPUs that, you know, graphics processing unit or the full name was GPGPU, a general purpose graphic processing unit. Right now, they're getting more and more specialized in just that AI acceleration workload as well. So they're almost becoming more fit for purpose rather than these general purpose computing devices. They're really targeted to particular use cases.
Alistair Speirs [00:04:43]:
And then if you think about the scale we operate in, in the Microsoft Cloud, as that infrastructure gets bigger, as the scale gets bigger, then there's more opportunity to optimize for kind of a very targeted workload like AI or like storage or like security as well.
Jordan Wilson [00:05:00]:
Okay. And for the dorks of us out there, I'm a little dorky, but, you know, what are some of the specs? Like, as an example, on the DPU, I think it I I read, is it like 4 x more efficient, something like that?
Alistair Speirs [00:05:11]:
So, you know, this DPU is really, looking to run about a 100 watts per server. And you think about that, the traditional data center server with, like, a traditional CPU, traditional network card, it's probably drawing about 400 watts. And so, you know, 4 x reduction in power is super important for us, especially in this world of, like, GPUs and AI accelerators doing more and more power draw as they grow as well.
Jordan Wilson [00:05:39]:
That's that's a big thing to think about because I think there's a couple of things. So you have your, you know, your bigger companies that have to worry about where are they getting their power for AI. Right? And then there's also the even bigger picture, right, the environmental impact. Mhmm. So how does, you know, some of what's announced, what that that was announced today at Ignite go to address both of those things. So how is this going to impact companies having, you know, access to DPUs that are, you know, 4 times more energy efficient. And then what does that mean for the bigger picture, like the environment? Yeah.
Alistair Speirs [00:06:15]:
That's a great point. Like, the cloud's always been designed on this, like, economy of scale efficiencies. And so working, you know, in a cloud environment is generally far more energy efficient than running, like, an on premises server. And we're building, like, fit for purpose buildings that are really designed for servers, for data centers, and for these cloud workloads. So as you think about, like, building something like AI or an Azure service or m 365 or something like that, it's not just a software problem anymore. It's a software problem, and the software engineers need to talk to the hardware engineers. The hardware engineers need to talk to the power engineers, the data engineers, the construction engineers, even the sustainability engineers to understand, like, the full footprint of what they need. And so as we build all this new infrastructure, you know, optimizing the power draw of particular servers is important, optimizing the power sources that we use.
Alistair Speirs [00:07:14]:
And we sourced about 34 gigawatts of renewable energy around the world through power purchase agreements. So that's essentially securing renewable energy to continue to run these operations in the in the future. So kind of forward planning that and targeting those those, those contracts are important as well. And then, of course, if you, you know, zero out your carbon emissions from power, what's left is carbon emissions from construction, building materials, server materials as well. So by building our own infrastructure and servers, we're able to improve our recycling rates. We're able to define, like, the types of, components that we use in that device. We can either reduce landfill by reusing these components rather than rather than throwing them out. And then on the data center construction side, we have opportunities for lower carbon intensive building materials as well.
Alistair Speirs [00:08:08]:
And one of the ones that we talked about today was cross laminated timber. And this sounds like a crazy idea, but building a data center out of wood. Okay. And it turns out, like, wood has been used in construction for a long time, but the innovation with cross laminated timber is to essentially build these wider planks of layered timber, that's laminated together, and is essentially much lighter, much easier to work with, and has less, embodied carbon than, say, a concrete or steel construction as well. So having, like, lighter flame lighter frames that are as strong as as a steel or a concrete construction means that you need less concrete for the foundation, less you know, easier to move, less equipment, less logistics costs. And so all of those things add into essentially what we call our scope 3 emissions. So that's the emissions caused by our broader operations as well.
Jordan Wilson [00:09:04]:
Yeah. Y'all are in for a treat because you thought you were tuning in to to to learn about, you know, Azure and and and deep u chips, which we are. But I think the environmental piece is huge because I think sometimes big tech gets a bad rap. Right? They're like, oh, you know, AI, and it's so, you know, costly and inefficient and all these things. But, I mean, here you are just explaining, even going down to the materials that you use to build these centers. That is
Alistair Speirs [00:09:28]:
the system. Right? When we think about the system, it's not a software system. It's not a software plus hardware system. It is an ecosystem that guides down all the way from the construction and then, of course, the water, the power usage, and how that infrastructure runs in its environment, in the built environment, in the natural environment as well.
Jordan Wilson [00:09:47]:
Yeah. Okay. So let's let's maybe take it take it back
Alistair Speirs [00:09:49]:
to the
Jordan Wilson [00:09:50]:
back to the office, back to yeah. So, I mean, what is this ultimately going to mean, number 1? Number 2, when our customer is gonna be able to get their hands on things that were announced today, or did they already? And then, you know, both, I wanna look at it through the scope of, you know, the more technical people a little bit. Right? Even though that's not always our audience, but, you know, what does it mean for people in the IT department for big enterprise companies? And then what does it mean for everyone else? Are are are they gonna be noticing all of a sudden they're working with, you know, everything in Azure is much faster, like overnight, or how is this rollout gonna work for those different groups? Yeah. I mean, the general promise is it just gets better.
Alistair Speirs [00:10:29]:
That's I think the what we're really trying to do here. So as we do these optimizations at kind of a component level and other things, they're sort of out of sight. Right? They're running in a data center somewhere else, so you don't necessarily see the change in, like, architecture or server design or rack design, but you'll see the benefits. You'll see the services being faster, the services being more integrated, the services being more cost effective to run as well. On the other side, being able to control our energy draw, and being more optimized means that essentially we're placing less burden on the grids that we operate in as well. So that has a benefit for for every consumer of energy on a particular energy grid as well.
Jordan Wilson [00:11:13]:
Yeah. The energy thing. Yeah. Because, I think we just shared in our newsletter last week something like there's gonna be 40% of data centers. We're gonna be facing power struggles by 2025. And, you know, here we are at the end of 2024. So this was timely news. Right? Yeah.
Alistair Speirs [00:11:28]:
And, you know, for us, like, long term sourcing of that power is really important to us as well. We wanna make sure we have stable operations for years to come. And so providing, like, a demand signal to our utility providers is really important. And so these long term agreements, you know, some of them 5 years, some of them 10 years, some of them even longer, help us kind of guarantee a certain amount of new energies joining the grid, so that when as we need it, we have the energy there and we don't become like a burden on the overall energy grids themselves.
Jordan Wilson [00:11:59]:
Yeah. And you know what? This is helpful for me too as a nontechnical person. I was I was sitting in the keynote, to today or, you know, by the time that, everyone hears this yesterday. And, you know, everyone was clapping when this came out, and I'm I'm like, okay. But now I understand it so much more, because of its impact, that it makes really just all across, all across the enterprise. But, you know, I'm I'm curious because it's almost like it seems like one of those things that's not like too good to be true, but it's it's it's faster. It's cheaper, it's more secure. So what are some of the challenges? Right? Like, as you guys continue, this this this work and continue the development.
Jordan Wilson [00:12:35]:
Yeah. You know, the
Alistair Speirs [00:12:37]:
the thing with the as we get lower into the stack and the infrastructure, being wrong in software is bad. You have a bug. You gotta fix the bug. You can generally fix software relatively fast. Being wrong in hardware is more expensive. Right? So you wanna catch those bugs earlier in the cycle, preferably before you even made the chip. It's very hard to, you know, replace a chip once it's in production and out there as well. And then being wrong in construction is another problem as well.
Alistair Speirs [00:13:04]:
It's multi year complex infrastructure problem, projects. So building in the wrong place or building in areas that won't have the energy needs, or the energy supply that you need in 5 or 10 years is a challenge as well. So as we get closer to the infrastructure and the physicality of the cloud, being accurate longer term is really important for us. And so that's the constant pressure between, like, what can be fungible? Software can be fungible. Right? We can swap out one software for another software. What can be a little bit more fungible? Hardware. We can swap out one hardware for another type of hardware. It's a little harder, but both use electricity.
Alistair Speirs [00:13:43]:
Mhmm. Then as you get down to, like, space, facilities, square footage, power supply, water usage, all these other things, there's less flexibility there. So we spend a lot more time on making sure we're right and we're in the right places that we expect the demand to be in the future as well.
Jordan Wilson [00:14:03]:
So So you kinda talked about some of the challenges that the rest of us may not see.
Alistair Speirs [00:14:07]:
Mhmm.
Jordan Wilson [00:14:08]:
So what are you all focused on now? Because at least to me, when I see this, I'm like, oh, you know, this checks off all the boxes. You guys have made it. But what are the next challenges, you know, from the Azure side, from the, you know, processing side? What are the next big, challenges that you need to tackle? And maybe not just Microsoft, but the industry at whole. Like, what is the next couple of hurdles that maybe not the things that keep you up at night, but the things that you're still actively working on? Yeah.
Alistair Speirs [00:14:36]:
It's a great question. What keeps me up at night? A couple things are really paramount right now. It's, like, security. How do we build security into every layer of this stack as well? And you can think about software security. We announced today our new HSM, so it's a new hardware security module that will go in all our servers going forward. When you think about network security, perimeter security, quantum technologies are coming as well. And, you know, one of the scary parts of quantum is just how fast it can work and how it can cut through traditional cryptography like butter, which, again, not a good thing. And so building in, like, quantum cryptography protections into our platform now Mhmm.
Alistair Speirs [00:15:21]:
Is really important to us as well. So that security layer of making sure that we can run this trusted, secure platform for all of our customers Mhmm. Is really important for us to focus on. Then secondly, I'd say is is the environmental impact. We set some bold goals around being carbon negative, being water positive, being zero waste. And so all of that really translates to how we build and architect this data center infrastructure and this server infrastructure as well. And part of that is also working with our community, working with our broader supply chain, working with our energy providers and others to integrate into these grids. And as you think about renewable energy, wind, and solar, when the wind's blowing, the sun's shining, like, the power price fluctuates.
Alistair Speirs [00:16:10]:
And unlike a traditional factory, the data center is kind of software defined. So we have fairly granular controls of when we use energy Mhmm. When we sip, when we slurp from the grid, if you like. So being, again, fungible with all of these workloads, being able to move them around that global infrastructure is something that's really interesting as well. It's really take advantage and support kind of renewable energy Yeah. To fossil as well.
Jordan Wilson [00:16:35]:
Yes. I feel we focus a lot on the DPU and maybe scathed over the HSM a little bit. But can you explain a little bit more how does that work and how does it actually make operations more secure? Yeah.
Alistair Speirs [00:16:48]:
Great question. So let me give you the the 101 on, like, cloud cryptography. So HSM, a hardware security module, has traditionally been kind of an appliance, a separate rack that does nothing but generate cryptography keys. So they use for encryption and decryption. These things are generally really large prime numbers. If you want to really dumb it down, you can think of it as like a really expensive random number generator Okay. That will give you kind of uncrackable keys or very hard to crack keys. Sure.
Alistair Speirs [00:17:23]:
And so in that model, you have this separate appliance. You use it when you have something really important to secure. And so you'll talk to this machine, you'll get back your cryptography keys, you'll do the encryption, you move the data around, someone will decrypt it, they'll talk back to that HSM machine as well. As you think about a world where you want more security, more defense in-depth, more layers of this, essentially, in a world where every transaction, every message, every API call, every database read will be encrypted Mhmm. You can have more and more traffic on that. And so moving that encryption decryption off a dedicated device, or maybe still there in a dedicated device for specific use cases, but embedding it in the server allows you to do it much faster. It allows you to even have scenarios where you may not even completely trust all of the components on the server. You'd have transaction between 2 processes on the same server encrypted between the traffic or something like that.
Alistair Speirs [00:18:22]:
And this is called ephemeral key cryptography, where you may be just, encrypting and decrypting transactions that might just last seconds or milliseconds, but you're just essentially creating that secure chain between that transaction for the life of that transaction, but it may be ephemeral. It's there for that transaction, in memory, secure, and then gone away after that. So these sorts of chips support that kind of next level of security that we're really look looking to bring.
Jordan Wilson [00:18:52]:
I feel so much smarter now. So so, Alistair, we I mean, we've covered a lot. We've talked about some of the new announcements today from Microsoft at the Ignite conference here in Chicago. We talked about HSS. We talked about DPUs, cryptography. We've talked about water cooling, the environment, f timber. My gosh. Everything.
Jordan Wilson [00:19:09]:
But, you you know, maybe what is the one most important takeaway that you want the average business leader listening in on what this actually means for them and their business moving forward?
Alistair Speirs [00:19:20]:
Yeah. I think the key takeaway here is that this cloud technology and this AI technology is rapidly becoming more and more optimized. And our vision is that every application, every user, every workload, every business is going to be using these AI capabilities. And to be kind of democratized or commoditized and just be part of the fabric of how we do business is gonna require, like, a new type of infrastructure, a new type of architecture that we're building up. We need to build it out in a way that it will scale to every business, every user, and that's in line with our mission at Microsoft. Right? So, yeah, it's with every user and every, every person on the planet every person or organization on the planet to achieve more. Right? And so that scale out model is really what we're thinking about with all of these decisions across software, hardware, all the way down to physical space planning.
Jordan Wilson [00:20:12]:
Wow. I think that was, an extremely impressive and important way, to wrap up today's show. So hey. Audience, next time you're out there, don't take what's happening, under the hood at Azure for granted. There's a lot going on, and it's only gonna get apparently faster, more secure, and cheaper. So thank you so much for tuning in. Make sure if you haven't already, go to our website at your everydayai.com. Alistair just dropped a whole bunch of knowledge on our heads.
Jordan Wilson [00:20:39]:
We're gonna be breaking it down for you in our newsletter, so make sure you go check that out. Thank you for tuning in, and we hope to see you back tomorrow and every day for more everyday AI. Thanks, y'all.
Alistair Speirs [00:20:51]:
Thanks. Hopefully