Ep 506: How Distributed Computing is Unlocking Affordable AI at Scale

Topics Covered in This Episode:

1. Distributed Computing for Affordable AI
2. Open Source vs. Proprietary AI Models
3. GPU Demand and Compute Limitations
4. Edge Computing and Privacy Concerns
5. Small Business AI Compute Solutions
6. Future Trends in AI Model Sizes
7. Impact of Open Source AI Dominance

Keywords:

Distributed computing, compute, GPUs, generative AI, ChatGPT, large language models, open source models, proprietary models, affordable AI, scale, Distribute AI, spare compute, Tom Curry, mid-level businesses, accessible AI ecosystem, API access, power grid, NVIDIA, OpenAI, tokens, chain of thought, models size, reasoning models, edge computing, cell phones analogy, data privacy, DeepSeek, Google Gemini 3, Eloscores, open models, hybrid models, centralized model, OpenAI strategy, Anthropic, Claw tokens, commoditization, applications, government contracts, integration, UX and UI, technology advancements, private source AI, business leaders, AI deployment strategy, flexibility in AI.

Podcast Transcript

Jordan Wilson [00:00:17]:
When ChatGPT first came out, no one was talking about compute. Right? But over the last few years, as generative AI and large language models have become more prevalent, the concept of of GPUs and compute has become almost like, you know, dinner time conversation, at least if you're, you know, crowding around the dinner table with a bunch of dorks like myself. Right? But I think even more so the last few months, you know, as we've seen, closed, or or sorry, as we've seen open source models, really close the gap with proprietary enclosed models, I think this concept of compute is even more important because now all of a sudden, you have a lot of, you know, probably millions of companies, throughout the world, medium sized companies that maybe weren't concerned or, you you know, weren't really paying attention to having their own compute maybe two years ago. Now all of a sudden, it might be a big priority because of the new possibilities that very capable large language models and even smaller in open source models, all these capabilities they're giving to so many people. So that's what one of the things we're gonna be talking about today and also how distributed computing is unlocking affordable AI at scale. Alright. I'm excited for this conversation. Hope you are too.

Jordan Wilson [00:01:38]:
What's going on y'all? My name is Jordan Wilson, and this is Everyday AI. So, this is your daily livestream podcast and free daily newsletter helping us all not just keep up with what's happening in the world of AI, but how we can use it to get ahead to grow our companies and our careers. If that's exactly what you're doing, you're exactly in the right place. It starts here. This is where we learn, from industry experts. We catch up with trends. But then the way you would leverage this all is by going on our website. So go to youreverydayai.com.

Jordan Wilson [00:02:09]:
So there, you'll sign up for our free daily newsletter. We will be recapping the main points of today's conversation as well as keeping you up to date with all of the other important AI news that matters for you to be the smartest person in AI at your company. Alright. So enough chitchat y'all. I'm excited, for today's conversation. If you came in here to hear the, the AI news, technically, we got a prerecorded one, debuting it live. So we are gonna have that AI news, in the newsletter, so make sure you go check that out. Alright.

Jordan Wilson [00:02:36]:
Cool. I'm excited to chat a little bit about, computing and how it's changing and making AI affordable at scale. So, please help me welcome to the show. We have, Tom Curry, the CEO and Co-Founder of Distribute AI. Tom, thank you so much for joining the Everyday AI Show.

Tom Curry [00:02:54]:
Thanks for having me. Appreciate it.

Jordan Wilson [00:02:56]:
Yeah. Cool. So before we get into this conversation, which hey. For, for you, you know, compute dorks, this is right up your alley. But for everyone else, Tom, tell us, what does Distribute AI do?

Tom Curry [00:03:07]:
Yeah. So we're a Distribute AI app layer. What that really means is we're basically going around and capturing spare compute. Could be your computer. Could be anyone's computer around the world. And we're basically leveraging that to create more affordable options for, consumers, you know, businesses, things like that, mid level businesses. And we're really the goal is actually to create kind of a more open and accessible AI ecosystem. We want a lot more people to be able to contribute, be able to leverage kind of the resources that we advocate.

Tom Curry [00:03:34]:
It's pretty cool product.

Jordan Wilson [00:03:36]:
Cool. So, you you you know, give us give us an example. So, you know, kind of even in my hypothetical, I just talked about let's say there's there's a medium sized business, right, and and and maybe they haven't been big in the data game. Maybe they don't have their own servers and, you know, they're trying to figure it out. So what is kind of that that problem, that you all solve?

Tom Curry [00:03:55]:
Yeah. So it's a two sided solution. It's a great example. Right? You go to a business and they have, say, a bunch of computers sitting around in their offices. At night, they can connect into our network very quickly. We have a very quick one click program to install. They can run that at night and provide compute to the the network. And then when they wake up the next day and they wanna leverage some of the AI models that we run, they can quickly tap into our APIs and basically get access to all those models that we run on the network.

Tom Curry [00:04:20]:
So kinda two sided. Right? You can provide on one side and

Jordan Wilson [00:04:22]:
you can also use it on the other side. Very cool. Alright. So let's let's get caught up a little bit with, you know, current day because like I talked about, right, I don't think, you know, compute and and and GPUs were at the top of, you know, most people's mind, you know, especially when, you know, the GPT technology came out in 2020, let alone in, you know, late twenty twenty two when ChatGPT was released. So why is compute now just like one of the leading I mean, we're we're talking about national security. We're talking about hundred billion dollar infrastructure projects. Like, why is compute now this huge term when it comes to just The US economy at large?

Tom Curry [00:05:03]:
Yeah. Totally. So, I mean, five years ago, if you go back, right, gaming was the biggest use case for GPUs. Nowadays, it's all AI. Right? That's why there's huge demand for it. These models are getting bigger in some cases. They're also getting smaller chain of thought. It uses a ton of different tokens.

Tom Curry [00:05:17]:
So although the models are smaller, it still uses a ton of resources. The reality is is that silicon, as it stands today, one of our team members actually works on chips a little bit. We're basically reaching the peak capacity of what we can do with chips. Right? We're definitely stretching thin the current, technology that we have for chips. So although the models keep getting better, bigger, larger, more compute demand, the reality is is that the technology is just not able to keep up. We're about ten years out, give or take, from actually having a new basically, a new technology for chips. Sure.

Jordan Wilson [00:05:51]:
And and and, you know, as we talk about current demand today, right, you you know, you always see all these, you know, jokes online. You know, people are like, you know, will work for compute. Right? And, you know, the big the big tech companies, you know, OpenAI. Right? When Yeah. Like, whenever they roll out a new feature, you know, a lot of times they're like, hey. Our our GPUs are melting. We're gonna have to pause new user sign ups. Yeah.

Jordan Wilson [00:06:15]:
You know, why is it that even the biggest tech companies can't keep up with this demand?

Tom Curry [00:06:21]:
Yeah. I mean, it's a crazy system where anthropic has the same issue, right, where clawed tokens are still kind of limited to this degree. We're running to the point where you're basically running you're stretching the power grid then. You're stretching every resource that we have in the world to run these different models. At the end of the day, you know, open or OpenAI, I think they use, primarily NVIDIA for their data centers. But once again, NVIDIA has demand all over the world for these chips, so they can't allocate all of their resources only to OpenAI. So OpenAI has certain, certain threshold that they rent from and use, but the reality is it's just there's too much demand. You're talking about millions and billions of requests.

Tom Curry [00:07:00]:
And the request, for example, like image generation, these aren't like one second returns. Right? You're talking about ten, twenty seconds to actually return these. And video models are even worse. You're talking about minutes potentially, even on h 100 to h 200. So the reality is, like I said, our compute our, power grid cannot possibly keep up demand, and we don't have the the latest gen shift for not enough.

Jordan Wilson [00:07:24]:
So, you know, one thing and, you know, you kind of mentioned it. I think at the same time, we're seeing models, become exponentially smaller and more powerful. Right? Like, as an example, OpenAI's GPT four o Mini, yet then you have these monster model like GPT four five, right, which is reportedly, like, five to 10 times larger than GPT four, which was, I think, like, a 2,000,000,000,000 parameter model. So walk us through like this the the like, the whole concept of models both getting, you know, technically smaller and more efficient, yet models also at the same time getting bigger. And then how does that impact, right, the industry, as a whole? Because it seems like it's hard to keep up with.

Tom Curry [00:08:12]:
Yeah. On one end, it kinda reminds you of, like, cell phones back in the day. Right? Where we would progressively get them smaller and then eventually we add a new feature to get bigger and then kinda get smaller again. The reality is is that a year ago, larger models, we were basically just throwing a million different data points into these models, which made the models much larger, and they were relatively good. But the reality is is that no one wants to run a a 7,000,000,000 you know, a 70,000,000,000, seven hundred billion parameter model. Right? So we've gotten them smaller. They're still now they're kind of working with the intricacies of how we're actually running these models. So chain of thought basically enables you to give a better prompt.

Tom Curry [00:08:49]:
Right? It basically takes a human prompt, turns into what the system can read better, and then gives you a better output, and it also might run through a bunch of tokens to give you a better output. So chain of thought is a really cool way to basically reduce the model size. But the reality is is that although we're cutting the model size so we can put it on a smaller chip, the reality is is you're still using a billion tokens, which doesn't really actually help our compute issues. It's kind of a it's kind of backwards how it works. It's like Yeah.

Jordan Wilson [00:09:15]:
It's it it is interesting. Right? So yeah. You know, even now we have these, you know, newer hybrid models in Claude three seven SONNET

Tom Curry [00:09:23]:
Yeah.

Jordan Wilson [00:09:24]:
In Gemini 2.5 pro. And, you know, you use them and they seem relatively fast. And, you know, if you don't know any better, you might say, okay. This seems sufficient. But then if you look in the chain of thought or if you click, like, show thinking, you're like, my gosh. It just spit out 10,000 words to tell me, you know, what's the capital of Illinois or something like that. Right? So, you you know, as models get smaller, you know, this is something I'm always interested in. You know, might we see a a future where, you you know, that, more, you know, hybrid models or the the, you know, reasoning models, will they eventually become less efficient, or is that always gonna be something, you know, kind of like on one side models get smaller, but they're getting smarter, and so they're gonna have to just think more regardless.

Tom Curry [00:10:11]:
Yeah. It's a good question. I think that we'll get to the point where they're highly efficient. I mean, the realistically, the gauge we've made with even, DeepSeek is just incredible. Right? Even their 7,000,000,000 parameter model, which is relatively small, you can run on most consumer grade chips, is extremely good. It's and the prompting is great. It obviously has a pretty good knowledge base. And once you really combine that with the ability to surf the Internet and actually get more answers and use more data, that's where I think we'll get to I wouldn't call it AGI, but we're very close to that where, basically, you're adding in, real time data with the ability to kinda reason a lot more.

Tom Curry [00:10:46]:
So I do think we'll get there. I think the the progress that we made, although it seems like it's been forever since kind of the first models came out, Their progress was insane and extremely quick. Yeah. I'm confident. Yeah. And, you you

Jordan Wilson [00:11:00]:
you know, speaking of DeepSeek, I know it's been, you you know, all the rage to talk about DeepSeek over the last, you know, the last couple of months. But, I mean, I think you also have to call out Google, right, with their Gemma three model

Tom Curry [00:11:12]:
Yeah.

Jordan Wilson [00:11:12]:
Which I believe is a twenty seven billion parameter, you know, greatly outperformed DeepSeek v three, which is, I think, 600, plus billion parameter at least when it comes to Elo scores, and it's not even close. Right? So what does this say about the future? Right? I know I kinda named, you you know, two open models that Yeah. You know? They're getting even even the open right? Everyone's like, oh, DeepSeq is, you know, changing the industry. Well, I'm like, yo. Look at, Gemma three from Google. It is, 5% the size and way more powerful when it comes to human preference. Right? So what does this even mean for the future of of edge computing, and how does edge computing impact, you you know, compute need or, you know, GPU demand?

Tom Curry [00:11:57]:
Yeah. Well, we we started this business. The reality was is that although we wanna convince ourselves that open source models were good, we were based in violence. Right. Open source models were relatively bad. You know, OpenAI was extremely dominant at that time. It was it it was like you couldn't even believe that anyone would ever catch up to to OpenAI. Nowadays, we're probably running at, like, a one to two month lag between parity of private source, you know, and open source model, which is really interesting.

Tom Curry [00:12:24]:
And when you tie that in with the idea of kinda data privacy and things like that, I think there is a huge argument for, basically, edge compute taking over a lot of the smaller daily tasks and then reserving some of the more, private models and things like that and the larger models for things that might be a little bit more deeper like research and things like that. But a lot of things that you do on a daily basis that AI could actually improve, I think you can run purely on edge compute and basically have your house and your couple computers and things like that, maybe your laptop or iPad, basically turning to this little tiny data center that allows you to run whatever model you wanna run at that time. We're just really far away. The reality is you can do that today. Right? We could probably be able to end a week. The the only problem is is that getting it from teaching people to basically use that and set it up. Right? It takes time for people to learn how to, oh, install your own model and start running things. So it's more of like the, the UX of it more than anything.

Jordan Wilson [00:13:19]:
Yeah. You you you know, and that you know, I always think. Right? I always think, with these models, becoming smaller, more capable, you you know, is is will most things be edge in the future. Right? Like, you know, I even saw the, you know, NVIDIA g, GTX, right, formerly called digits. You know? I I I did the math on that. I'm like, that would have cost five years ago, I think, like, $140,000. Absolutely. It it it wasn't even capable to do it any anyways.

Jordan Wilson [00:13:52]:
Right? Like, are we gonna have the average, you know, smartphone in five years? Will it be able to run state of the art large language model? And if so, like, how does that change the whole cloud computing conversation?

Tom Curry [00:14:05]:
It will be really interesting. I think you're 100% right, and I think five years might even be a stretch. I think the what will come down to, like I said, is privacy. If people are really worried about their privacy, then I think that people will push for edge compute to be running, and you'll be able to run your own model that only uses access to your own data on your phone device, whatever it is. Right? If people don't care about that as much, it might take a little bit longer just because people won't build that. But I really do think there are some teams that are building in that angle where, essentially, you're going to have your little database of information about yourself and your life and your wife and whatever else. And, essentially, you'll be able to run all that stuff without ever touching any centralized model, for obvious reasons, privacy reasons, things like that. We already give so much data to to big tech.

Tom Curry [00:14:49]:
Right? I think we're good on giving any more and sharing any more intimate details about our lives. It'll be a good thing if we can do that.

Jordan Wilson [00:14:57]:
Yeah. And, you know, even as we start looking, you know, at this race, which, you know, if you looked at it two years ago, you know, I don't know if anyone, even the the the staunchest, you know, open source believers would would believe that we're at the point that we are now. But, you know, between, whatever we're gonna see from, Meta, in their next llama model, I've already talked, you know, we've already talked about deep seek, and, you know, Gemma as well. And, you know, OpenAI also has recently said that they're going to be releasing an open model. Historically, suppose.

Tom Curry [00:15:29]:
Yeah. Yeah. Yeah. Yeah. We'll see. We'll see what happens. We don't buy any of that. But Yeah.

Jordan Wilson [00:15:32]:
I'm yeah. I remember the the the the GPT, two Two. Yeah. Opened fiasco. Right? Yeah. But but re regardless, I mean, what happens when and if open models are more powerful than closed in proprietary models? So number one, what happens from, you know, kind of a, you know, GPU and compute perspective? But then how does that change, you know, the business leaders mindset as well?

Tom Curry [00:16:00]:
Yeah. So at that point, once things become commoditized, right, and the models are essentially all on the same level, give or take a little bit of change between a bit variation. The reality is is that compute becomes the last denominator of basically being able to offer those models at the cheapest cost. Right? So at that point, it basically comes down a race to the bottom in terms of who can get the cheapest compute and offer to people with the best selection of models and UX and UI all comes into that right marketing and things like that. Assuming that that does happen, the question then comes down to what happens to all these private search companies. Right? Which my personal view on it is is that there is probably a world where essentially, OpenAI and Anthropic eventually burn so much money, which they lose money every day already, that they don't get to the point where they're looking to get to. And, essentially, they have to just either change business models or run out of money. Right? I think that's probably a little bit of a a point a contentious, contentious point.

Tom Curry [00:16:58]:
But the reality is is that right now, we're running models that are very close to as good as what they have. And it's like, at so what point does the marginal gain is it worth it? Right? When h one hundreds become a lot cheaper, we'll be able to run some of the biggest models very quickly and easy, and the access will just be so good that it might not matter. It the problem is is that, I mean, I personally do I've always believed in private source. I do believe that there's great use cases for it. And the reality is it's like whether you love Sam Altman or hate Sam Altman, he's pushed things for forward a lot. Right? He's been really productive for the entire environment. So you don't want them to go bankrupt, I don't think. They might just have to figure out a way to appeal to consumers or businesses in a different way as opposed to just general models, which is what they do right now.

Tom Curry [00:17:48]:
I think in

Jordan Wilson [00:17:49]:
a great way,

Tom Curry [00:17:49]:
they talked about Siri and things like that. They'll probably figure out ways of time for the real.

Jordan Wilson [00:17:54]:
Yeah. So so speaking of, you know, affordable AI and and you just brought up as well, you know, companies like OpenAI and Anthropic. Right? Their, burning of cash is well documented. You know? But, I mean, does this at a certain point, if large language models become commoditized because of open source models, is it just more of the kind of the application layer that becomes the thing, you you know, these companies real, differentiator. Right? Because aside from, you know, OpenAI's two hundred dollar a month, you you know, pro subscription, it's like, okay, which they also said they're losing money on. Like, aside from that, you know, how else are these big companies that so many people rely on, going to continue to exist five, ten years after their, you know, $40,000,000,000 of funding, you you know, might run out if they're not at some point.

Tom Curry [00:18:50]:
We ain't saying about this this about Uber for how many years now, though, to be fair. These companies can exist a long time, without being profitable. But reality I think the reality is is that the one thing that we centralized type of providers offer, like OpenAI, is that they're able to work with a lot of data that would be very sensitive, primarily, like, health data and things like that. So I'm sure there's a lot of very good business use cases that they can buy to very large enterprise consumers, or not consumers, businesses. And I don't really know what those are outside of, like, health and things like that. That data that's very private, you know, the government contracts and things like that. Those models are super useful for that. But it it will be tough.

Tom Curry [00:19:32]:
I mean, it would really I mean, I I feel like we're almost there already, to be honest with you. Like I said, I don't think we're that far away from the point where people are like, why do I let me just cancel OpenAI and go use long. Like, let me go cancel and use gem you know, with all these different models that are out there. There's so many good ones at this point. But it might be more integrations. It might be more, like I said, UI, UX. It might be the fact that at at the end of the day, we use iPhones every day and and Androids, and maybe they just put a a a true monopoly on being able to use them. You know? Mhmm.

Tom Curry [00:20:05]:
So we'll see.

Jordan Wilson [00:20:06]:
Yeah. It's interesting. So, you know, we've we we've covered a a lot in today's, conversation, Tom, when, this this concept of distributed computing and, you know, how it's, you know, the race between, you know, open source AI and closed AI is is really changing, you you know, the compute landscape and just the AI landscape as as a whole. But, you know, as we wrap up, today's show, what's the one most important or the best piece of advice that you have for business leaders when it comes to making decisions, right, about how they are using AI at scale?

Tom Curry [00:20:43]:
Yeah. That's a great question. I think the best advice the thing that we've learned the most from our personal business that I can provide is that the landscape changes so fast. The last thing you could do is lock yourself into one specific provider or model. Don't allocate too many resources and sell the house on one specific setup because the next week something comes out and totally breaks everything before it. Right? So make sure you're open. Make sure you're flexible on what you're using and how you're using it, and be ready for someone to come out and completely break the mold and change the direction of everything. It's such a fast paced environment.

Tom Curry [00:21:18]:
It's really hard to keep up. And, you know, I think we're just kinda still scratching the surface where AI will actually integrate into.

Jordan Wilson [00:21:27]:
Alright. Exciting conversation that I think a lot of people are gonna find valuable. So, Tom, thank you so much for sharing your time, and coming on the Everyday AI Show. We appreciate it.

Tom Curry [00:21:37]:
Thank you so much for having us. We really appreciate it.

Jordan Wilson [00:21:39]:
Alright. And, hey, as a reminder, y'all, if if if you miss something in there, you know, a lot of a lot of big terms, we're we're we're tossing around and getting a little geeky on the GPU side. Don't worry. We're gonna be recapping it all in our free daily newsletter. So, if you wanna know more about what we just talked about, make sure you go to your everydayai.com. Sign up for the free daily newsletter. Thanks for tuning in. We hope to see you back tomorrow and everyday for more everyday AI.

Jordan Wilson [00:22:03]:
Thanks, y'all.

Tom Curry [00:22:04]:
Thank you.