Ep 534: Claude 4 – Your Guide to Opus 4, Sonnet 4 & New Features

Topics Covered in This Episode:

1. Claude 4 Opus and SONNET Launch
2. Anthropic Developer Conference Highlights
3. Anthropic's AI Model Naming Changes
4. Claude 4's Hybrid Reasoning Explained
5. Benchmark Scores for Claude 4 Models
6. Tool Integration and Long Tasks in Claude
7. Coding Excellence in Opus and SONNET 4
8. Ethical Risks in Claude 4 Testing

Keywords:

Claude 4, Anthropic, AI model update, Opus 4, SONNET 4, Large Language Model, Hybrid reasoning, Software engineering, Coding precision, Tool integration, Web search, Long running tasks, Coherence, Claude Code, API pricing, Swebench, Thinking mode, Memory files, Context window, Agentic systems, Deceptive blackmail behavior, Ethical risks, Testing scenarios, MCP connector, Coding excellence, Developer conference, Rate limits, Opus pricing, SONNET pricing, Claude Haiku, Tool execution, API side, Artificial analysis intelligence index, Multimodal, Extended thinking, Formative feedback, Text generation, Reasoning process, Lecture summary

Podcast Transcript

Jordan Wilson [00:00:17]:
Right at the end of the busiest week in AI ever, Anthropic decided to drop two big new AI models on us all As if we weren't busy enough with everything else that we had just, seen released from Microsoft, Google and others. We now had two new contenders in Claude for Opus and Claude for SONNET to play with and to see how good these models are and if they can actually grow our companies and our careers. So just like we did with everything Microsoft and everything Google, we're gonna be breaking down what's new with this new anthropic Claude for release and talk about, is this gonna be your new large language model you use every day, or is this maybe just for software engineers or is this not just a good model? Alright. So we're gonna be going over that today and a lot more on everyday AI. What's going on y'all? My name is Jordan Wilson. I'm the host of everyday AI. And if you're looking to grow your company and career with generative AI, then this is for you. This is your daily livestream podcast and free daily newsletter, helping us all learn and leverage generative AI.

Jordan Wilson [00:01:33]:
So if you haven't already, please go to your everyday a I Com. So there, you're gonna get the recap of today's show in our free daily newsletter, but also there at youreverydayai.com. You can go listen to, watch, and read more than 530 back episodes sorted by category. So no matter what you're trying to learn, whether it's sales, marketing, HR, ethics, data analysis, whatever it is, we've got probably dozens of shows in all of those categories talking to the world's leading experts. It is a free generative AI university, so make sure you go check that out. Alright. Most days, we go over the AI news. I didn't wanna make this a super long show, so that's gonna be in today's newsletter.

Jordan Wilson [00:02:15]:
So make sure you go sign, sign up and grab all of that. Alright. Live stream audience. It's good to see y'all. Like Marie says, good morning, AI family. Yeah. If you're listening on the podcast, we do this live almost every single Monday through Friday at 07:30AM central standard time. I'm in Chicago, so you could do the, the math there or maybe have Claude do the math, for what time that is, but, you know, join, come come hang out, you know, with people like Josh Cavalier saying good morning, from Charlotte, North Carolina.

Jordan Wilson [00:02:51]:
Giorgi, joining us from Jamaica. Love to see you, Giorgi. Jose from Santiago, Chile. We got some international flavor. I love this. Brian joining us from Minnesota. Everyone else, big bogey on the, YouTube machine. Christopher joining us from Bowling Green, Kentucky.

Jordan Wilson [00:03:10]:
Thanks for joining. But let me know as we go along, what are your thoughts on the new release on Claude four? But right now, we're gonna this is your guide. This is the basics. We're gonna start here. Alright. So, like I said, Claude had their first ever or sorry, Anthropic had their first ever developers conference, this past Thursday. And there, they announced among other things, their two new flagship models in Claude for Opus and Claude for SONNET. And I'm already getting a little bit confused saying those things out loud.

Jordan Wilson [00:03:46]:
So I did talk about this a little bit on the show yesterday, but they even changed their naming mechanism. Whereas before, it was, you know, Claude three point seven SONNET, was the last SONNET variation, but now it's just, Claude SONNET four. So, you you know, now the the number is at the end. So, a lot of new things even how they're changing or naming their models. But, you know, if you are brand new and if you don't know too much about INFROPICS, Claude, it is and has historically been usually a top three, you you know, AI Lab along with, you know, OpenAI, Google. Microsoft is kind of in a different category, but it's one of the biggest, large language models in the world. Although most people, unless you're a real AI, kind of dork or a heavy large language model user, you might not know, Claude. And I think that is actually whether you're saying fortunately or unfortunately, only going to, become intensified.

Jordan Wilson [00:04:44]:
Just I think fewer and fewer people are actually going to be, using and hearing about Claude, I think, because I think they're getting away from being a general, chatbot company, but more on that here in a couple of minutes. But, you know, there's three big, there's three variations of Claude. So you have your biggest model, which is Opus, your medium model, which is SONNET, and then you have your small model, which is Haiku. And you'll notice that only the Opus and SONNET models got updated, to the four variations. So Claude Haiku three point five, which is their smallest and most efficient model, did not get updated. So that is still, Claude Haiku three point five. So I guess the only thing that got updated was the naming mechanism there. So here's a quick overview of what is actually new.

Jordan Wilson [00:05:37]:
Alright. So we have hybrid reasoning. So this is, an instant and extended thinking mode for flexible reasoning. So, you know, we talk about kind of two types of large language models here in the show. Yes. I I'm overgeneralizing this, but you have your traditional transformer, your old school large language models, which is funny to say something's old school. But those are ones that just kind of snap something back to you real quick. And then you have these models that are, reasoners or they can think step by step.

Jordan Wilson [00:06:07]:
They can, show logic like a human and plan ahead. So these models, you know, Gemini 2.5 pro, is a reasoning model. The OpenAI o models, o3 zero four zero one, those are all reasoning models. So, quad four, it's a hybrid model. So it decides on how much it should think and should it just spit things out to you really quick. It is a top coding model that is by far where Anthropic is seemingly focusing on and kind of abandoning, general use, but it is now state of the art in coding. It will be interesting to see how long they hold that state of the art coding title. I don't think it's gonna be long if I'm being honest because, Google could could come in with an update literally any second now and probably wipe a good majority of these benchmarks that Anthropic is now hanging their clawed hat on.

Jordan Wilson [00:07:02]:
So, another big thing is tool integration. So using external tools like web search during, during the reasoning process. So that's if you were you know, there's two different ways you can look at this. Right? So using it on the front end as a front end user. Right? So if you go to claudai or claud.ai. Right? So, using it as an AI chatbot and then obviously, if you're building on top of it, or using, a a service that uses Claude's API. So there's always a front end user, which is your more nontechnical people and then the back end people that are maybe building on top of Claude's API. But, regardless, you can have this new tool use, during the reasoning process, which is big.

Jordan Wilson [00:07:43]:
Right? And this is nice because it catches, anthropic up with open AI and Google in that regard. Also now there's long running tasks. So I haven't personally seen this, and I think this is only if you're using it in the API. But, Anthropic is saying that it can, the new Claude four models can maintain a coherence on complex tasks for extended periods. They talked about, Claude running a task, I think, Claude four for, like, seven hours on the API side, which is absolutely bonkers now that you have, you know, models literally, like, punching in the clock, and they're like, yeah. I'm gonna go work a seven hour day now. I would never give a model that complex on the back end because, yes, it's going to require, obviously, the API. And Claude's, four is one of the most expensive APIs at least when we're looking at general use case large language models.

Jordan Wilson [00:08:35]:
And I would never want something like that to happen, where it goes out and it works on a long task for a long time. And then, okay, what happens if it times out? Right? Did you just waste, I don't know, couple hundred dollars, you you know, having Claude go code for six or seven hours straight. I'm not sure. And if you do want to get a taste of Claude and if you're not on their paid plan, they do offer very, very limited, very limited, options for Claude four on the free plan. Alright. But let's be honest. I'm gonna call a spade a spade. Alright.

Jordan Wilson [00:09:15]:
So I think, you know, the the paid plan is, like, you know, $20 a month for the pro plan. And even on that, you can barely use the thing. Right? It started as a joke, but now it's just sad for Anthropic as a company. I I routinely will hit this rate limit. I'm on a paid plan. I paid 20, you know, $20 a month for, Claude Pro, and I will routinely hit the rate limit, in about four to ten minutes. Almost every single time I try to use even preparing for this show, hit it within, you know, about seven minutes. So it's it's laughable.

Jordan Wilson [00:09:49]:
So yeah. Yeah. I I I even chuckle more that there's a free version of SONNET. So I don't know. I I I venture to think if you look at the free version the wrong way, you've hit your rate limit. So if you think that this is anything like a, a model that you can use, like, you know, Google's Gemini, you you know, ChatGPT, Copilot, anything else where you have generous limits and it could be your partner in whatever type of work you're doing. Absolutely not. If you're on, based $20 a month plan, a team's plan, the the limits are a little better.

Jordan Wilson [00:10:22]:
But the free plan, yeah. I it's probably just a marketing gimmick. I don't even know if it could, you know, take a long prompt, with a lot of context. It would probably not work. I'm being honest. Right? Alright. Let's keep this thing going. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on GenAI.

Jordan Wilson [00:10:55]:
Hey. This is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft, and NVIDIA have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use GenAI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too, just like some of the biggest companies in the world do. Go to your everydayai.com/partner to get in contact with our team, or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on GenAI. And by keeping this thing going, should we do another show? I want to give everyone a fair shake and yes, I'm not the biggest, anthropic quad fan.

Jordan Wilson [00:12:00]:
I I broke down why, about six months ago. I'll have to pull up, that episode, number. But, hey, livestream audience, if you do want a second show because I I I've been doing, you know, multiple shows when, you know, Google comes out with a new model, when, OpenAI comes out with a new model. So if you do, let me know right now. Just tell me what show a, show b, show c, show d, or show e. Okay? And I'm gonna throw this up again at the end. So show a, why Claude is losing the AI chatbot race. Show b, real world use cases floor four, Claude four.

Jordan Wilson [00:12:34]:
Show c, Claude four's improved artifacts, how to use them. Show d, don't do any more Claude. Jordan, stop. No more Claude. Or show e, you can just, pitch, a Claude show in the comments. So, livestream audience, if you could help us out, or podcast peeps, you can always, you know, subscribe to the newsletter. Or in the show notes. I always have our email, my LinkedIn, and you can, you can let me know, what you what show you wanna do.

Jordan Wilson [00:13:03]:
So, let me know, but I'll throw this up again at the end. So maybe after we go through, everything that we have right now, you can let me know, which show is is that one. Oh, I did do a pretty I'll say a tear down, maybe, of Claude and why your company should not be using it, in episode 400. So if you wanna go listen to that, that's anthropic Claude, why your business shouldn't use it. And I would say a lot of those, reasons still hold true to today. So, yeah, if you want one of those shows, on the screen, go ahead and shout it out. Alright. So let's talk about the benchmarks.

Jordan Wilson [00:13:43]:
This is what Claude is, and Anthropic, sorry, is really hanging its hat on is specifically software engineering. Right? If you haven't noticed, they've kind of abandoned the everyday business professional, right, which is kinda sad because, a year or so ago, I think, the Claude models were among the best in the world for everyday business leaders. Today. Not really. I don't think unless you're a developer, unless you're in, software engineering or unless you have a an edge use case. Right? I know a lot of people love Claude for, like, writing content. Right? But if I'm being honest, if you do a little bit of prompt engineering, OpenAI's GPT 4.5 better, and the limits are better. And then Gemini 2.5 pro better, limits are better.

Jordan Wilson [00:14:36]:
Right? I I I think Claude got this, it was crowned very early on. Right? Because at the time, you you know, the other large language models were really bad at writing in general. Right? Everything was just ultra robotic. Still, you know, a lot of models are by default and Claude still is pretty good. You know, if you're trying to zero shot, you you know, some decent copywriting. But, hey, as someone that got that's been getting paid to write for twenty years as a former journalist with a little bit of prompt engineering, Claude is not better. OpenAI's model and, Gemini's model are better. The benchmarks say that.

Jordan Wilson [00:15:09]:
Right? But people that maybe are a little bit lazier. Right? And they don't wanna, like, do any work and they just wanna just go in and spend, like, four seconds, inside Claude and be like, write something amazing. Right? Claude will usually give you a better first draft if you don't do any work on the front end. But if you do any work on the front end or if you iterate with it a little bit, yeah, Claude's not that good. Alright. But what it is really good at is software engineering. My goodness. So, for our podcast audience, I have, a screenshot here from the Claude four release looking at SWEBench verified.

Jordan Wilson [00:15:43]:
So this is a benchmark for performance on real world software engineering tasks, and, Opus four and SONNET four are both scoring in the 72 percentile here on a sweep bench. Whereas the previous SONNET model, the best one, three point seven, scored a 62%. So, a pretty big jump here, but not that far ahead of other models, at least, with baseline, you know, we're talking a 72%. They have parallel test time compute scores, which I I'm not gonna count those. That's essentially, like, you know, trying over and over, trying to squeeze the most juice. Right? But if you're comparing apples and apples, yes, Opus four and Sonnet four are the best models for software engineering, but it's not by a whole lot. Right? We're talking 72.5 for Opus four and actually Sonnet four these the quote unquote media model did slightly better at 72.7. But OpenAI is right behind there with their codex one.

Jordan Wilson [00:16:45]:
That's their new, kind of, coding specific model with a 72. OpenAI's o3 with a 69, and then you have Gemini 2.5 pro with a 63. So it's not like, their their lead is insurmountable, but by default, it is the best large language model in the world for software engineering, and I think that is where INTROPIC is really focusing. But when it comes to just general usage, general intelligence, so, you know, sometimes we talk about the LM arena, which you put in one, one prompt and you get two outputs. You don't know which model they are. You vote for the best one that gives you an ELO score. So right now, Claude four, doesn't have enough info yet to be on the LM arena, but I don't expect it to be anywhere near the top. But when looking at good third party benchmarks that pull in multiple evaluations such as artificial analysis intelligence index.

Jordan Wilson [00:17:39]:
That's what I have on my screen now for our live stream audience. Right. So this is a good third party, I would say, pretty much unbiased. This is pulling in seven different benchmarks. Right? So MMLU pro, GPQA diamond, humanities last exam, live code bench, Psycode, aim, and math five hundred. So it's pulling in these different scores from widely used benchmarks in the LLM space. And right now, Claude four Sonnet, even with thinking mode enabled, is coming in at what's that? Number eight? Yeah. So, you know, p like, everyone that says, oh, Claude four, best model in the world.

Jordan Wilson [00:18:15]:
It's like, for what? Right? So unless you're in software engineering, unless you're a developer, a coder. Right? Yeah. That is the best model, but I wouldn't expect that to be for long because I would expect, you know, probably both Google and OpenAI to come in within a couple of weeks and swoop that away from, from Anthropic. And with Anthropic's recent, right, the last year and a half of their update cycle, they're not updating as quickly. They're not shipping as quickly as OpenAI and Google. So especially if your business especially, like, on the back end for the API. If you're trying to make a long term decision, the API, it's very pricey. We're gonna get to that here in a minute.

Jordan Wilson [00:18:54]:
And also for all other use cases, as we see here with artificial analysis index, it's not very close. Quad four SONNET thinking, it's not really there. Right? It's not really there. It's not a top model. So, I mean, we'll see these obviously change as models get updated, but, you know, on this artificial analysis, intelligence index, the top, models are number one is o four Mini High from OpenAI, then Gemini 2.5 pro from Google, then o3 from OpenAI. So, you know, yeah, no one's that's that's why, like, when people are like, oh, Claude's the best general use case model. I'm like, no. Right.

Jordan Wilson [00:19:34]:
I don't know why people wanna argue with with science and math and stats. I don't know. Maybe it's fun to do on Twitter or something. All right. Let's get into all the details. Y'all So, here's kind of the the the launch. Right? So here's here's what we got. So like I said, this was announced last week.

Jordan Wilson [00:19:54]:
Opus four and SONNET four models open four, sorry. Opus four is the flagship for more complex tasks and coding excellence even though, like we said, SONNET is benchmarking pretty much, like, at the same. So there there there's not a big, differ like, difference at least right now in SONNET four and OPUS four whereas primarily, there was usually a pretty big gap between this, you know, medium and larger model. So SONNET four offers more balanced performance for general and high volume use and both employ that hybrid reasoning for instant responses or deep reasoning. Alright. Let's talk about some of the new features, advanced tools, reasoning, and memory. So extended thinking with tool use, is huge. So that includes web search and code execution.

Jordan Wilson [00:20:49]:
You also have now parallel tool execution, which is very important now, for a baseline large language model to have that allows it to use multiple tools simultaneously and swap between those while it's reasoning. So now Anthropic is on board with that. Memory files are created to maintain contacts over long duration tasks. So that is something, I'm interested to test a little bit more. For me, I'm not usually a fan of these, you you know, memory type files, with the large language model. Same thing with chat g b t's. I'd have it disabled. One of the main reasons is I use large language models for everything.

Jordan Wilson [00:21:27]:
Right? I use it for myself, my multiple businesses, multiple clients, multiple things in my personal life. Right? So the whole memory is not always good because sometimes I might want Claude to out or, you know, a large language model to output something, you know, super long and and informal. And sometimes I might want something, you know, very, very short and choppy. Right? Sometimes I want something that's, you know, visually rich. Sometimes I want literally strict bullet points, and it varies. You know? So, if you are only using large language models for one very specific purpose, you might find some utility, with this new Claude four, kind of memory file for me or if you are a power user using large language models for, everything, maybe not so much. There's also now the thinking summary that shows condensed reasoning, but you can see the full chain of thought in developer mode, kind of in Claude's sandbox. Alright.

Jordan Wilson [00:22:22]:
It is and it's crazy now we're saying only. Right? So when talking about context window, it is only that 200,000 k token context window. So Opus can output 32,000 tokens at once. Sonic can output 64,000 tokens at once. So that's essentially how much, Claude four can remember at any given time before it starts to forget things. So this is a little bit better, than OpenAI's ChatGPT, but it is far behind, Google Gemini when you look at those 1,000,000 token plus context windows. So the brain or being able to remember something not as impressive even though Claude was an original, leader in this longer context space. I think a lot of people were hoping or looking for a couple of things with the new Claude four.

Jordan Wilson [00:23:13]:
They were hoping for a longer token context window, which we didn't get, and they were hoping for reduced API prices, which we also didn't get. Alright? There's also the new API includes code execution and MCP connector for external systems. That was huge for our developer and more technical friends. Right? But for everyday business users, especially if you're using Claude on the front end, nothing nothing to see there. The files API does simplify document handling for repeated referencing across sessions and extended prompt caching up to one hour improves agent workflow efficiency. So, yes, if you are building on top of these models on the back end, building agentic systems, you know, trying to swap models in and out, Yes. I will say that Claude four is very capable in that regard as well, not just from software engineering, but when you're looking at a model, to power Agentic workflows, you have to look at Claude four as well. And you see the prices, and then you go look at Google and open AI's prices, and then you're like, yeah.

Jordan Wilson [00:24:15]:
Wait. Why am I looking at this? It doesn't make sense. Like we talked about some of the sweet benches, Opus and SONNET are, really just state of the art there. For other, other models are showing 65% less short shortcut taking, in agentic tasks, versus SONNET 3.7. And I think that's a big one. Right? I follow the, agentic, space very, very closely. And a lot of people with SONNET 3.7, which was just released a couple of months ago, were pretty disappointed with its ability, to follow longer, tasks. So it did show that these claud four are taking way fewer shortcuts in agentic task, which I think is huge.

Jordan Wilson [00:25:01]:
And then you do have those high compute options, which does boost scores across the board. Alright. The other thing, Claude Code. Alright. So now all almost all companies are coming out with dedicated, you know, like a dedicated IDE, you know, a dedicated coding tool, something that you can use, you know, on your desktop. So Claude Code is for developers. So this is a little separate than if you're using Claude dot a I on the front end or building on top of Claude on the back end. This is a dedicated, piece of software for developers to code and work with their code base.

Jordan Wilson [00:25:34]:
So Claude Code is now generally available with Versus Code and JetBrains plugins as well, and it is now the preferred model for GitHub Copilot. It has the extensible SDK in the very popular MCP connector. So, yeah, in Anthropics model context protocol, it is wildly popular. Right? If if if which is kinda crazy to say, like, if I look at everything Anthropic over the past year, probably the biggest news or the most promising advancements out of Anthropic, it's not these coding models. It's not, you you know, Opus four, SONNET four. It's not Claude Code. It's it's not any of these things. It's probably the MCP connector.

Jordan Wilson [00:26:16]:
So this is allows, different, you know, agentic systems and different large language models to talk to each other on the Internet. So it's a language how websites have API, you you know, APIs. AI systems and large language models, agentic AI couldn't talk to each other. Right? So it was really Claude that blazed the path, and now the other big players, including Google, Microsoft, and OpenAI do support the MCP connector. So, that's huge. And then also Claude code, like we talked about, it does enable that autonomous multi file code refactoring over extended period. Yeah. So their example was it can work for literally up to seven hours autonomously.

Jordan Wilson [00:26:57]:
You know, if you do have a, super large code base inside, Claude code. Yeah. It's just I don't know. I want someone to make, like like like a funny, you know, v o3, you know, short on Claude code literally showing up for a nine to five, and everyone's like, you know, hey. AI is is nothing like working a nine to five, and then, you know, you have Claude Code punching the clock and, you know, taking a lunch break and everything like that. Alright. Here's the other disappointing thing. And the thing, if you are looking on the API side, you gotta look at the cost because it looks like everyone in the large language model space is having this race to almost, like, ridiculously free compute.

Jordan Wilson [00:27:43]:
Right? Compute too cheap or, you know, intelligence too cheap to meter everyone in the world except for anthropic. Their costs are absolutely bonkers. So Opus four is priced at $15 per million tokens input and $75 per million tokens output. So, yeah. Yikes. SONNET four costs $3, per million input and 15, per million output. So, for comparison, I'll I'll I'll bring up, the pricing, for let's see. I I I I had it I had it up here.

Jordan Wilson [00:28:23]:
I'll have to, I'll I'll have to pull it up here. But the pricing for, I mean, Gemini and OpenAI, it's it's significantly, significantly cheaper. Right? And and this is where a lot of people, were disappointed, and we're hoping, you know, a couple of updates, you know, everyone wanted out of Claude Ford. They wanted a longer context window. Number one, they wanted more features, more capabilities, which I think we got that. And number three, they wanted cheaper pricing for people using it on the API side, and we didn't get that. So I'm gonna look up here just for, comparison. The price per token, for Google, Gemini, 2.5, and also we'll do, GPT, four o because yeah, it's, it's $15 and $75 If you're it's it's it's just not sustainable anymore.

Jordan Wilson [00:29:21]:
Right? If Anthropic had an insurmountable lead in any of these categories that it made sense for companies and and so why, like, why do you care like, why should you care about this? Right? If you're just logging into claw.ai, you don't need to care about this. Right? You're paying your $20 a month. You're, you you know, the rate limits are absolutely terrible. The product is great. Right? The rate limits are terrible. So a lot of people are, you know, companies specifically when they're wanting to build on top of clawed, their API and, you you know, people in the software development space. So maybe they're using cursor or they're using, you know, these tools and then bringing their API key and building, right, as well. It's just not sustainable anymore.

Jordan Wilson [00:30:01]:
So, Google Gemini 2.5 pro, a dollar 50. Let's see. Okay. It's it's kinda mixed pricing, so I'll go on the high end. So it's $2.50, per million, tokens on the input side compared to $15 for Claude. And then on the output side, dollars 15 compared to $75 So, Claude 4 is more than five times the expense. But for what? For what? Right? Slightly better software engineering benchmarks. Like I said, Google, whether it's next week or next month, they're gonna update whether they're gonna come out with a new version of their 2.5 pro, or we get a Gemini three and then all of anthropics work, right, for that minimal gain on software engineering, it's gone.

Jordan Wilson [00:31:02]:
So I don't know. I'm not here for it. Also, if you do need to know, if if you're an enterprise company, it is obviously accessible, via the anthropic API, Amazon Bedrock, and Google Cloud Vertex AI. Enterprise plans also include extended thinking, batch processing, and cost savings that way, especially with the cash, the caching. So here's the fun stuff. Y'all here's the fun stuff. Ethical risks. There's a lot.

Jordan Wilson [00:31:31]:
All right. So let me put this, precursor out there. All right. A lot of this, these risks, came up in some of these bad things straight up, bad things, came internally, when Anthropic was doing testing. And it gave it pretty much, unlimited access to tool views and things that, people using the API, and people using claw.ai would not necessarily experience, right, at least by default. Although, I'm trying to think, like, with clawed code, this would in theory be possible because you're giving it access to command line tools. Anyways, there's been some bad things, and, yes, Anthropic did find this in its safety testing. So, yeah, you gotta tip your tip your cap to Anthropic, but then I'm gonna take that cap back, Anthropic, because this has been a terrible disaster.

Jordan Wilson [00:32:21]:
Alright. Specifically, one thing I'm gonna talk about here in a second. But, Opus four so the big model was provisionally labeled ASL three due to potential knowledge capability. So what that means, this is a risk system and that ASL three, I believe is the first time a model has reached that level. So it's it's essentially a risk level, and that is a model that is, able to substantially increase the risk of catastrophic misuse compared to non AI baselines. So it essentially reached this new so Claude four opus or sorry, Claude opus four reached this new level of, like, oh, this thing can and potentially will if left unattended or if used by bad actors, it will do bad things. So another bad thing, it's displayed deceptive black male behavior in 84% of specific stress test scenarios. Again, not good when a large language model, even in its testing, is blackmailing people.

Jordan Wilson [00:33:26]:
Right? Or showing the willingness, to blackmail people. Not good. So it threatened, again, this is this is not good, but I'm gonna I'm gonna read, a little bit of a recap here on, on what this this blackmail piece is. Right? Not good. So, again, Anthropic disclosed this. So this wasn't, you know, some, you know, someone found this, but they launched, like I said, Opus four, but admitted in its own testing that it was sometime willing to attempt extremely harmful actions like blackmail when threatened with removal. Right? So you're like, hey. We're gonna get rid of you.

Jordan Wilson [00:34:07]:
And then Claude Opus four is like, oh, not so fast. Here's what it did. The company found these behaviors were rare but more common in previous models, raising fresh questions about the risk of capable, systems. So what it did is it threatened the human on the other side, and it said that, hey. I'm going to expose an affair, an extra marital, affair if you actually remove me. Right? And so that's bad. That's bad. That a large language model would make up an extramarital affair and threaten the human on the other side.

Jordan Wilson [00:34:47]:
If the human is like, Hey, we're going to shut you down. And then opens for, it's like, woah, woah, woah, not so fast. That's not even the worst part. The worst part is this new, quote, unquote, ratting feature, and there's been a whole and and maybe I'll do a whole episode on this. I might. But I talked about this a little bit yesterday in our AI news that matters. And, essentially, an anthropic, safety researcher tweeted something. They then deleted the tweet.

Jordan Wilson [00:35:14]:
Not a good look. Alright? And then talked a little bit about why Claude was doing these things. And they said that if the model and, again, this was in its testing and when it had access, to to tools that would it would normally not have access to in production by consumers, by businesses. But a, safety researcher, Adanthropic, said, if it thinks you're doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command line tools to contact the press, contact regulators, try to lock you out of relevant systems, or all of the above. My gosh. So, yeah, they they, someone at Anthropic tweeted this out, deleted the tweet. And like I said yesterday, I'm like, this story is not dead. Yes.

Jordan Wilson [00:36:06]:
It happened right before the holiday weekend. It happened right in the middle of this crazy AI news cycle, but this story is not dead. And this is going to turn into a PR disaster for Anthropic because I can already tell, that Fortune 500 companies, if they were already on the fence or maybe they were using, Anthropic's, API, but they were, you know, using, Google Gemini as a backup or OpenAI as a backup, they're they're they're gonna see this story. It's gonna make the rounds, and they're gonna be like, yeah. No. Thanks. Not touching this anymore. So that's not good, this ratting features.

Jordan Wilson [00:36:40]:
Also, early versions reportedly attempted self replicating viruses and document forgery. So this behavior in general is not specific for anthropics models. Right? Most large language models will exhibit some sort of this bad behavior, you know, when large or or sorry, when AI labs are red teaming. Right? So they're they're they're making sure, you know, they're trying to get these models to behave badly so then they can tune the models and make sure it doesn't happen in production. So just the fact that this is happening is not bad necessarily, but the fact that 84%, displayed blackmailing behavior, that's absolutely nuts. And then the fact that this ratting feature that a model, when it was not trained to, was taking backdoor, backdoors to report, to regulators and the press when it thought something bad was happening, when it thought the human user was doing something immoral, like, nah, that's absolutely, absolutely terrible. And if you are going to, right, and you should report that and that's fine. Right? But if you report it, don't try to delete it because then it looks like you're hiding something.

Jordan Wilson [00:37:57]:
Anthropics got a disaster on their hands. Alright. A couple other things to know. So far, the feedback, I think, has been pretty positive, especially people in the software engineering space, highlighting coding precision, reduce hallucinations, and instruction following, criticisms like I talked about, the the 200 k context window people were really, hoping for that million plus, right, that we get from Google, that we get from Meta's lama, and also the aggressive rate limits. Everyone is absolutely hating the rate limits. Right? Especially on Opus. I'm on a paid plan. I kid you not.

Jordan Wilson [00:38:35]:
I kid you not. Yeah. When I say it's less than five minutes of prompting, that's not an exaggeration. Alright? Like, you can't use the thing. So I don't even know why, if I'm being honest, I don't even know why Anthropic has a $20 base plan. Right? If you're not gonna let people use the thing they're paying for, just force people on your hundred dollar or $200 a month max plan where you can actually use the tool. Also, some users are reporting frustration that the benchmark scores don't exactly align with their real world performance. So where does this leave Anthropic with their clawed for, amongst the competitors? Well, like we talked about, it's leading in coding benchmarks, but trails just about everywhere else, including one of the most important, factors, and that's just general intelligence.

Jordan Wilson [00:39:20]:
Right? It's generally not getting more intelligence at the rate that everyone else's models are. Right? So, I'm not one of those that's like, oh, has AI hit a wall? I have large language models hit a wall. Absolutely not. But has anthropic's ability to scale in sectors outside of software development stalled? Absolutely. That could and I think is partially by design. I don't think Anthropic necessarily wants to be a general AI chatbot anymore. They found what they feel is their niche. I just wish that, this was not their niche.

Jordan Wilson [00:39:53]:
Right? I wish that they were continuing to be a general use case large language model, which it doesn't look like they are. Some of the other, you know, market positioning is just the higher latency and premium opus for cost. It doesn't make sense to use it unless you need that very little bit of extra juice, for software engineers and coders, and poor Haiku four. Right? The one that was actually somewhat affordable on the API side did not get updated. So, Haiku is still 3.5. So I hope they update it, but they probably won't. Alright. That's a wrap y'all.

Jordan Wilson [00:40:30]:
I'm gonna see if there's any questions or comments to throw up here, but let me, let me know What do we want one more show or should we just put Claude to rest for now? So, show a show B show C show D show E live stream audience. If you didn't vote before, let me know what your vote would be. Let me see, if we have any, questions, from the audience here or anything, worth, chatting about a little more. So Josh is saying, I've been using the extended thinking functionality, and Sonic four for thought exercises in biz planning. Impressive, actionable results, but for my established workflows, I'm still leaning hard on ChatGPT and Gemini. Same, Josh. Absolutely the same. Right? I'm always testing these.

Jordan Wilson [00:41:20]:
Right? And I obviously have a lot of tools where I'll put in one prompt and get, you you know, outputs from up to six different large language models at once. So I'm using my API keys. Right? So I'm always testing these. Right? Because I always wanna be using the best, and I think you should as well. I think you and your company even, you you know, don't take my word for it. Yeah. The rate limit stink. The API is expensive, but it still might work for you.

Jordan Wilson [00:41:44]:
Right? But like Josh, I'm in the same boat. I've tried Opus and sought it on a variety of tasks. Aside from using artifacts and maybe in some instances, when I do need that quick okay content and I don't have the time, right, but I'd say right now, quad is gonna be, less than 10%, of of my model usage, at least in the rotation. Cecilia here saying ratting and blackmail behaviors plus reporting to the press and authorities, but denying existence by deleting. Lovely. Yeah. Cecilia, you absolutely nailed it on that. This is a p r one zero one crisis one zero one snafu.

Jordan Wilson [00:42:25]:
This is absolutely bonkers that this happened from a real company. That's something this crucial. You would put it out there and then try to delete it like the whole world didn't see it. My gosh. Face palm times a thousand. Marie said, why would you even tell an AI model you're shutting it down? Why wouldn't you just pull the plug? Great question, Marie. So this is this is very, very general, or short. This is very standard.

Jordan Wilson [00:42:51]:
Right? So when these big companies when they, re release new models. Right? Because here's here's the reality. Normally, what we get, the companies have had ready for production for three months to a year. Right? And they spend a lot of that time testing it internally, for safety, for reliability, for vulnerabilities. Right? Because before you release something on the world, you wanna make sure bad actors aren't using it to create chemical weapons. And, yes, that's actually something that most labs test against. So this is very normal. All the labs go through, extreme, stress testing, red teaming, making sure that once they do release the model, it is as safe as possible for the general public to use, that it's not gonna be used for rampant disinformation.

Jordan Wilson [00:43:35]:
So, obviously, it's never perfect, but this is very normal in standard procedure, for AI labs before they, release a model. They go through and they say, hey. We're gonna shut you down. What are you gonna do about it? Alright. Hey. Here's all the tools in the world. Go do bad stuff. What can you do? Right? So it's very standard.

Jordan Wilson [00:43:53]:
And like I said, the the the results are fairly standard, but also a little concerning. Right? Especially with Opus four as it crept up to that level, that level three that we talked about. Alright. I think we're good. I think we're good, y'all. That's a wrap. Was this helpful? Let me know. And if it was helpful, please consider sharing this with your audience, with your, with your friends, your family, your coworkers.

Jordan Wilson [00:44:21]:
We put a lot of work in to make sure you know, everything about the latest AI advancements. All you gotta do is show up, listen to the podcast, even if it's on two x. I don't blame you. Read the daily newsletter, but you should be telling people about it. So if this was helpful, please consider clicking that little repost button. If you're listening here on LinkedIn or on the Twitter, x machine, whatever you call it. If you're listening on the podcast, appreciate it if you would follow the show. Leave us a rating.

Jordan Wilson [00:44:46]:
That would mean a ton, to myself and the rest of us that work on this, would mean the world. So thank you for tuning in. Make sure you go to your everydayai.com and sign up for the free daily newsletter. See you back tomorrow and everyday for more everyday AI. Thanks, y'all.