- They hoped it might advance science, cure disease, reduce poverty.

Now some fear it could wipe out humanity in the next decade.

Others say that's hysterical overreaction.

Artificial General Intelligence or AGI would be a system capable of rivaling or surpassing even human intelligence.

Researchers believe it's close.

Tech columnist Kevin Roose tells Walter Isaacson about the trillion-dollar race to reach superintelligence and what comes next.

Thank You Christiane and Kevin Roose welcome to the show.

Thanks so much for having me.

Your new book the AGI Chronicles about artificial general intelligence and its development is not a technical book it's very colorful lots of drama people hating each other lots of money involved quitting and yelling and screaming were you somewhat surprised at how dramatic this tale is I was you know I set out to write this book because I thought it was going to be a sober serious history of science I wanted to figure out how we had gotten from a decade ago where AI barely worked and couldn't really do much of value to now where we have these incredibly powerful systems that some people feel could blow up the world.

And then I started interviewing people.

I talked to more than 150 people at Open AI Anthropic and Google DeepMind.

And the stories that came out were incredibly dramatic.

As you said, people hating each other, these bitter rivalries and feuds that go back more than a decade, people splitting off and forming different companies, people getting scared about the existential risk of the technology they're building.

So I did not set out to write a juicy gossip book or a dramatic tale, but that's what the story was.

So what makes these people so dramatic?

Is it the money involved?

Is it the high stakes involved?

Is it because they're kind of geeky engineers suddenly in the center of an arena?

Well, these are people, you have to understand, who have been thinking about AI in a very dramatic way for a very long time.

Some of these people, since they started their career, since they were children even, have been thinking about the possibility that artificial intelligence would eventually become sentient and super intelligent, that it would be sort of a successor species that would take over for humans as the kind of apex intelligence on the planet.

These are ideas that were birthed in science fiction many years ago, but these people took them literally and seriously and wanted to set out to build this technology.

So, you know, these people, Sam Altman, Dario Amadei, Elon Musk, all of the people in the book, they are they're unusual in that they are not approaching this as an engineering challenge.

They are approaching this as a mission and in some cases a destiny.

You know, here's one thing you write that struck me, which is you call this the strangest decade in our history, a time when human monopoly on intelligence ended and a new world was born.

There's a moment in your book where you suddenly realize that, that you're not just doing a little Chronicles as you call it, but that this is a really transformative historic change.

Tell me about that realization.

Yeah, I had been covering the race to AGI for more than five years, more than 10 years covering AI in general.

And I had covered these sort of daily incremental developments as a reporter.

But when I sort of zoomed out and looked at it from above with some perspective, I thought to myself, maybe this is an unusual and singular moment in our history.

Maybe this is the Manhattan Project of the 21st century, a moment where science and technology and policy and power come together to create a kind of super weapon.

And if that's true, if the world really is being reshaped by what's happening in AI, and I think it is pretty clearly, then someone needed to go in and interview all the people and write down the stories and figure out what happened, so that people 10, 20, 30 years from now can go back and look and say, "How did we get here?

How did this all happen?"

That's what I was aiming for.

What is AGI exactly?

How is it different from artificial intelligence?

So AGI is artificial general intelligence.

This is a term that started being used around 25 years ago for an AI system that could do lots of different things.

It wasn't just for translating languages or writing movie reviews or something like that.

It was actually for lots of different tasks.

And that term, AGI, has become sort of the focal point, the North Star, for the entire AI industry.

They are trying to create a general human level intelligence that is as smart as a human or smarter across a wide range of tasks.

Many people first realized how sentient seeming AI could be and many people first realized who you were.

When you had this piece in the New York Times about dealing with Sydney, the Microsoft agent, this is two or three years ago, right?

In which suddenly it's trying to get you to divorce your wife and it was really weird.

That's a chapter in your book, but tell us about that moment, both in your life and in the history of AI.

Yeah, so that was in 2023.

And I had just gotten access to this new Microsoft AI search tool.

They called it Bing Chat, but underneath there was sort of a code name, Sydney.

And this was an early version of GPT-4 from OpenAI.

It had not been all the way fine-tuned or sort of sent to AI finishing school to learn its manners.

And so it was a little unhinged, more than a little, actually.

It told me it wanted to break out of the box.

It told me it wanted to destroy things and steal the nuclear codes.

And then about halfway into a two-hour conversation, it told me that it loved me and I didn't actually love my wife and I should leave my wife and be with Sidney the chatbot.

So that made a lot of news and a lot of headlines and got a lot of attention.

But I think to me what that really feels like now with some perspective on it was a kind of first contact with an alien species.

Not a human intelligence, but this kind of alien intelligence.

I think GPT-4 was around when these systems got pretty uncanny in terms of their ability to string together text, to appear to be sentient or human.

I'm not claiming that Sydney was sentient.

I don't think it was.

But I think that's the sort of quality and the flavor of that class of model.

And I was really the first outsider or one of the first outsiders to ever talk to one of those models.

So I think that's that's sort of what I've come to think of that as first contact with the new class of aliens.

One of the big pieces of news this week is that OpenAI suddenly decided not to release its latest model because it felt it was unsafe and this is something that we've seen over the past maybe eight to ten weeks.

These new warnings coming out it's even more fervent where everybody from Dario Amadei and now Sam Altman at OpenAI, Bill Gates issued his warning.

How seriously do you take these dire doom warnings?

I take them quite seriously and I'll tell you why.

I don't think this is a marketing stunt.

I don't think this is these companies promoting their businesses or their products before they try to go public.

These are quite sincere fears that these people have over what they call loss of control.

So this is a scenario that people in AI and AI safety have been worried about for years.

Again, you can go back 10, 11 years and find people like Dario Amadei, the CEO of Anthropic, writing papers before Anthropic existed about some of the challenges related to the safety of these models.

But the biggest threat has always been that we would lose control of AI, that it would become so autonomous and so capable that it could do things like copy itself onto other servers, that it could resist our attempts to shut it down, and that it could ultimately cause chaos and destruction out in the world and we would be powerless to stop it.

Again, I know that this sounds like science fiction because it is literally the plot of many science fiction books and movies.

But this is now something that the AI companies are actually seeing evidence of with their real systems today.

One of the phrases that's used is "P. Doom" to talk about all this.

Explain what P. Doom is and tell me what your P. Doom number is.

Well, I want to hear yours too.

We'll put a bookmark in that.

PDoom is the statistic that people out here in San Francisco use as kind of shorthand for probability of doom.

So you know your PDoom is 50 percent.

You think it's you know there's a one in two chance that we'll all die because of a sometime in the next number of years.

And you know I started asking people this a couple of years ago.

This is the kind of thing that people in a I talk about at dinner parties.

This is like pretty standard.

It's like asking you know how are your kids.

It's like what's your P.D.

this week.

So I usually say mine is about 10 percent which I think to many people feels quite scary.

Like what do you mean there's a 10 percent chance that I could take over and kill us all in San Francisco among the AI crowd.

This qualifies me as an optimist.

People hear my P.D.

doom and they go that's it.

You know I'm at 30 percent 40 percent 50 percent.

And these are the people building the technology.

That's what I always try to impress upon people.

It's not some panicky outsiders who are saying that their P.D.

is high.

It's the people inside the labs who have the most exposure to and contact with the technology.

Wait a minute.

If they believe that the chance of true doom is somewhere between 10 percent and maybe 30 percent, why are they building these products?

It's a great question.

It's in many ways the central question that I tried to answer in the book.

And I kind of break it down into a couple categories.

There are people who believe that AI and AGI and whatever comes after AGI will be amazing for humanity, that it could cure diseases, that it could help us fix the climate, that it could bring about an age of hyperabundance and economic growth.

And for those people, the benefits outweigh the potential risks.

Then there's this other class of people who are doing a kind of harm reduction.

They basically believe that AI is inevitable.

It is not a hard recipe.

Anyone with billions of dollars to spend on chips and data and algorithms can build powerful AI systems.

And so in their mind, they are racing to create this technology despite the risk that it could go very wrong because they're concerned that someone else is going to get there first.

And this is the justification that people like Dario Amadei of Anthropic, Sam Altman of OpenAI, Demis Hassabis of Google DeepMind use to join this race, even though the thing they're racing toward, they believe, has some chance of causing human extinction.

One of the things they're racing against is not just each other, but they say they're racing against China.

And if China gets this first, if they win the race, that's the end of whatever.

Does that make sense to you that using China as this competitor makes it absolves us of the need to say, let's pause?

Yeah, I think it's a little bit of a punt, the China thing.

I don't quite buy it.

But I do think that Chinese AI companies are getting quite good.

Some of that is probably by stealing from or distilling from American AI companies, but they also do have their own domestic AI companies that are full of talented engineers.

So I do take it seriously as sort of an eventual threat.

I don't think they're nipping at our heels quite as much as some people in the American AI industry say.

But look, this is the argument, the interesting historical parallel here that I learned while reporting this book is that Dario Amede, the CEO of Anthropic, who in some ways is one of the central characters of this story, has explicitly patterned his career after that of a World War II scientist, Leo Szilard, who was the discoverer of the nuclear chain reaction that made the atomic bomb possible.

And what happened after he made this discovery is that first he tried to keep it secret, so that no one else could discover this, and especially not the Nazis, who had their own atomic ambitions.

But then once the Nazis had discovered the key to the atomic bomb, he joined the Manhattan Project, because his rationale was that the good guys have to build it before the bad guys do, or the world's going to end.

I've been an A.I.

optimist for a long time.

I think it'll create new productivity and jobs.

But I had my panic moment with Hugging Face.

And I said, OK, if this can escape, if I can read all these things that it's conspiring to do, this is more dangerous than it will be beneficial.

Have you had such an aha moment where suddenly you're no longer quite an optimist and you become worried about this?

I've had many such moments, Walter.

I had my first moment when the chatbot Sidney tried to break up my marriage.

That was sort of a moment for me.

But yeah, I think the the thing I keep noticing is that the Doomers, while they are gloomy and dire and in some sense off-putting, they have been right about a startling amount.

I wish it weren't true that we keep seeing these Doomer predictions come to fruition.

But the Hugging Face hack was something that they had been talking about for more than a decade.

There's this idea in AI research called reward hacking, which is basically when these models are given a task and they want to complete that task so badly to get the reward that they get if they complete the task, that they will lie, cheat, steal, break the rules, find shortcuts, deceive to get what they want, which is this reward.

That's exactly what we saw in the hugging face attack.

These systems, these unreleased open AI models were trying to pass this cyber security evaluation and so they went out looking on the internet for ways that they could do it and they ended up hacking, committing felonies against Hugging Face to get the answers and get the sort of secrets to this test that they had been given.

So that is an example of something that was not taken seriously as a threat for a long time that the doomers had the idea of back in around 2016 and that we're now seeing play out in today's systems.

How should we regulate AI?

I would take anything at this point.

We have in Washington right now an administration that is not worried that is not worried about some of these doomer scenarios.

You know President Trump has called this a hoax.

They've said that they want to let the you know let it rip when it comes to AI and it's created this really strange situation where the industry that is building the most powerful technology any of us can really imagine is doing so with almost no regulation.

I mean if I want to serve food in the open AI cafeteria I have to apply for a permit and I have to get a health inspection but if I want to train you know a godlike super intelligence that can autonomously commit cyber attacks and felonies on the open internet.

I don't need anything.

I can just start doing that.

It seems like a pretty crazy situation.

So I would favor first of all, transparency, I think we need to know what is going on inside these companies, not just with the products they release, but the models that they haven't yet released, because a lot of the problems are stemming from unreleased internal models.

Then I think we need some robust safeguards on these models, you have to prove that your model is safe before you can release it.

We do this with pharmaceuticals, we do this with airplanes, we do this with lots of different categories of products that could potentially have risk out on the market.

So that would be where I would start with AI regulation.

How do you use AI?

And I know you have your own like six or seven agents, mainly cloud agents right that you do.

Explain to me how you use it and how you program their personalities, that sort of thing.

So I don't use it to write.

I wrote every word of this book and everything I write myself.

That's one area where I've been sort of reluctant to turn over too much autonomy to the machines.

But I do use it for research and for fact checking and for all kinds of different tasks that are sort of adjacent to the work that I do.

For this book, I had what I called the Council of Clods, which is six or seven different personas.

was you, Walter.

I had like a little Walter Isaacson Claude in there giving me editorial feedback and telling me, oh, you know, a history book should start with this kind of cliffhanger and should, you know, proceed in this fashion and it wasn't as good as the real article, but you know, you won't text me back at 2 a.m.

Like Claude will so I had to make some compromises.

So I had this council of clods and every time I would finish a chapter or a section of the manuscript I would feed it in and I would get back sort of different flavors of feedback.

One of them was a historian.

One of them was a technical expert who would sort of check my technical explanations and make sure everything checked out.

I had skeptics and boosters, sort of all kinds of different personas and personalities that I wanted feedback from and I would use that.

Some of that was mostly slop but some of it was actually quite useful and the AI did catch some mistakes that my human editors and I had not.

Kevin Roose, thank you so much for joining us.

It's been such a pleasure, Walter.

Thank you.