I had a text message with a friend of mine recently about AI threats. It went like this.

Friend: I equate it to the same dilemma the Manhattan project had about lighting the atmosphere on fire… it’s nonzero until it’s not

Friend: I’m not so much concerned about extinction level threats as I am about the risk of concentration of power and catastrophic economic disruption.

Me: But the probability was never greater than zero. It was always zero. What we believed about the probability has no impact on reality. It was just a belief, just as fictitious and stupid as the king of Babylon taking the hand of Marduk at sunset on New Years meant the harvest would be fruitful the next year.

This is where the frequentist philosophy of probability fails really hard. Because if you imagine something that isn’t real, like an infinite number of trials, you end up with a situation where literally everything is somehow greater than 0 probability. And if you do the expected value calculation and weigh all of human existence against a .000000000000000000001 probability, the EV is always negative. It’s a calculation that relies entirely on a false premise.

Frequentist approaches to risk calculations aren’t actually falsifiable. And they are, in my opinion, unscientific. Popper would agree with that, at any rate. When you allow for infinity in your math, you have a similar but inverted problem as dividing by zero.

The problem with dividing by zero isn’t that it somehow breaks the universe. It just makes all of our beliefs about quantity meaningless. If you can get a numeric result from dividing by zero, you can get any numeric result from any calculation. We reject that. We define it as undefined.

Under certain circumstance, like frequentist probability theory, we should reject infinity because it does essentially the same thing. If you define reality as a space where everything can happen, then you have to account for the possibility that everything will happen. And this is just a plainly false view of reality. It’s not meaningfully different from a religion.

Your second point is incredibly important, but it has nothing to do with AI. That’s an entirely human problem that we create for ourselves over and over since the beginning of always. That’s why I say that the problem is confounded.

People are using frequentist fears about what might happen based on the religion of numbers to consolidate power and do economic and historical arson. That is absolutely happening right now. It has happened in the past, and it will happen again. Terrible things are going to happen. Just like they did when our primary means of creating tools moved from stones to bronze. And then iron. And then steel. Or when we figured out that a pottery wheel could be flipped on its side and used for transportation.

Entire civilizations existed at some point and stopped existing, and they were wiped out so completely and totally that we know nothing about them other than they must have existed, otherwise nothing about our history makes any sense at all.

At every stage in history, we have looked at the new thing and blamed the new thing for what we were already doing to each other. AI isn’t any different. We are going to human each other just like we have always humaned each other, and it’s going to be terrible.

My Bayesian outlook on the situation is that the probability of awfulness is nearly 100%. We live, in certain way, in a very small universe of possibilities, and we should have a strong prior that we are going to keep doing what we have always done: scare the shit out of people to herd them into a hunting pen so that we can slaughter them like cattle.

There is no stopping human horrors. The only question is where we are going to end up. Are we going to participate in them? Are we going to fight against them and get crushed in a noble but hopeless resistance? Are we going to try and find a middle path? Are we going to risk being judged by future generations as being collaborators and conspirators because we tried to get our bag and take care of our families and loved ones?

Here is one thing I am certain of: we should not act out of fear. We should not allow our lives to be ruled by these assholes who have overpromised and underdelivered and are now so completely desperate to justify their trillion-dollar valuations that they must pivot to scare tactics about the fate of the universe.

There is no god-mode AI. There won’t be an RSI intelligence under the current architectures. You remember the meme from 10 years ago when people were like, “Hey we are AI!” but it was really just some dudes in southeast Asia doing mechanical turk stuff with a Google Doc? That’s RLHF and RLVR now. It’s humans all the way down, and Sam Altman and Daario AmIAMotherfucker or whatever his name is are using us and our fears of entirely human things to create a legal moat where no technological moat exists.

The problem you’re talking about is real, and it’s a big problem. But it’s a human problem. Star Wars got it wrong. Fear leads to anger leads to hate leads to suffering. That’s bullshit. Fear leads to begging our leaders to turn us into slaves. It’s a really short path on a graph.

We need to get our shit together and act like men. We need to take care of ourselves and our families and friends and loved ones. We need to get as high up in the chain of command as we can in these companies or start our own, and we need to mitigate the total and complete economic and social destruction that’s about to happen and in some areas is already happening.

There is a real test ahead of us, me and you <redacted>. <redacted>. <redacted>. Everyone we know and care about and love. Our parents. Your kids. The test isn’t if we’re going to stop this. It’s what we’re going to do to make the blast radius as small as we can. The test is whether we can do something different than humans have always done.