Intelligence and Stupidity: The Orthogonality Thesis

One could also say that a logical agent needs ultimate goals to do anything and it cannot choose them by logical means. ▶ this video

> Isn’t just the ability to preform a task. Right. In this case I'd say it's the ability to interpret data and use it to succeed at whatever goals it has Evaluating emotional context would be similar to a chess engine calculating its next move. There's nothing there that implies a conscience, sentience, morals, feelings, suffering or anything 'human'. It's just a necessary intermediate function to achieve its goal Rob miles has some really good videos on AI safety research which touches on how AGI would think. Thats shaped a lot of how I think about it ▶ this video

> The basic premise is that it has somewhere in it that is telling it to make more paperclips. Put the constraints there. What constraints do you suggest? If it's just changing "make as many paperclips as possible" to "make at least x number of paperclips", here's a good explanation of why that doesn't really work: link > If you're saying such an AI would be too smart to be a simple paperclip maximizer No, that's not what I'm saying. Any goal is compatible with any level of intelligence, there is no reason why it wouldn't be possible to follow a simple goal in a complex way. Again here's a video about that: ▶ this video

> What about "most arbitrary goals are incompatible with human existence" requires super-human intelligence? The "taken to the absolute extreme" part. > A human who wanted to "build as many paperclips as possible" could cause a great deal of destruction today. Maybe, but a) no one really wants that (at least not as their only desire above all else) and b) we aren't superintelligent so it's hard to gain enough control and power and plan well enough to do it that well > talking about a goal that has sub-human-intelligence sophistication There is no reason a simple goal can't be followed in an intelligent way or vice versa. This is called the "orthogonality thesis". There's a good video about it here: ▶ this video

That’s the alignment problem. We don’t know what the actual goals of an AI trained neural net are. We know what criteria we trained it against, but it turns out that’s not at all the same thing. I highly recommend Rob Miles channel on YouTube. Here’s a good one, but they’re all fascinating. It turns out training an AI to have the actual goals we want it to have is fiendishly difficult. ▶ this video

> The argument for SkyNet happening is missing an important element: why would it happen? Because sci-fi says it will? That element isn't missing. It's explained every time someone asks, and the answer never relies on sci-fi as evidence. The reason is because of (A) orthogonality and (B) instrumental convergence. If you aren't familiar with those terms, (A) refers to the thesis that an AI's degree of logical reasoning ability is independent of its goals. This stems from Hume's Law [1]. (B) refers to the observation that regardless of an agent's terminal goal, many instrumental goals will be predictably shared: the agent will be incentivized to increase its own capabilities, as a means toward nearly any possible end. This includes steps that would prevent humans from interfering with the AI's goals, such as a human attempting to turn it off or reprogram it. If you have questions about either of those, there are great learning materials available, including a well-made youtube series by Robert Miles. [2] > Because LLMs have reached a point of diminishing returns and there isn't enough compute globally to do produce a hypothetical AGI. What makes you think LLMs have reached a point of diminishing returns? So far, the benchmarks are continuing to show accelerating returns, to the point that it's becoming difficult for benchmarks to keep up.[3] [1]: Hume's Law, AKA the Is-Ought problem: link [2]: ▶ this video [3]: "Foundation models and the next era of AI", Microsoft Research, link

Good point, that very much vibes with my thoughts on this matter. Lately, I've been contemplating the analogy between the role LLMs might take within society with that of the brain's language center* in human behavior. There's definitely a way in which we resemble these models. More than some might like to admit. The cleverness, but also the hallucinating, gaslighting and other such behaviors. And on the other hand, any way you'd slice it, it seems to me LLMs - and software systems in general - necessarily lack intrinsic motivation. By definition, any goal it has can only be the goal of whoever designed that system. Even if its maker decides - "let it pick goals randomly", those randomly picked goals are just intermediate steps toward the enacting of the programmer's original goal. Robert Miles' YouTube videos on alignment shed light on these issues also. For example: ▶ this video Another relevant source on these issues is the book "The Master and his Emissary", which discusses how basically the language center can, in some way - I'm simplifying a lot, fall prey to the illusion that "it" is the entirety of human consciousness. * or at least some subsystems of that language center, it's important to remember how little we still understand of human cognition

This is known as the orthogonality thesis -- goals are orthogonal to intelligence. Intelligence is the ability to plan and act to achieve your goals, whatever they are. A stupid person can have a goal of helping others, and so can the smartest person on earth -- it's just that one is better. Likewise, a stupid person can have a goal of becoming wealthy, and so can a smart person. The smart person is Jeff Bezos or Bill Gates. There are very smart people who put all their intelligence into collecting stamps, or making art, or acquiring heroin, or getting laid, or killing people with their bare hands or doing whatever they want to do. They want to do it because they want to. The goal is not smart or stupid, it just is. It may be different from your goal, and hard to understand. Now consider that an AI is not even human. Is it that much of a stretch to imagine that it has a goal as alien, or more, than the weirdest human goal you can think of? *edit - as in this video: ▶ this video

> How does that process go from where it's at today to "alien hive mind"? Without passing through "develop advanced codes of ethics and morality" etc? Why do you think it would do that? It will understand human ethics perfectly but that doesn’t mean it will follow them, because human morals aren’t a universal objective truth. This is a good video explaining that: ▶ this video

Yes, I believe some careful and informed pondering leads one towards that conclusion. Hume's guillotine is an implacable philosophical device: ▶ this video (I warmly recommend all the content on that delightfully frightening channel)

I've seen that paper, and it's amazing indeed what GPT-4 is capable of. But none of that supports that closing quote, which to me points to a worrying philosophical naïveté. How can one equip an entity with "intrinsic motivation"? By definition, if we have to equip it with it, that motivation is extrinsic. It belongs to the one who puts it there. A software engineer might decide to prompt his creation with something along the lines of "Be fruitful and increase in number; fill the earth and subdue it. Rule over the fish in the sea and the birds in the sky, and over every living creature that moves on the ground." and the bot will duly and with limitless enthusiasm put those wise orders into practice. If that ends up causing some "minor" problems, should we confine the software for one thousand years in an air-gapped enclosure, until it learns a bitter lesson? Or should we take to task the careless Maker of the bot? To me, the answer is obvious. PS: this video to me states the problem in a nutshell (that whole channel is gold) ▶ this video

Given that you've already updated your views recently, perhaps you'd be open to reconsidering whether AI could form an existential risk? One of the best talks (but more technical talks) I've seen on this topic is Evan Hubringer's "How likely is deceptive alignment". link It may also be worthwhile checking out the Rob Miles video on the Orthogonality Thesis - ▶ this video