Intro to AI Safety, Remastered

If only. There's a lot of space in between "our morality" (please do define that) and "what we want to be" where AGI can royally screw us. Check out Robert Miles on AI safety, here's a good intro: ▶ this video

I recommend Robert Miles's YouTube channel, and his "Intro to AI Safety, Remastered" is a good place to start: ▶ this video I find Robert Miles worryingly plausible when he says (about 12:40 into the video) "if you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever"

The basic argument is trivial: it is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don't know how to make AI systems encode any complex preference robustly. Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction. Eg. this one on instrumental convergence: link Eg. this introduction to the topic: ▶ this video He also has the community-led AI Safety FAQ, link , which gives brief answers to common questions. If you have specific questions I might be able to point to a more specific argument at a higher level of depth.

Take your pick? link link ▶ this video link link I could keep going but the reading on this spans for tens of thousands of pages of detailed reasoning and analysis, including mathematical proofs and lab-scale experiments.

I strongly suggest anyone doubtful of the reasoning behind these concerns or asking questions like "well why don't we just do x to solve this?" go check out Robert Miles' YouTube channel[1]. It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set. He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details. One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously. I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now. [1] ▶ this video

Is there any chance you could add some links to AI Safety Resources such as Joe Carlsmith's Power Seeking AI ( link ) or Rob Miles Introduction to AI Safety ( ▶ this video )? Perhaps you could link to a resource on the AI box experiment as well? I know you vaguely gesture at it, but would likely be better to explicitly link people to resources if they haven't heard of it themselves.

Venkatesh’s argument is so poor it comes close to “not even wrong” territory. He doesn’t really cite any specific people or literature to disagree with, and instead makes only vague references to the “non-sense” of others. The most significant concerns around AI safety are not discussed nor even mentioned leading it to read almost as if he has no expertise whatsoever in this field, and indeed that seems to be the case. No need for an appeal to authority though - just taking a little time to hear the actual concerns and opinions on topic would be welcome. Great intro to the subject here, I wonder what his response would be? ▶ this video