orthogonality

Robert Miles explains the orthogonality thesis: how an agent can be highly intelligent while pursuing goals that seem stupid or alien to humans. He separates intelligence from goals and touches on the stamp-collector example.

Robert Miles explains instrumental convergence: why an AGI with almost any goal would tend to pursue sub-goals like self-preservation and resource acquisition. It builds on Omohundro's basic AI drives and his orthogonality thesis video.