Multi-Agent Hide and Seek

True RL is not limited by being tethered to human-annotated data, and it is able to create novel approaches to solve problems. True RL requires a very clear objective function (such as the rules of Go, or Starcraft, or Taboo!) that the model can evaluate itself against. Andrej Karpathy talks about the difference between RLHF and "true" RL here: link > The other thing is that we're doing reinforcement learning from human feedback (RLHF), but that's like a super weak form of reinforcement learning. I think... what is the equivalent in AlphaGo for RLHF? What is the reward model? What I call it is a "vibe check". Imagine if you wanted to train an AlphaGo RLHF, it would be giving two people two boards and asking: "Which one do you prefer?" -- and then you would take those labels and you would train the model and then you would RL against that. What are the issues with that? It's like, number one -- that's just vibes of the board. That's what you're training against. Number two, if it's a reward model that's a neural net, then it's very easy to overfit to that reward model for the model you're optimizing over, and it's going to find all these spurious ways of hacking that massive model is the problem. > AlphaGo gets around these problems because they have a very clear objective function, and you can RL against it. > So RLHF is nowhere near [true] RL -- it's silly. And the other thing is that imitation is super-silly. RLHF is a nice improvement, but it's still silly, and I think p

Here [1]. They called it "emergent" property, but I only see that as inevitable behaviour. [1] ▶ this video

> There's just something I can't describe with words about what it lacks today - where it's possibly more philosophical than technical My personal viewpoint is that there's nothing inherently special about biological intelligence that can't theoretically be reproduced with electronics, but that today's AI has some practical limitations like no significant persistent internal state. I saw an "Inner Monologue" paper by Google Robotics recently which could be a step in the right direction. > Humans for example operate within the universal set and within all dimensions we can percieve. Our brain's direct output is electrical signals to a set of muscles. I don't think there's anything in particular stopping a physical robot from having similar generality of operating space. > Even in todays state of the art if we gave robots all the sensors that we have they still couldn't evolve with their environment like we could Ability to evolve seems mostly orthogonal to intelligence to me. If humans instead came about by an intelligent creator or gradient descent, wouldn't we still be intelligent? AI adapting to the environment through gradient descent or self-modification should be fine and faster than evolution. > or invent fire like we would with current algorithm or neural nets. They would just keep doing whatever they're programmed to do within their set of "stuff" for lack of a better term In virtual environments, neural networks have adapted to changes and learned new strategies, eve

I think evolution is not an undirected process in that sense because it's an optimization process, that optimizes to create more copies of itself. Superintelligence will likely use some Evolutionary Computation (see en.wikipedia.org/wiki/Evolutionary_computation ). Also see Karl Sims 'Creatures' from the 90s: youtube.com/watch?v=JBgG_VSP7f8 or OpenAI's Multi-Agent Hide and Seek: ▶ this video

In statistics and machine learning this problem is called concept drift and it's an open research problem. Indeed all the models I listed are already out-of-date with the present state of the world; unless they are fine-tuned on new data. The regime for the above networks is called "pre-training". The idea here is that rather than training _directly_ on some challenging, specific task, you instead train a more generic task on a _lot_ of data. This gives you a "backbone model" that winds up being very strong on more specific tasks as well. In many cases, the ability to (cheaply) curate or create enough accurate data for specific sub-tasks might not even be possible. It's easier to scrape 400 million captioned images from the internet than it is to find/create millions of visual Q&A prompts/images. CLIP is a great example of this. While it was trained explicitly "just" to compare images and captions and to output a score of the cosine similarity between the features of the two - I have seen it approach effective state of the art on text-to-image generation tasks, image-captioning, Q&A, etc. This style of training needs to be updated after it is trained, typically because of the way these datasets are curated. Automatic curation via pretrained models is one option. Another option is to give the model direct access to the internet which is starting to be explored. Pre-training helps a lot with general distribution shift; but it's _probably_ not going to be able predict memes befo

The work I've seen recently from them on multi-agent play seems to indicate this is a problem they're very successfully working on. ▶ this video

Specialized AI is unlikely to be a threat. The issue will be with general AI, and simple but not fully thought out utility functions. This short video makes the concern much more real: ▶ this video Consider what would happen if you took that same AI, make its utility function similar to this: link , and released it on the internet.