The Rise of Self-Dreaming AI: Imagination Becomes the New Frontier in Machine Learning

Artificial intelligence has long sought to replicate the remarkable learning efficiency and flexibility of the human mind. Now, a new breed of AI that learns from its own "dreams" or imagined scenarios is bringing us closer than ever to that goal. Researchers David Ha and Jürgen Schmidhuber have developed an AI agent that uses an internal world model to hallucinate new experiences and learn from them – essentially training itself without human guidance.

This self-dreaming AI not only matched but dramatically outperformed leading deep reinforcement learning algorithms while requiring orders of magnitude less data. On a challenging car racing game, it achieved scores nearly triple that of a state-of-the-art Deep Q-Learning agent, a result that stunned many in the field.

The Architecture of Imagination

The key insight behind the new approach is to decouple the perception, prediction, and planning components of the AI into a "world model" and a lightweight controller. The world model consists of:

1. A vision module using a Variational Autoencoder (VAE) to encode high-dimensional image observations into compact, information-rich embeddings
2. A memory module using a Recurrent Neural Network (RNN) to learn a predictive model of how these embeddings change over time in response to actions

Together, these modules learn a compressed spatio-temporal representation of the environment. Crucially, the memory module can be used to generate or "dream up" plausible future trajectories, even for situations the agent has never directly encountered.

The controller module is then trained to optimize its actions based on the representations provided by the world model. In the experiments, even a simple linear controller was able to learn highly effective strategies using the world model‘s guidance.

Schematic of the World Models architecture (Image source: World Models website)

Imagination Becomes Reality

The researchers demonstrated their approach on the VizDoom environment and a custom-built 2D racing game called CarRacing. In both cases, the self-dreaming AI was able to learn high-performing policies with far less experience than pure model-free methods.

On CarRacing, the World Model achieved an average score of 906 ± 21 over 100 random tracks, compared to only 343 ± 18 for a Deep Q-Learning agent trained for the same number of time steps. Even more remarkably, the World Model was learning from the equivalent of only 3 hours of real experience, while the DQN was learning from 10 days worth of playing – a 100x sample efficiency advantage.

The CarRacing environment used to test the World Model approach (Source: World Models website)

This huge performance leap was enabled by the World Model‘s ability to learn from imagined trajectories in addition to its real experiences. By hallucinating different scenarios and outcomes, the agent was able to gather far more useful training signal than would be possible from live interaction alone.

Examining the World Model‘s dreams provides a fascinating window into how it learns and plans. The following figure shows a sequence of imagined frames as the model predicts what will happen when taking certain actions:

Examples of imagined rollouts from the trained World Model. Each row shows a different dreamed trajectory. (Source: World Models paper)

In these dreams, we can see the model considering different possible futures and contingencies as it decides how to act. Compared to the simple trial-and-error of model-free RL, this more closely resembles the "mental simulation" and forethought of human cognition.

Related Work and Next Steps

The World Models approach builds on a number of recent trends in deep learning and AI, particularly at the intersection of unsupervised learning, predictive modeling, and reinforcement learning.

Other notable related works include:

• Learning to Think (Hamrick et al., 2017) – an architecture for model-based imagination in physically realistic 3D environments
• Imagination-Augmented Agents (Weber et al., 2017) – combining model-free and model-based learning via rollout planning in imagined trajectories
• MERLIN (Wayne et al., 2018) – an end-to-end RL agent that learns probabilistic predictive models and uses them for model-based planning
• SimPLe (Kaiser et al., 2019) – model-based RL that learns world models purely from video inputs and uses them to train policies via simulation

Each of these works demonstrates the power of learning abstract models or representations of the world and using them to drive reasoning and decision-making. World Models takes this a step further by showing the models can be learned in a primarily unsupervised fashion, reducing dependence on reward signals.

However, significant challenges remain in scaling imagination-based learning to more complex, open-ended domains. The experiments so far have focused on relatively constrained environments with simple visuals and dynamics. Creating models that can capture the full richness of the real world – or even just more advanced game worlds – will likely require substantial architectural and algorithmic innovations.

Some key areas for future work include:

• Learning hierarchical and compositional world models that can reason at multiple levels of abstraction and generalize to truly novel situations
• Efficiently learning world models in high-dimensional, partially observed environments with complex physics and semantics
• Integrating multiple learning and reasoning modalities, such as symbolic knowledge, causal models, and language-based inference
• Ensuring the safety and robustness of increasingly autonomous learning systems that rely heavily on their own imagined experiences

Implications and Outlook

Despite the challenges, imagination-driven learning and reasoning could prove to be a major stepping stone towards artificial general intelligence (AGI). The ability to learn abstract models of the world and use them to simulate different possibilities is a crucial part of what makes human intelligence so flexible and efficient.

As AI systems become better at self-generating their own training data and experiences, they may be able to rapidly bootstrap themselves to higher levels of intelligence and capability. Just as humans learn by dreaming up hypotheticals and counterfactuals, AIs may soon be able to continuously improve themselves "offline" without the need for further interaction or instruction.

This has immense implications not just for the practical applications of AI, but for the future trajectory of the technology as a whole. Self-training AI could greatly accelerate progress in areas like robotics, autonomous vehicles, scientific discovery, and creative design. At the same time, it raises important questions about the transparency, safety, and controllability of increasingly self-directed learning systems.

As a society, it is crucial that we proactively shape the development of imagination-driven AI with the right incentives, safeguards, and oversight. This includes:

• Developing technical tools for monitoring and interpreting the inner workings of self-training models to ensure their reasoning is sound and aligned with human values
• Fostering interdisciplinary collaboration between AI researchers, ethicists, policymakers, and domain experts to surface potential risks and mitigation strategies
• Investing in research on AI safety and robustness, including work on value learning, corrigibility, and scalable oversight for autonomous systems
• Promoting public awareness and dialogue around the evolving capabilities and implications of self-training AI to ensure responsible development and deployment

With the right approach, the emergence of imaginative, self-dreaming AI could mark an exciting new chapter in the quest to create machine intelligence that rivals and extends the capabilities of the human mind. As these systems become more sophisticated, they may not only match human performance but discover new forms of knowledge and ways of thinking that we cannot yet envision.