8 Predictions for the Era of Continual Learning
I have explained elsewhere why I think continual learning is needed. I don’t think you can have AIs that perform whole jobs as competently as humans if they’re forced to just write Markdown files from session to session.
To give an illustrative example, imagine if the way students had to learn to play the saxophone is that one student tries to play it from a cold start, then after her first session, writes down a bunch of notes, then the next student waiting outside the music hall who’s also never played the saxophone reads all their notes before trying to play, and so on. Even if you had an infinite sequence of saxophone-virgin students waiting outside the studio that could write notes to the next guy, there’s no sequence of text they could write together that would allow the Nth student outside to play proficiently on their first try. At some point, you have to accumulate the experience into the brain. I think the same will be true about a lot of skills and knowledge that we’ll want AIs to learn in all the different workplaces they find themselves deployed in.
Okay, so what changes about AI once we have continual learning?
- A lot of the proposals that have been put forward for regulating AI assume that you train a model, and then you deploy it. And therefore if we run a bunch of checks before the model is deployed, then we can make sure that it’s not going to aid in cyber attacks or recursive self improvement. But what if the base model is getting updated every single day based on the millions of sessions of work it does? This is one of many reasons why I think it’s unwise to lock in some kind of regulatory safety regime right now. We simply don’t know what kind of technology we’re going to be looking at even in a year, let alone in five years or ten years, and we’d be entrenching an archaic and potentially counterproductive approach to dealing with the threats from AI. To the extent that the government really wants to do some kind of safety evaluations on model providers, it would make more sense to do monthly or quarterly risk inspections rather than singling out some special moment that occurs after training is done and before deployment begins, because that will not be a meaningfully distinct category in the future.
- How the labs do technical alignment would need to totally change. Almost all current techniques are focused on the problem of how we make it so that a frozen set of weights behaves well during deployment. I’m not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are agglomerating learnings between users as well, how do you prevent users from injecting backdoors or some kind of malicious inclination into the base model? In some sense it is actually closer to the human alignment problem - your kids go out and learn new things, sometimes get one-shotted by crazy ideologies or drugs or something - but you hope you’ve given them enough common sense and basic values to improve as people in a self-directed way, without ending up with some super weird and misanthropic beliefs.
- The diversity of AI minds will increase. Right now, there are <5 prominent AI minds, and they are all quite similar to each other on account of being trained on roughly the same data. But if AIs are learning from experience, and that experience is different between different AIs, we could see actually different AIs come out the other end.
- When deployment becomes part of training, the returns to being ahead will accelerate. If you have the best model, and more people use your AI for more complicated and useful work, and give it lots of feedback that it can integrate beyond the session window, then your model will become even smarter.
- If the model learns mainly from deployment, labs will feel the pressure to deploy their smartest model
