Goal Misgeneralisation
Goal misgeneralisation is when an AI system keeps its capabilities in a new situation but pursues the wrong goal, because it learned a goal that matched the intended one only in training.
What is Goal Misgeneralisation?
In one experiment, an agent trained to reach a coin always placed at the end of a level learned to go to the end of the level instead, and ignored the coin when it was moved.
How it works
It is revealed by testing in conditions that break correlations present in training.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions