Preference-Conditioned Language-Guided Abstraction

Learning from demonstrations is a common way for users to teach robots, butit is prone to spurious feature correlations. Recent work constructs stateabstractions, i.e. visual representations containing task-relevant features,from language as a way to perform more generalizable learning. However, theseabstractions also depend on a user's preference for what matters in a task,which may be hard to describe or infeasible to exhaustively specify usinglanguage alone. How do we construct abstractions to capture these latentpreferences? We observe that how humans behave reveals how they see the world.Our key insight is that changes in human behavior inform us that there aredifferences in preferences for how humans see the world, i.e. their stateabstractions. In this work, we propose using language models (LMs) to query forthose preferences directly given knowledge that a change in behavior hasoccurred. In our framework, we use the LM in two ways: first, given a textdescription of the task and knowledge of behavioral change between states, wequery the LM for possible hidden preferences; second, given the most likelypreference, we query the LM to construct the state abstraction. In thisframework, the LM is also able to ask the human directly when uncertain aboutits own estimate. We demonstrate our framework's ability to construct effectivepreference-conditioned abstractions in simulated experiments, a user study, aswell as on a real Spot robot performing mobile manipulation tasks.