How to Prompt Your Robot: A Prompt Book for Manipulation Skills with Code As Policies

Large Language Models (LLMs) have demonstrated the ability to perform semantic reasoning, planning and write code for robotics tasks. However, most methods rely on pre-existing primitives, which heavily limits their scalability to new scenarios. Additionally, they use examples prompting style where the LLM is provided few-shot examples of robot code. This presents a challenge for LLMs to implicitly infer task information, constraints, and API usage from examples alone. Meanwhile, research outside robotics has successfully studied instruction-based prompting, where providing LLMs with API documentation and detailed descriptions can improve code synthesis capabilities. However, it is not clear how to document robotics tasks and naively providing full robot APIs presents a challenge to context-length limits in LLMs. However, it is not clear how to document robotics tasks and providing full robot APIs presents a challenge to context-length limits in LLMs. In this work, we discuss how to combine different LLM prompting styles to write code for new manipulation skills. Firstly, we evaluate different prompting styles across 3 robots in a high-level sorting task, and present a collection of empirical observations: (i)including both instructions and examples improves performance, (ii)interleaving state predictions in the examples helps reasoning,(iii)instruction-based prompting benefits from human feedback. Our observations lead to a prompt recipe we refer to as PromptBook that combines: example-based, instruction-based and chain-of-thought prompting to write robot code; as well as a method to build the prompt leveraging LLMs and human feedback. Secondly, we show PromptBook can write code for new low-level manipulation skills on the fly zero-shot. The prompt extracts motion trajectories from LLMs that the robot can execute directly with an IK controller. Finally, we evaluate the new skills on a mobile manipulator with 83% success rate at picking, 50-71% at opening drawers.