Abstract
Multi-modal robot programming with natural language and demonstration is a promising technique for efficient teaching of manipulation tasks in industrial environments. In particular with modern dual-arm robots, which are designed to quickly take over tasks at typical industrial workplaces, the direct teaching of task sequences hardly utilizes the robots' capabilities. We therefore propose a two-staged approach that combines linguistic instructions and demonstration with simultaneous task allocation and motion scheduling. Instead of providing a task description and demonstration that is replayed to a large extent, the user describes tasks to be scheduled with all relevant constraints and demonstrates relevant locations and storages relative to workpieces and other objects. Constraint optimization is used to schedule task and motion sequences to minimize the makespan. Naming and grouping enables systematic reuse of sub-tasks ensembles and referencing of relevant locations. The proposed approach can generalize between different workspaces and is evaluated with gluing showcases from furniture assembly.