The paper introduces the BEHAVIOR Vision Suite (BVS), a comprehensive set of tools and assets designed to generate fully customized synthetic data for the systematic evaluation of computer vision models. Traditional real-world vision datasets often lack the extensive and tailored labels required for thorough model assessment under varying conditions. While existing synthetic data generators offer an alternative, they frequently fall short in terms of asset and rendering quality, diversity, and realistic physical properties, particularly for computer vision tasks.
BVS addresses these limitations by providing a robust framework based on the newly developed embodied AI benchmark, BEHAVIOR-1K. This suite supports a wide range of adjustable parameters at multiple levels: scene level (e.g., lighting, object placement), object level (e.g., joint configuration, attributes such as “filled” and “folded”), and camera level (e.g., field of view, focal length). These parameters can be varied arbitrarily during data generation, allowing researchers to conduct controlled experiments and systematically evaluate model performance.
The paper highlights three example application scenarios to demonstrate the utility of BVS. First, it can be used to systematically evaluate the robustness of computer vision models across different continuous axes of domain shift. Second, it enables the evaluation of scene understanding models on a consistent set of images. Third, BVS facilitates the training and evaluation of simulation-to-real transfer for a novel vision task: unary and binary state prediction.
Overall, the BEHAVIOR Vision Suite represents a significant advancement in the generation of synthetic data for computer vision research. By offering high-quality, diverse, and customizable data, BVS enables more rigorous and comprehensive evaluation of computer vision models, ultimately contributing to the development of more robust and reliable AI systems.
