Browser Use updates to allow models to choose observations and actions via CDP

The update to Browser Use marks a significant shift in how models interact with web pages, moving from a fixed set of actions and observations to a more dynamic approach. By allowing models to write their own observations, this change enhances their ability to adapt to different page structures and elements more effectively than before.

The cost of adopting this new method involves implementing the necessary infrastructure to enable direct access to CDP. Teams will need to ensure that their models are capable of leveraging this flexibility, which could require additional training and testing to optimize performance. However, the potential for improved automation efficiency could justify these initial investments.

This approach stops working in scenarios where the CDP cannot provide necessary data, or if the model fails to interpret the page structure correctly. While this method removes some limitations, it also introduces new dependencies on the model's understanding of DOM structures and the ability to generate effective queries.

The decision to use CDP over alternatives like Playwright is notable, as it allows for deeper access to browser functionalities without the constraints imposed by higher-level APIs. This choice emphasizes the importance of direct control when building automation tools that need to interact with complex web elements, such as closed shadow roots.

Overall, the bitter lesson learned is the importance of flexibility in agent design. This new direction encourages ongoing evaluation of how agents interact with browsers, suggesting that as models evolve, so too should the frameworks and interfaces that support them.