Parity: Auto-evals for harness changes

Parity is an AI tool that helps developers automatically evaluate changes made to AI behavior. When developers update things like prompts, instructions, or other parts of how an AI works, Parity steps in to check these changes. It looks for existing tests, finds where more testing is needed, and suggests new tests that fit with the systems already in use. Parity does not run tests itself but works with the testing systems developers already have.

Benefits

Parity makes evaluating AI behavior changes easier and more thorough. It helps ensure that new changes do not break existing AI functions and that new behaviors are properly tested. By identifying coverage gaps, it prevents potential issues from being missed. It also helps integrate new evaluations smoothly into existing workflows.

Use Cases

Developers can use Parity when making changes to AI models, especially those involving prompts, instructions, or other elements that define the AI's behavior. It is useful for projects that use AI and require ongoing testing and validation of AI responses. Parity can be integrated into the development process to automatically analyze pull requests that modify AI behavior, ensuring that changes are safe and well-tested before they are merged.

Vibes

Parity supports several evaluation platforms, including Promptfoo, LangSmith, Braintrust, and Arize Phoenix. It offers a bootstrap mode for situations where a safe evaluation target cannot be immediately identified. Users can install Parity via pip and initialize it in their repository. The tool then guides users through configuration and analysis, with commands likeparity initto set up the project andparity doctorto check the environment. Specific commands likeparity run-stage 1,parity run-stage 2, andparity run-stage 3allow for step-by-step analysis of changes.

Additional Information

Parity can be installed using pip with the commandpip install parity-ai. After installation,parity initsets up the necessary configuration files and a GitHub Actions workflow. Users need to provide API keys as GitHub secrets and fill in context files. When a pull request that alters AI behavior is opened, Parity analyzes it. Merging changes after adding theparity:approvelabel allows Parity to write approved evaluations back to the platform.