RuleProbe

RuleProbe is a tool that helps make sure AI coding assistants follow the instructions they are given. When developers use AI to help write code, they give the AI specific rules to follow, like how to name variables or whether to include tests. RuleProbe checks if the AI actually followed these rules in the code it produced.

Benefits

RuleProbe provides proof that AI coding agents followed instructions. It reads the same rule files the AI uses and checks the AI's code output. This ensures that the code meets specific requirements. It offers deterministic checks, meaning results are clear pass or fail with evidence, rather than relying on subjective AI judgment. The tool supports many built-in checks for things like naming conventions, forbidden code patterns, code structure, and ensuring tests are written. It can also perform type-aware checks for languages like TypeScript and JavaScript, and has some support for Python and Go. RuleProbe can be set up to work offline for most of its functions, which helps keep things secure.

Use Cases

This tool is useful for developers who use AI coding agents and want to ensure the code generated meets their project's standards. It can be used to verify code quality and adherence to coding guidelines. RuleProbe can be integrated into GitHub workflows to automatically check code on pull requests and provide feedback. It can also compare the output from different AI agents to see which one followed the instructions better. For more complex or subjective rules, RuleProbe can optionally use AI to break them down into more concrete checks.

Vibes

RuleProbe offers deterministic checks, providing binary pass/fail results with file paths and line numbers as evidence. It avoids LLM evaluation or judgment calls for its core verification process. The tool supports extensive rule sets, with 53 built-in matchers covering various aspects of code quality and structure. It also allows for custom rules and configurations through its configuration files. For languages like TypeScript and JavaScript, it can perform type-aware checks. For Python and Go, it uses Tree-sitter for certain checks. The tool is designed to work offline for most features, enhancing security.