LLMtest

AssertLLM is a testing tool for artificial intelligence (AI) outputs. It works like a testing framework for regular computer code, but specifically for AI models. This helps developers check if their AI models are working correctly and reliably.

Benefits

AssertLLM makes testing AI efficient. It reduces the number of times it needs to contact the AI model for checks, making most tests very fast. It uses a system called Pydantic to manage data, which helps automatically check if the AI's output follows the right format and structure. AssertLLM offers more than 22 different ways to test AI results. These tests can check if the AI's text includes certain words, if it matches specific patterns, if it's valid JSON, or if it follows a required structure. It can also measure how long the AI takes to respond, how much it costs to use, and how many processing units it uses. For AI agents, it can check if the AI uses its tools correctly, if it gets stuck in loops, or if it makes calls in the right order. The tool also has built-in ways to try again if the AI gives an unexpected answer, which can happen with AI.

Use Cases

AssertLLM can be used to test AI models from many different providers like OpenAI, Anthropic, Ollama, and others. Developers can use it to ensure that an AI's response contains specific information, like a particular city name. They can also test if the AI's response is a valid piece of code or data in JSON format. Performance testing is another use case, checking if the AI responds quickly enough and stays within a budget for usage costs. For AI agents, it helps verify that they can use external tools properly and follow instructions step by step without errors.

Vibes

AssertLLM is an open-source tool, meaning its code is publicly available for anyone to use and improve. It is licensed under the MIT license, which is very permissive. It is designed for developers who want a strong way to test their AI models and agents.