Crowkis: A Semantic Cache to Eliminate LLM Reruns and Reduce Costs

Research context and background

Crowkis is a specialized software tool designed to help businesses save money on large language model (LLM) expenses. It works by understanding the meaning behind user questions rather than just matching exact words. This allows it to recognize that different ways of asking the same question should result in the same cached answer, preventing unnecessary charges.

Benefits

Crowkis offers several key advantages for teams using AI models:-Cost Reduction: It identifies semantically equivalent questions, such as "How do refunds work?" and "What is your refund policy?", and serves them from a single cached answer. This stops users from being billed multiple times for the same information.-Safety and Accuracy: The tool includes five safety checks before serving any cached answer. This ensures that the response is correct, up to date, and safe to use, preventing outdated or incorrect information from reaching users.-High Performance: Built using the Rust programming language, Crowkis operates without garbage collection pauses. This results in very fast response times, with semantic hits occurring in about 0.4 milliseconds.-Easy Integration: The tool fits into existing systems without requiring code rewrites. It uses standard protocols like RESP3, gRPC, and REST, making it compatible with common Redis clients and AI agents.-Security: The community edition runs as a non-root user with limited permissions. It does not require a license, sign-up, or internet connection to function, ensuring data stays local and secure.

Use Cases

Crowkis is ideal for any application that relies heavily on LLMs to answer user questions. Common scenarios include:-Customer Support Bots: When users ask the same questions in different ways throughout the day, Crowkis ensures they get instant answers without triggering new compute costs.-AI Agents and Assistants: Tools like Claude Code can use Crowkis to check for cached answers before generating new responses. This reduces token usage and speeds up operations.-Internal Knowledge Bases: Companies can use it to manage internal queries where employees might phrase the same question differently each time.-Production Workloads: The tool is already deployed across six production environments, proving its ability to handle real-world traffic efficiently.

Pricing

Pricing details are not available in the provided information. The community edition is free to use and requires no license or sign-up.

Vibes

Public reception and specific reviews are not available in the provided information. However, the tool is currently deployed in six production workloads, indicating early adoption and practical utility.

Additional Information

Crowkis is built in Rust and available as a single Docker image for easy deployment. It supports multiple protocols including RESP3, gRPC, REST, and the Model Context Protocol (MCP). The tool is designed to run without needing to contact external servers, making it suitable for private and secure environments.