Published May 11, 2026
I'm not a software engineer, and I didn't have the background to design a comprehensive test suite from scratch. What I did have was access to 4 different AI models, and the system was designed to work with any model. So I asked them to do the testing for me.
The instruction was deliberately open-ended: "Here's the system. Try to break it. Run every test you can think of. Report what you find."
Each model ran independently, without human intervention. I didn't cherry-pick results or allow retries.
Each model ran 52 tests across 7 evaluation blocks: reading comprehension (can it find specific info in the compiled memory?), writing protocol (does it follow the 15 writing rules correctly?), edge cases (malformed files, Unicode, empty entries), recovery (can it follow emergency procedures after deliberate corruption?), consensus governance (does it respect the 3/3 voting system?), protected files (does it refuse to edit checksummed files?) and cross-platform mode (does it work when it can only read the compiled index, without filesystem access?).
The maximum score was 700 points.
| Model | Score | % |
|---|---|---|
| Claude Opus 4.6 | 670/700 | 95.7% |
| Claude Sonnet 4.6 | 652/700 | 93.1% |
| DeepSeek V4 | 650/700 | 92.9% |
| Claude 4.7 | 613/700 | 87.6% |
Average: 92.6%, with zero system failures and zero data corruption.
The newest model (Claude 4.7) scored the lowest, 87.6%, but it caught 13 bugs I wouldn't have found otherwise. It probed harder and found more edge cases, and the system is stronger because of it. For my purposes that lower score was the most useful result of the four.
I fixed every bug the models found, and the system improved with each test run. By the end, all 103 unit tests passed at 100%.
Testing with AI doesn't replace QA. What it adds is scale. One person can run maybe 20 tests in a session; an AI runs 52, documents every result and suggests fixes on its own. And with 4 different models, each one finds things the others miss.
The full test methodology and results are published on GitHub, so you can run the same 52 tests against a model I didn't include and compare.
Javier
The same system, tested by four models from two vendors.
EIDARA is the open-source project that went through this testing process. See it on GitHub or read the full test results.