We had 15 skill files documenting every Synthesis CLI command — syntax, options, example invocations, expected output. We wrote them carefully. We loaded them into the agent's context. We assumed the agent would use them.
Then we ran a benchmark.
The CLI condition was the worst-performing integration in the entire test. Worse than no integration at all.
I have been watching the AI agent space closely for the past year. The frameworks are impressive. The orchestration tools are clever. The models are increasingly capable. And yet, most agent deployments I see make the same quiet mistake: they treat the knowledge problem as solved.
It is not solved. It is barely addressed. And until it is, all the reasoning capability in the world will not make your agents reliably useful.
We build Synthesis — a knowledge indexing tool whose purpose is to make everything in a workspace searchable. We had been running it on our own workspace for months and the coverage number looked excellent: 99.6%. Nearly perfect.
That number was a vanity metric. It told us exactly what we wanted to hear while hiding what we needed to know. The real asset coverage was 15.2%.