The most common setup mistake I see with AI voice agents isn't a bad prompt or a robotic voice. It's a good-faith upload. Somebody gathers every PDF the company owns — the public FAQ, the price sheet, the internal discount matrix, the escalation playbook, the draft policy that never shipped — drops the whole pile into the knowledge base, and calls it "training the AI on our business." What they've actually built is an agent that will read any of those documents aloud to anyone who dials the number and asks the right question.
Retrieval-augmented generation solved the problem everyone worried about in 2023: models making things up. It replaced it with a quieter problem almost nobody budgets for. Once your agent answers from your documents, the question stops being "is the answer accurate?" and becomes "was this caller supposed to hear that answer at all?"
Retrieval is an access-control problem wearing a search costume
OWASP's 2025 Top 10 for LLM applications gives this its own entry — LLM08, Vector and Embedding Weaknesses — and the language is blunt: inadequate or misaligned access controls can lead the model to "retrieve and disclose personal data, proprietary information, or other sensitive content." The recommended fix isn't a better prompt. It's "permission-aware vector and embedding stores" with "strict logical and access partitioning of datasets," plus tagging and classifying data in the knowledge base to control access levels.
That's the right frame. A vector store does one thing: it finds the chunks most semantically similar to the query. Similarity has no opinion about who's asking. If the internal retention-offer sheet is the closest match to "what's the best deal you can give me if I'm thinking about canceling," the retriever will hand it to the model, and a well-behaved model will helpfully summarize it. Telling the model "don't reveal internal pricing" in the system prompt is a request, not a control. I made the same argument about tool access a few weeks ago, and it holds here: the only document an agent reliably won't disclose is one it can't retrieve.
Why the phone makes this worse
On a website, a leaked answer at least leaves a trail — a page view, a chat transcript, a link someone can audit. A voice call is synchronous and screenless. The caller can't see which document an answer came from, and neither can your team in the moment. The spoken sentence is the commitment: once your agent says "we can waive that fee for you," you're having a different conversation with that customer.
The phone also invites probing. Anyone can call, call back, rephrase, and try again — no login, no rate limit a human would notice. And the inputs you ingest are attack surface, not just reference material. Researchers at UT Austin's SPARK Lab demonstrated an attack they called ConfusedPilot against RAG systems like Microsoft 365 Copilot: plant crafted content in a document the system indexes, and the AI treats it as instructions — with the corrupted answers able to linger even after the malicious document is removed. Their recommended mitigations read like a governance checklist: limit who can upload or modify indexed documents, audit stored data, and segment sensitive information. None of those are model features. All of them are knowledge-base design.
What governance actually looks like
Strip the vendor language away and a defensible knowledge layer for voice needs four properties:
- Default-deny. A newly uploaded document should be searchable by nobody until someone decides who it's for. The default-open alternative means your exposure equals whatever the last person uploaded.
- Scoping per consumer, not per account. The after-hours sales agent, the billing queue, and the support line should each retrieve from a declared subset. "The company's knowledge" is the wrong unit.
- Hard tenant isolation. OWASP calls out cross-tenant context leakage in shared vector databases explicitly. If your provider pools embeddings across customers, ask how partitioning is enforced — and listen for whether the answer is architecture or policy.
- Answers traceable to a source. If you can't say which document produced a spoken answer, you can't fix the wrong one.
How we approached it at Dial800
This is the design AI Knowledge is built around, and I'll be specific rather than slogan-y. Every document is tagged with Document Categories, and each AI Voice Agent — and each contact center queue — is assigned the categories it may answer from, "and only those." A document with no categories isn't searchable by any agent until someone tags it. The document store is isolated per account and never mixes with another customer's content. Answers come from your uploaded documents, not the open internet.
The part I like most is that one store serves two consumers. The same categories that scope an autonomous agent also scope AI-suggested answers for human agents in CallCenter, so a billing queue never sees shipping answers — and when you update a policy document, every agent assigned to its categories answers from the new version. One source of truth, one set of gates, for bots and people alike.
I'll be honest about the limits, too. Categories decide who can retrieve a document; they don't make a bad document good. If you tag the confidential discount sheet "pricing" and assign "pricing" to your public-facing agent, the system will do exactly what you told it to. Governance is a mechanism, not a judgment — and controlling who can upload and tag documents still matters, which is precisely the ConfusedPilot lesson.
The audit worth doing this week
Pull the list of every document your voice agent can retrieve from. For each one, ask a single question: would I be comfortable if a stranger called at 2 a.m. and heard this read aloud? Anything that fails the test shouldn't be "handled by the prompt." It should be out of that agent's reach entirely.
Your agent should know exactly what its job requires. Everything beyond that isn't intelligence. It's exposure.