Can AI Actually Answer Questions
About Your Company Data?

A director asks why they cannot just type which customers are at risk of churning into a chat box and get an answer. It is a reasonable question, the technology genuinely exists, and the honest answer is: yes, for some kinds of question, if specific groundwork is done first.

The gap between the demo and a system your business can rely on is almost entirely that groundwork. Here is what actually works, what does not, and where the money goes.

The Question Everyone Is Asking

Two things drive this. Copilot is now embedded in tools people already use, so asking questions of data feels like it should be standard. And everybody has seen a demo where somebody types a question and a chart appears.

What the demo omits is that it ran against a small, clean, well-named dataset built for the demo. Your data is none of those things, and the difference is not cosmetic.

Two Different Things Called the Same Thing

When people say AI over our data they usually mean one of two quite different problems with different solutions and different failure modes.

Conflating these is the most common reason a project underdelivers. They need different engineering, and a vendor who does not distinguish them is worth questioning.

Asking Questions of Numbers

The AI does not read your database. It writes a query against a semantic layer, and its accuracy depends on how well that layer describes the business.

Which means the work is modelling work. Fields named the way the business speaks. Measures with descriptions saying which definition they use. Synonyms so turnover and revenue reach the same field. A structure where relationships carry meaning.

Industry guidance puts roughly 80% of answer quality down to model quality rather than the AI. That matches what I see: the same tool over a well-built model is genuinely useful, and over a flat table of raw exports it produces confident nonsense.

Same question, two models

Question: "What was margin by region last quarter?"

Over a raw export:
  Columns: COL_A, COL_B, REG_CD, AMT_1, AMT_2
  -> AI guesses AMT_1 is revenue. It is gross of tax.
  -> Answer looks plausible. It is wrong by the VAT rate.
  -> Nobody checks, because it looked plausible.

Over a described model:
  Measure: Net Margin %
  Description: "Profit after COGS and returns,
                divided by net revenue. Excludes freight."
  Dimension: Region (synonyms: territory, area, zone)
  -> AI selects the right measure and states which it used.

The second case is not smarter AI. It is the same AI given something worth reading.

Asking Questions of Documents

This is retrieval-augmented generation: your documents are split into passages, indexed, and when somebody asks a question the system retrieves the most relevant passages and has the model answer using them.

It works well for questions whose answer sits in one place — a notice period, a policy clause, a specification. It works badly for questions requiring synthesis across many documents, because retrieval returns a limited set of passages and anything outside that set does not exist as far as the model is concerned.

The failure mode is specific and worth knowing: the system finds a passage that looks relevant, answers confidently from it, and the actual answer was in a clause it did not retrieve. There is no signal to the user that this happened.

Why Most Attempts Fail

The Guardrails That Matter

• Citations on every answer. Which measure, which filters, which document, which page. Non-negotiable for anything used in a decision.
• An honest refusal path. The system must be able to say it does not know. A model that always answers will always answer, including when it should not.
• Security that follows the user. The AI must respect the same row-level and document permissions the person already has, or it becomes a very efficient way to leak data.
• A logged question history. You need to see what people actually ask. It is the best available signal for what to build next, and for spotting where answers are going wrong.
• A scored evaluation set. Twenty to fifty real questions with known correct answers, re-run whenever anything changes. Without it you have no idea whether a change improved or degraded the system.

What It Actually Costs to Run

Two cost lines, and people usually anticipate only one.

The visible one is inference: every question sends tokens to a model and you pay per token. For an internal tool used by a few dozen people this is usually modest, often less than the licensing for the BI platform it sits alongside. Document systems cost more per question than structured ones, because retrieved passages make the prompts considerably larger.

The one people miss is maintenance. A retrieval system needs reindexing as documents change, and a semantic model needs its descriptions and synonyms kept current as the business evolves. Neither is large, but both are ongoing, and a system nobody maintains degrades quietly — answers get worse gradually rather than breaking visibly, so nobody raises it until trust is already gone.

Budget for someone owning it. Not full time, but named, with the evaluation set re-run on a schedule so degradation shows up as a number rather than as a vague sense that it used to be better.

What We Would Do

We start by separating the two problems, because the answer differs. For structured questions the work is semantic modelling: naming, descriptions, synonyms, structure — then testing against the questions people actually ask. For documents it is chunking strategy, retrieval tuning, and citation.

In both cases we build the evaluation set before the system, so there is an objective measure of whether it works rather than an impression formed in a demo.

And we will tell you when the honest answer is that a well-built report would serve you better than a chat interface. Sometimes it would.