An AI demo can turn a long document into a summary in seconds. It can draft a customer reply or answer questions about a set of files. Watching it work, it's easy to imagine the time your team could save.
Then you try it with your own documents.
The answer misses an exception. An old policy appears in the results. The draft sounds convincing, but someone still has to check every detail.
A demo shows what a tool can do with a particular example. Your first business project needs to show whether it helps your team do a real task, with the information and constraints they work with every day.
Choose a problem you can describe without mentioning AI
Start with something your team finds slow or difficult.
Perhaps customer service staff search several documents before answering a routine question. Someone reads incoming enquiries and works out which department should handle them. A project manager spends an hour turning meeting notes into a usable account of decisions and actions.
Describe the problem in those terms. For example:
“Our support team spends too much time finding the current guidance for customer questions.”
That gives you a task to investigate and a result to measure. “We need an AI chatbot” commits you to a tool before you've established what it needs to improve.
Check whether a simpler change would solve the problem. Better document organisation may be enough. A fixed routing rule may handle an enquiry more reliably than AI. Use AI where the task involves interpreting language or working through material that varies.
Keep the first project narrow
Suppose you want to help staff answer questions about your services.
A manageable first project might search an approved set of service documents and suggest an answer, with links to the passages it used. A staff member checks the suggestion before replying to the customer.
You can test that with a small group and a defined set of questions. You can also see where it fails.
A system expected to answer every customer question, access account records and take action across the business is much harder to assess. Each additional responsibility introduces more information to manage and more ways for an error to affect someone.
Start with a task whose output a person can check without repeating the whole job.
Check the information before testing the tool
If two documents contain different answers, decide which one is current. If important knowledge exists only in someone's head, record it. If staff shouldn't see certain customer information, the AI system must respect that restriction.
For a document search project, give someone responsibility for maintaining the source material. Remove superseded versions from the searchable collection or clearly identify them as historical.
Decide what the system should do when it cannot find an answer. Asking a colleague is a useful outcome. Producing a plausible response without evidence is a problem.
You also need to establish what information you may send to the chosen service and how that service handles it. Do this before testing with confidential business or customer records.
Test ordinary work and awkward cases
Use examples from the task you want to improve, including cases that regularly cause difficulty.
For a support assistant, test a straightforward question alongside one that is ambiguous, one involving an exception and one the documents do not answer.
Check whether it finds the right source and whether its proposed answer matches that source. A relevant-looking link does not, by itself, prove the answer is correct.
Keep a record of mistakes. That will help you distinguish problems with the documents from problems with the tool or its instructions.
If the system takes actions, such as changing a record or sending a message, test those actions separately and decide where human approval is required.
Measure the time left after checking
A draft produced in seconds may still take ten minutes to correct.
Measure the whole task, including review and rework. Compare it with how your team completes the task today. For the support example, useful measures might include time spent finding an answer and the proportion of suggestions staff can use after checking.
Ask staff what happens when the tool gets something wrong. Can they spot the error? Can they correct it? Do they know when to stop using the suggestion and handle the question themselves?
Those answers matter when deciding whether to expand the project.
Make the next decision from the trial
Set a trial period and agree what would justify continued use. Include a limit on acceptable errors as well as the improvement you hope to see.
At the end, you may decide to expand the project, change its scope or stop it. A trial that identifies an unsuitable use of AI can save you from a larger investment that delivers little.
Your first AI project should give you evidence about one business task. Choose a recurring problem, use information you can trust and leave time to assess what the tool produces.
Speak to CMSD about choosing a first AI project and defining what a useful result would look like for your business.