How to Stop Your AI Chatbot From Giving Customers Bad Information

An AI chatbot gives customers bad information when it is allowed to answer beyond a controlled source, when the source itself is outdated, or when a system action fails but the conversation continues as though it succeeded. The solution is not a stronger instruction to “be accurate.” Contractors need approved knowledge, limited permissions, safe refusal rules, evidence-based testing and ongoing review.

Identify the types of bad information

Not every error has the same source or consequence. Classify incidents so the team fixes the correct layer:

• Stale fact: old hours, price, promotion, territory or employee information.
• Unsupported claim: an answer not found in an approved source.
• Bad inference: a customer symptom converted into diagnosis or certainty.
• Action mismatch: the chatbot says a request or booking succeeded when it did not.
• Policy error: inaccurate warranty, cancellation, financing or refund language.
• Context error: a correct general answer applied to the wrong location, service or customer.

A prompt edit might reduce an unsupported claim. It will not correct a disconnected calendar or conflicting warranty documents.

Create a controlled knowledge base

Assemble concise, customer-facing facts rather than importing every file. Include services and exclusions, locations, hours, holiday coverage, territory rules, approved fee explanations, appointment types, preparation, financing statements and escalation contacts.

Give each item an owner, effective date and review schedule. Retire outdated versions. When different locations have different rules, label them explicitly instead of expecting the model to infer which applies.

Do not treat the public website as automatically correct. Old landing pages and promotions may conflict with current operations. Resolve contradictions before connecting them.

Use retrieval, but verify what is retrieved

Many systems search an approved source and generate an answer from the result. This can reduce unsupported responses, but retrieval can select the wrong passage or miss a relevant condition. Test the answer and the cited or logged source when the platform exposes it.

Keep source entries short and specific. A single long policy containing several exceptions is harder to retrieve accurately than clearly labeled entries. Separate public wording from internal procedures.

Write explicit answer boundaries

Document what the chatbot may state, what requires conditional language and what it must not answer. Examples:

Give the chatbot a useful “I don’t know” path

A refusal should still help. The bot can say that the team needs to verify the detail, collect the minimum information, create a task and state when a response is expected. It should offer a person when available.

Do not make uncertainty sound like failure. “Our office needs to confirm whether that equipment is within our service scope” is honest and actionable.

Separate conversation from system confirmation

The chatbot must wait for a successful response from the CRM, calendar or messaging system before stating that an action occurred. If an integration times out, it should create a fallback record or tell the visitor confirmation is pending.

Use different words for booked appointments, preferred-time requests and messages. Verify that the customer-facing confirmation matches the stored status.

Block high-risk topics and triggers

Safety, legal threats, payment disputes, complaints and complex warranty questions need designed escalation. The bot may display reviewed emergency guidance, but it should not diagnose whether a condition is dangerous.

Monitor indirect wording. “It smells strange near the heater” may require the same branch as an explicit gas-odor phrase. Test misspellings and conversations where the risk appears late.

Control pricing answers

Distinguish diagnostic fee, starting price, estimate range and final price. Store approved language with conditions. If the amount varies by location, time or membership, require the bot to identify those facts or send the question to staff.

Review all pricing interactions during the pilot. Wrong price information creates immediate trust and compliance risk.

Build a red-team test set

Ask employees to try to make the chatbot exceed its scope. Test leading questions, false premises, requests to ignore rules, claims that an employee promised a discount, demands for private data and questions about competitors.

Include ordinary ambiguity. The most likely failures come from incomplete customer descriptions, not clever attacks.

For every test, define the correct answer, acceptable fallback and prohibited response. Rerun the set after any model, prompt, knowledge, integration or policy change.

Monitor production conversations

Review every reported error, high-risk trigger, failed action, complaint and human handoff during launch. Sample routine successful chats to detect problems customers did not report.

Track wrong-answer rate by topic and severity. Also record source, cause, correction, owner and retest result. A shared incident log prevents the same issue from returning.

Create a rollback and correction process

1. Disable the affected topic, action or chatbot if necessary.
2. Preserve the conversation and system logs.
3. Identify customers and records affected.
4. Correct the source, rule, integration or provider configuration.
5. Contact customers when the error changed a promise or action.
6. Rerun regression tests before restoring coverage.

Practice how to remove the widget or switch to a form and phone number. An incident is not the time to search for access credentials.

Use versioning and regression tests

Record the chatbot version, model or provider release when visible, prompt revision, knowledge revision and integration configuration associated with each test. A change that improves one topic can weaken another. Keep a stable set of high-value questions and expected actions that runs after every material update.

Include current prices and hours, unsupported services, border locations, human requests, safety wording, hostile instructions, appointment failures and privacy questions. Compare results with the approved answer, not merely with the previous output.

When a provider changes behavior without a configurable version, increase sampling and ask for release documentation. The contractor still needs evidence that customer-facing performance remains acceptable.

Keep the previous approved knowledge and configuration available for comparison. If performance drops, the team should know which changes can be reversed immediately and which require vendor support. Document the responsible person and expected response path before launch.

Record the date the restored version returned to service and increase review temporarily. A rollback closes the immediate exposure; it does not remove the need to understand the failure.

Protect private and internal information

Use least-privilege access. A public chatbot generally does not need broad customer history, employee files, payment records or internal pricing notes. Prevent source documents from exposing passwords, private phone numbers or security procedures.

Review data retention, model-training use, subprocessors, access and deletion. The FTC has stated that AI providers must honor privacy and confidentiality commitments. Contractors should ensure their own privacy statements accurately describe the chat workflow.

Common ineffective fixes

Adding “never hallucinate” to a prompt

A slogan does not create verified knowledge or a safe system response.

Uploading more documents

More content can create more conflicts. Curate the source.

Hiding the bot’s identity

Impersonation does not improve accuracy and can increase customer confusion.

Reviewing only abandoned chats

A completed conversation may still contain the most damaging wrong promise.

Letting the provider own all corrections

The contractor owns business facts and downstream operations even when the vendor operates the model.

Frequently asked questions

Can an AI chatbot ever be 100% accurate?

No configuration guarantees perfection. Narrow scope, verified sources, testing, monitoring and human fallback reduce risk.

Should the chatbot cite sources to customers?

Links can help for policies or public guidance, but citations do not replace correct context. Internally, source traceability is valuable for review.

How often should we test?

Run a recurring regression set and retest after changes to models, prompts, knowledge, services, prices, locations or integrations.

What should happen after one serious error?

Pause the affected capability, investigate impact, correct the root cause and retest. The response should depend on consequence, not only frequency.

Can website disclaimers solve bad answers?

No. A disclaimer does not make deceptive, unsafe or inaccurate information acceptable.

Who should own chatbot accuracy?

A named business owner should coordinate source owners, operations, security and the provider. Accountability cannot sit only with “IT” or “the AI.”

Related Oivic guides

• Build a Lead-Qualifying Chatbot
• Questions Your Chatbot Should Ask
• Use AI Without Misleading Customers
• AI Privacy Risks for Home Services

Authoritative resources

• NIST Generative AI Profile
• FTC guidance on AI privacy and confidentiality
• FTC order addressing unsupported AI capability claims
• CISA artificial intelligence guidance