The AI Capability Checklist

    Ten concrete capability tests for evaluating AI in business software, from reading customer history to executing a workflow with an audit trail.

    How do I evaluate AI in business software?

    Evaluate AI by capability, not by label. Ten tests separate the categories: can it read history, understand context, create documents, send them, schedule work, answer customers, trigger workflows, call other systems, act within permissions, and produce an audit trail of everything it did.

    Key takeaways

    • Capability tests beat feature lists in every software evaluation.
    • Reading context is the first threshold most products fail.
    • Executing and auditing is the second.
    • If a demo cannot show the outcome in the record, it is a draft.

    The ten tests

    • 1. Read a specific customer's complete history.
    • 2. Understand the current stage of a job and what comes next.
    • 3. Produce a document from real business data, not a generic template.
    • 4. Deliver that document — send, share or route for signature.
    • 5. Create or modify money records such as estimates and invoices.
    • 6. Book, move or cancel work against real availability.
    • 7. Answer a customer directly through a real channel.
    • 8. Trigger the next workflow step automatically.
    • 9. Operate within one user's permissions and one organization's data.
    • 10. Write every action to an audit trail that names the actor.

    How to run the evaluation

    Bring one real scenario to every demo — the same scenario each time. Ask the vendor to complete it live, then ask to see the resulting record and the audit entry. Scoring becomes obvious very quickly.

    Why the last two matter most

    Permissions and audit are not paperwork. Without them, an AI that can act is an unbounded liability, and no compliance program will accept it. With them, AI actions are more traceable than manual ones, because nothing depends on someone remembering to write it down.

    Chatbot vs Assistant AI vs Operational AI

    Three different classes of software, all marketed with the same two letters.

    CapabilityChatbotAssistant AIOperational AI
    Answer general questionsYesYesYes
    Summarize a conversationLimitedYesYes
    Read a specific customer's historyNoSometimesYes
    Understand job and job-stage contextNoSometimesYes
    Draft an estimate from real pricingNoPartialYes
    Create and send a contract for signatureNoNoYes
    Create and send an invoiceNoNoYes
    Book or move an appointmentNoNoYes
    Trigger a downstream workflowNoNoYes
    Act within permissions and audit rulesN/ARarelyYes

    Frequently Asked Questions

    More in AI Capabilities

    How to tell a chatbot from an assistant from operational AI.

    Browse AI Capabilities
    Share this page