title: "AI Center of Excellence for Small Teams (No Consultants)" slug: ai-center-of-excellence excerpt: "What an AI center of excellence actually does, why most guides assume a Big 4 budget, and how a 10-200 person company builds one with existing staff instead." description: "How a small or mid-size company builds an AI center of excellence without hiring an AI team or a Big 4 consultancy, using employees it already has." target_keywords:
- ai center of excellence
internal_links:
- /program
- /builder-scan
- /referent-residency
- /blog/ai-training-for-employees
- /blog/ai-certification-programs
sources:
- https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ai/center-of-excellence
- https://www.ibm.com/think/topics/ai-center-of-excellence
- https://www.oracle.com/artificial-intelligence/ai-center-excellence/
An AI center of excellence (CoE) is the internal function that sets AI standards, reviews use cases, and keeps automations running after the first pilot fades. Microsoft's Cloud Adoption Framework and IBM both describe it the same way: a dedicated team or working group that governs how AI gets built and reused across the company. The problem is that every widely cited version of this assumes you are hiring an AI engineering team or paying a consultancy to stand it up.
A company with 10 to 200 people usually cannot justify either cost. The function itself, standards, reviewed use cases, ongoing maintenance, does not require a new department. It requires 5 to 8 existing employees who go through a structured build sprint on their own jobs, plus one of them installed afterward as the internal AI referent who owns governance going forward. Same job, different delivery model: capability transfer instead of a new hire.
What is an AI center of excellence?
A CoE is the standing function inside a company that decides which AI use cases get built, sets the rules for how they get reviewed before going live, and keeps them running after the person who built them moves on. Oracle's framework frames it as governance plus enablement: it is not one project, it is the structure that lets a company keep shipping AI projects without every single one starting from zero.
Most published descriptions of a CoE, from cloud vendors, consultancies, and enterprise wikis, describe it at the scale of a company that already has a data science function, a dedicated AI budget line, and headcount to spare. That is accurate for a Fortune 500 company. It is not a useful blueprint for a 40-person services firm trying to figure out where to start.
How do you build an AI center of excellence?
At any company size, the function needs the same four pieces:
- A working group of people who actually use the tools, not just IT or leadership signing off from the side.
- A review step before an automation touches a real workflow, so nobody ships something nobody else can maintain.
- A shared library of what has already been built, so the fifth automation does not reinvent the first one.
- One person accountable for keeping the whole thing alive after the initial excitement wears off.
Large companies staff this with a dedicated team, often assembled by a consultancy engagement that costs six figures before anything ships. A smaller company builds the same four pieces differently: pick 5 to 8 employees across different functions, run them through a 90-day program where each person automates real work on their own job, then designate one of them as the internal referent who owns the review step and the shared library afterward. The Builder-Operator Program is built around exactly that sequence, and it starts with a Builder Scan to work out which roles at your company are worth picking first.
What does an AI center of excellence do day to day?
Once it exists, a CoE's daily work is unglamorous and mostly about maintenance:
- Reviewing a new automation idea before it touches a live workflow.
- Checking that an existing automation still works after a tool update or a data source change.
- Answering "has someone already built this" before a team starts from scratch.
- Training the next employee who wants to build something on their own job.
- Deciding what gets retired when a workflow changes and the automation no longer matches it.
None of that requires a large team. It requires someone whose job explicitly includes it, which is the gap most companies hit: they run one successful pilot, nobody owns what happens next, and the automation quietly breaks six months later with no one assigned to notice.
When does a company actually need one?
Not every company with a ChatGPT subscription needs a formal CoE on day one. The signal to watch for is not company size on its own, it is repetition: the same manual workflow shows up in two or three departments, or the same automation question keeps getting asked by different people who do not know someone already answered it elsewhere.
A few concrete signs it is time:
- More than one team has built its own AI workaround for a similar problem, without knowing about the others.
- An automation someone built six months ago quietly stopped working and nobody noticed until a customer or a manager did.
- Leadership wants to know "what are we actually doing with AI" and the honest answer is a scattered list, not a shared one.
- New hires ask who owns AI tooling internally and the answer is "nobody, really."
If none of that describes your company yet, a single pilot project run by one motivated employee is still the right next step, not a full CoE. The four pieces described above (a working group, a review step, a shared library, one accountable owner) become worth building once the scattered-workaround pattern shows up more than once.
Do you need a Big 4 consultancy or a new AI team to run one?
No, for a company this size. A consultancy engagement or a new AI engineering hire makes sense when the scale of use cases justifies a dedicated budget line, typically the point where a company is running AI across many business units at once. For a single business with a handful of clear automation targets, the same governance function runs on existing staff who have been through a structured build process, not a fresh headcount line.
That is the exact gap between how most CoE guides are written and how most companies our size actually operate. If your team has already sat through a generic AI certification or training session and nothing changed operationally, the missing piece usually is not more instruction. It is a group of people with a mandate, a review process, and one person accountable for what happens after the training ends. Apply to the Builder-Operator Program if that describes where your company is stuck.
AI center of excellence vs. Builder-Operator Program: what's the difference?
Traditional AI CoE (enterprise guides) | Builder-Operator Program | |
|---|---|---|
Who staffs it | New AI engineering hires or a consultancy team | 5-8 employees you already have |
How it starts | Six-figure consulting engagement or new department | 90-day cohort build sprint |
What "governance" means | Formal committee, often slow to convene | One designated internal referent, backed by playbooks |
Who owns the result | The vendor or consultancy relationship | The company, permanently |
Ongoing cost | Dedicated team salaries or a retainer | Optional Referent Residency for continuity |
Neither model is wrong for the company it is built for. The enterprise model fits a company already running dozens of AI use cases across departments. The capability-transfer model fits a company that wants the same governance function without adding headcount or signing a consulting contract first.
FAQ: AI center of excellence
How many people does an AI center of excellence need? Enterprise guides often describe a dedicated team of several people. For a 10-200 person company, 5 to 8 employees who have been through a structured build process is usually enough to cover the review, maintenance, and knowledge-sharing work, especially with one person designated as the internal referent.
Can a small company really run a CoE without hiring an AI engineer? Yes, if the goal is governance and maintenance of existing automations rather than building foundation models or custom infrastructure. Most companies at this size need the former, not the latter.
Is this the same as taking an AI certification course? No. A certification teaches an individual concepts and tools. A CoE, at any scale, is an ongoing internal function with a review process and an owner. Our breakdown of AI certification programs covers that distinction in more depth.
What happens if nobody owns the CoE function after the first project ships? The most common failure mode: the first automation works, gets praised, and then breaks a few months later because no one was assigned to check on it. Designating an internal referent before that happens is the single most useful step on this list.
Written by Tileo, an operator who learns AI by running businesses with it.