AI Center of Excellence: The Structure That Actually Works

The AI Center of Excellence (CoE) has become standard enterprise structure. Most are paper organisations: a director-level lead, a small team, a remit to "drive AI strategy," and an output that is mostly slide decks and vendor evaluations. They produce thinking. They do not produce agents.

The CoEs that work look different. This is the structure, the roles, and the operating model that actually produce working AI agents at scale.

What a working CoE is not

The patterns that fail:

• Pure strategy function: produces policy documents, slide decks, vendor lists. No actual agents.
• Captive engineering team: owns and builds every agent, becomes the bottleneck for every business unit.
• Compliance-led: focuses on policy and review; treats agent development as someone else's problem.
• Innovation lab: prototypes interesting things that never reach production.

Each of these patterns adds payroll without producing operational AI capability.

What a working CoE looks like

The functional structure that produces working agents at scale:

Platform team (technical operations):

• Owns the AI platform (whether built or bought)
• Operates the runtime, the gateway, the audit infrastructure
• Provides agent development primitives that business units use
• Sets technical standards for agent development
• Sizes: 2-6 engineers for a 20-50 agent estate

Governance team (risk and compliance):

• Owns the AI Act, GDPR, and sectoral compliance posture
• Reviews and signs off on agent risk classifications
• Maintains the DPIA process and the audit evidence pack
• Liaises with the DPO, legal, and information security
• Sizes: 1-3 people for a 20-50 agent estate

Solutions architects (embedded in business units):

• Work with the business unit to identify agent opportunities
• Lead agent design and the implementation project
• Coordinate with platform and governance teams
• Are members of the CoE but report partially into the business unit
• Sizes: 1 solutions architect per 2-5 business units

CoE leadership:

• Sets strategy and prioritisation
• Manages the agent portfolio
• Reports to executive leadership on outcomes
• Sizes: 1 director-level lead, plus a chief of staff

Total CoE size for a typical mid-size enterprise: 8-15 people, scaling with the agent estate.

How the work actually flows

The operating model that produces agents:

Demand intake: business units identify candidate AI use cases through a structured intake process. The CoE's solutions architects help shape the requests into agent specifications.

Prioritisation: the CoE leadership prioritises the candidate agents based on value, feasibility, and strategic fit. Quarterly portfolio review with executive sponsors.

Design: the solutions architect leads agent design with the business unit, the platform team for technical feasibility, and the governance team for compliance posture.

Build: the solutions architect and the business unit's team build the agent on the platform. The platform team supports with integrations and tooling. The governance team reviews compliance evidence as it is produced.

Launch: the agent goes live with the business unit. The solutions architect transitions the agent to the business unit's ongoing operation.

Operate: the business unit owns the agent's day-to-day operation. The platform team operates the underlying infrastructure. The governance team monitors compliance evidence.

Review: quarterly portfolio review. Agents that are not delivering are retired. New opportunities are picked up.

This is the operating cadence. The CoE does not own every agent; the business units do. The CoE enables and governs.

The reporting line that works

The CoE typically reports into the CTO, the CIO, or directly to the CEO. Each placement has trade-offs:

• Under the CTO: strong technical execution, sometimes too engineering-centric, may not have the executive air time it needs.
• Under the CIO: good integration with existing IT operations, sometimes too IT-centric, may not have business-unit access it needs.
• Direct to CEO: highest visibility, sometimes lacks operational integration with IT and the business, vulnerable when the CEO's attention shifts.

For most enterprises a CTO or CIO line works best, with explicit board-level reporting on outcomes for visibility.

The skill mix in the CoE

Engineering skills:

• Platform engineering: distributed systems, observability, security
• AI/ML engineering: prompt engineering, model evaluation, RAG architecture
• Integration engineering: APIs, data pipelines, MCP servers

Non-engineering skills:

• Solutions architecture: business analysis, agent design, project leadership
• Governance: AI Act, GDPR, sectoral regulations, audit
• Change management: training, communications, adoption tracking
• Product management: portfolio prioritisation, roadmap, lifecycle

The mix matters. CoEs heavy on engineering and thin on the rest produce agents that work technically but do not get adopted. CoEs heavy on governance and thin on engineering produce policies that the agent development teams ignore.

The interfaces that matter

The CoE works through interfaces to other functions:

Business unit leads: own the agents that affect their domain; the CoE enables, the business unit owns the outcome.

DPO and privacy office: review compliance posture; the CoE's governance team is the primary interface.

CISO: AI security is part of cyber risk; the CoE's platform team is the primary interface.

Procurement: AI vendor evaluations; the CoE leads the technical and compliance assessment.

HR and L&D: AI skills development across the workforce; the CoE provides the technical input.

Communications: internal and external messaging about AI; the CoE provides the substantive content.

Each interface needs a clear protocol. Without it the CoE either does work that should belong elsewhere or gets bypassed when other functions act unilaterally.

How a CoE transitions through enterprise maturity

Year 1: CoE is forming. 3-6 people. Focus is on standing up the platform and proving the first 3-5 agents. Mostly central execution.

Year 2: CoE expands. 6-10 people. Solutions architects start embedding in business units. The first 10-20 agents are operational. Governance practices are documented.

Year 3: CoE is operational steady state. 10-15 people. 20-50 agents in production. Business units own and operate their agents with CoE enablement. Portfolio management is the main CoE activity.

Year 4+: The CoE either contracts (because the platform is mature and business units are self-sufficient) or expands (because the agent estate has grown to a scale that needs more central infrastructure). Both can be right depending on enterprise scale and culture.

What the CoE measures itself on

The metrics that drive the CoE's accountability:

• Number of agents in production (count and trajectory)
• Aggregate business outcomes attributable to agents (deflection rates, cycle time reduction, cost savings)
• Compliance evidence quality (audit readiness on representative samples)
• Time from agent intake to production
• Cost per agent in operation
• Adoption rate of deployed agents
• Failed agent rate (agents retired without delivering expected value)

These are operational metrics, not strategic ones. Strategic value is captured in the business outcome metric. The other metrics ensure the operational discipline that produces strategic value.

Where AgentWorks fits in this picture

The platform is the operational foundation the CoE uses. The platform team operates the platform; the solutions architects build agents on it; the governance team reviews the compliance evidence the platform produces.

The platform does not replace the CoE. It enables the CoE to focus on the work that humans need to do (design, prioritisation, governance, change management) by handling the work that platforms handle better (runtime, observability, integrations, audit logs).

For the broader story on how AgentWorks supports enterprise AI programs see the AI workforce platform overview. The CoE structure above is the human and organisational layer that the platform supports.

The honest summary: a CoE without a platform is a thinking organisation. A platform without a CoE is engineering without strategy. Both are needed; both look different from the paper org charts most enterprises drew up two years ago and now need to refresh.