ai-operating-economics

AI Operating Economics

Overview

AI initiatives are operating interventions, not merely model purchases or ROI spreadsheets. Their value depends on what work changes, who benefits, what quality or risk changes with it, what the complete intervention costs, and whether the organization can observe and govern those changes.

This skill provides the cross-domain decision spine for evaluating an AI-enabled workflow. It does not replace financial modeling, product measurement, statistical inference, agent evaluation, runtime operations, or AI governance. It makes those inputs meet in one accountable decision record.

The core question is not “Did the model make people faster?” It is: “What changed in this workflow, for whom, at what full cost, with what outcome and countermetric evidence, and what authority should the organization grant next?”

Entry Points

When to Use

Load this skill when the user needs to:

• Build an evidence-backed business case for an AI use case or agentic workflow.
• Decide whether an AI pilot should scale, remain bounded, be redesigned, or stop.
• Review claimed AI productivity, savings, adoption, or transformation results.
• Design an AI value-realization or post-launch outcome review.
• Connect model and tool spend to workflow outcomes and worker or customer effects.
• Compare AI options while accounting for measurement uncertainty and non-comparable evidence.
• Prepare an executive, product, portfolio, or lifecycle decision about an AI-enabled intervention.

When Not to Use

Non-Negotiable Reasoning Rules

1. Workflow evidence beats model evidence. A benchmark, demo, or vendor claim does not establish value in the target workflow.
2. Speed is not value. Time saved can be spent on lower-value work, offset by review and exception handling, or enable higher-value work. Measure the business or user outcome directly.
3. Averages are not enough. Inspect worker, user, task, geography, tenure, risk, and quality slices. An aggregate gain can hide a subgroup loss.
4. Every benefit metric needs a countermetric. Pair throughput or cost with quality, safety, customer, worker, privacy, or reliability measures appropriate to the workflow.
5. Token cost is not total cost. Include model calls, tools, retrieval, storage, networking, observability, engineering, human review, change management, governance, and unused committed capacity when material.
6. Evidence classes must stay separate. Label observed results, causal estimates, inferences, vendor-reported findings, stakeholder assertions, and normative requirements distinctly.
7. Missing evidence is a decision input. Do not turn an unknown into a favorable assumption. Record the gap, owner, consequence, and next evidence needed.
8. Authority follows evidence. A positive pilot does not justify unrestricted autonomy. Scale capability and authority in bounded slices with explicit reversal conditions.
9. Do not manufacture precision. Use ranges, scenarios, sensitivity, and confidence where inputs are uncertain. Do not rank non-comparable studies or vendors.
10. The decision is reversible only if the artifact says how. Record the stop trigger, rollback or containment path, decision owner, and review date.

Core Workflow

Use this sequence for an AI initiative review. Load the detailed method and the evidence-record template when the task requires a durable artifact.

Quick Start by Need

Choose Review Depth

1. Define the intervention and decision

Name the workflow, population, task boundary, intervention mode, baseline, decision sought, and decision owner. State whether the AI assists, recommends, routes, executes, or replaces/removes work. Define what remains human-controlled.

Do not begin with the model name or a claimed percentage. Begin with the work that changes and the decision the evidence must support.

2. State the value hypothesis

Write a falsifiable hypothesis:

For [population] doing [workflow], [intervention] will change [outcome] by [direction/range] without exceeding [countermetric boundary], at [full operating cost boundary], compared with [baseline], over [period].

If the proposed outcome is only “productivity,” decompose it into the actual customer, employee, operational, financial, or mission outcome. If the outcome cannot be observed or credibly proxied, mark the initiative measurement-incomplete rather than inventing a proxy.

3. Build the outcome and countermetric map

Define:

• Primary outcome: the result the initiative exists to improve.
• Leading indicators: early evidence that the mechanism is operating.
• Countermetrics: quality, safety, customer, worker, privacy, reliability, or equity measures that could worsen.
• Adoption and substitution measures: who uses the system, what work changes, and what work is displaced or added.
• Guardrail thresholds: contextual limits with an owner and response.

Route metric definitions and instrumentation plans to product analytics. Route statistical or causal design to data science. This skill owns the connection between the evidence and the decision, not the detailed statistical method.

4. Establish the full economic boundary

Record both:

• Marginal economics: what changes when one more task, user, or workflow unit is served.
• Fully loaded economics: the costs required to make the intervention available and govern it.

At minimum consider inference, tool use, retrieval, storage, data transfer, observability, engineering, evaluation, human review, training, support, change management, governance, security, and committed capacity. Separate fixed, variable, step-function, and avoided costs. Define the denominator precisely: task, resolved case, completed workflow, active user, customer outcome, or another meaningful unit.

Route the detailed model to capacity-and-cost-engineering or financial-modeling. Never divide total spend by an undifferentiated request count when requests have materially different resource or outcome profiles.

5. Design the evidence comparison

Choose the strongest feasible comparison before interpreting results:

• Randomized or staggered rollout when feasible.
• Matched or difference-in-differences comparison when appropriate.
• Within-workflow baseline with explicit pre-period and seasonality limits.
• Controlled pilot with a documented task and population boundary.
• Descriptive before/after evidence only when stronger designs are infeasible, labeled accordingly.

Record selection effects, learning effects, concurrent initiatives, task-mix changes, worker self-selection, quality measurement gaps, and changes in pay or incentives. If the comparison cannot support the requested claim, narrow the claim rather than upgrading the method rhetorically.

6. Segment before aggregating

Report the overall result and inspect slices that could change the decision:

• Worker experience, skill, role, and training status.
• Task complexity, risk, volume, and exception rate.
• Customer or user segment.
• Geography, language, accessibility, and relevant demographic groups when lawful and appropriate.
• Human-review burden and escalation path.
• Quality, safety, and error severity.

Treat heterogeneous effects as a finding, not noise to average away. A tool that helps novices while harming expert quality may need differentiated assistance modes, not universal rollout.

7. Classify the evidence

For every material claim, label it:

Keep the source, access date, scope, version, caveat, and permitted interpretation with each claim. Load references/source-index.md for the research basis and evidence boundaries.

Minimum Claim Ledger

For each material claim, record: claim, evidence class, source and scope, what it supports, what it does not support, open challenge, and permitted language. Keep unknown claims visible; do not let a source URL or vendor report stand in for direct workflow evidence.

Minimum Decision Record

Every completed review must expose, in one durable artifact: the intervention and population, value hypothesis, primary outcome, countermetrics, comparison and limitations, cost boundary, relevant slices, evidence classes, missing evidence with owner, disposition, authority limit, reversal path, and review trigger.

Disposition Quick Pick

8. Produce a bounded decision

Choose exactly one primary disposition:

• Scale: evidence supports expansion within a named scope and authority boundary.
• Constrain: value is plausible, but cost, quality, risk, or distributional effects require limits.
• Redesign: the mechanism or workflow needs modification before another test.
• Hold: evidence is insufficient for the requested decision; specify the missing evidence.
• Retire: observed value is absent or countermetrics exceed acceptable bounds, with a transition path.
• Exception: proceed despite a named gap only with an accountable human approver, expiry or revisit trigger, and containment plan.

Closure Conditions

• Scale: next population, authority slice, owner, and review trigger are recorded.
• Constrain: the boundary, quota, human-review rule, and condition for expansion are recorded.
• Redesign: the changed mechanism, rerun comparison, and new acceptance boundary are recorded.
• Hold: the missing evidence, owner, method, and due trigger are recorded.
• Retire: transition, affected-person protection, decommissioning, and retained learning are recorded.
• Exception: named human approver, scope, expiry, containment, and revisit condition are recorded.

A decision is incomplete without an owner, review date or trigger, evidence gaps, and reversal path. Route launch or runtime consequences to the appropriate specialist skill.

9. Close the learning loop

At the review date, compare expected versus observed outcomes, cost, quality, worker or user effects, adoption, and incidents. Preserve the updated evidence record and state whether the prior hypothesis was supported, weakened, refuted, or still unresolved. Feed verified incidents and near misses into evaluation and governance work rather than treating them as anecdotal follow-up.

Load-on-Demand References

Common Pitfalls

• Treating an AI benchmark, speed increase, or demo as evidence of business value.
• Treating a vendor survey as an audited financial result or causal estimate.
• Reporting one average while omitting worker, task, quality, or customer slices.
• Calling token spend “AI cost” while omitting review, tooling, retrieval, infrastructure, or change costs.
• Choosing a denominator that makes the economics look favorable, such as all requests instead of completed or resolved workflows.
• Treating a missing baseline as zero or assuming adoption means benefit.
• Using a normative framework as proof that an intervention is safe or effective.
• Granting broader authority because a pilot had a positive mean result.
• Reusing a prior decision after the workflow, model, population, cost boundary, or evidence source changed.
• Writing a sophisticated recommendation without preserving the source-level evidence that supports it.

Verification Checklist

Before delivering an AI operating economics decision, verify:

• The workflow, intervention, population, baseline, decision owner, and human-control boundary are explicit.
• The value hypothesis is falsifiable and tied to an observable outcome.
• At least one countermetric is defined for each benefit claim.
• Fixed, variable, step-function, and fully loaded costs are separated where material.
• The denominator represents meaningful work or value, not merely requests or tokens.
• The comparison design and its limitations are stated.
• Relevant worker, user, task, quality, and risk slices are inspected or explicitly unavailable.
• Claims are labeled by evidence class and traced to sources.
• Missing evidence is visible with an owner and next step.
• The disposition, authority boundary, reversal path, and review trigger are recorded.
• Detailed statistical, financial, instrumentation, governance, runtime, and launch checks were routed to their owning skills.

Exit Criteria

Stop when the requested decision is supported by a durable evidence record, or when a bounded hold/escalation is the honest result. Do not continue refining prose to conceal missing evidence.