Deliver a governed support copilot.
A four-week client simulation that combines business discovery, data engineering, AI quality, controlled actions, and operations.
The client brief
Harbor Service Operations is a fictional internal support team. Staff need to find the correct policy, explain it with evidence, and propose a ticket action. Departments have different access permissions, and a supervisor must approve changes.
Build a copilot that retrieves authorized policy revisions, provides cited assistance, and proposes a validated ticket update. Keep execution behind trusted approval and permissions checks. Use synthetic tickets, customers, and documents throughout.
Four delivery milestones
| Week | Work | Evidence |
|---|---|---|
| 21 · Discover | Interview, map the workflow, audit data, define baseline and acceptance criteria. | Problem brief, stakeholder map, access matrix, architecture, prioritized backlog. |
| 22 · Implement | Build the pipeline, authorized retrieval, and initial application. | Reproducible data, source-linked answers, retrieval tests, stakeholder demo. |
| 23 · Harden | Add tool integration, approval, evaluation, and a release path. | Contract/security tests, versioned evaluation, CI configuration, release manifest. |
| 24 · Operate & defend | Test failures, rehearse recovery, analyze cost, and hand off. | Runbook, incident report, cost analysis, executive and technical demonstrations. |
Review rubric
| Dimension | Points | What a reviewer looks for |
|---|---|---|
| Discovery and value | 15 | Clear workflow owner, baseline, scope, and acceptance criteria. |
| Data and governance | 15 | Reliable ingestion, quality, access, freshness, and deletion. |
| AI quality and evaluation | 20 | Held-out cases, evidence support, failure analysis, reproducible results. |
| Engineering and integration | 15 | Maintainable code, contracts, tests, and safe retries. |
| Security and approval | 15 | Backend-enforced access, action binding, negative tests. |
| Deployment and operations | 10 | Reproducible release, observability, rollback, recovery. |
| Communication and handoff | 10 | Useful documentation and audience-appropriate demos. |
Proposed pass: 80/100, with critical access and approval tests passing and reproducibility demonstrated. Rework is allowed with a documented improvement report. This is an educational rubric, not an employer hiring standard.
Agree on the acceptance contract
Use at least 100 held-out task cases for the full capstone, plus a separate adversarial/access suite. Include answerable, ambiguous, unsupported, and failure cases. Document provenance and avoid tuning on the held-out cases.
An example educational quality target is at least 85% human-reviewed correctness on the defined answerable subset. Agree workload-specific p95 latency and cost targets before tuning. Require zero unauthorized retrievals or writes in the supplied test suite; this states a finite test result, not a universal security guarantee.
The evidence bundle
- Discovery brief, stakeholder map, and baseline method.
- Synthetic dataset, generator or fixtures, schema, and provenance.
- Source repository, tested setup, dependencies, and meaningful tests.
- Architecture, identity/access matrix, and decision records.
- Retrieval/answer evaluation with versioned cases and safe traces.
- Validated tools, approval records, audit trail, and idempotency evidence.
- Release manifest, deployment verification, and rollback rehearsal.
- Cost analysis, service indicators, runbook, and incident exercise.
- Five-minute executive demo and fifteen-minute engineering defense.
- Claim-to-evidence index with limitations and unverified items.