2.8 KiB
Quality Control & Operations Protocol
This document defines the automated decision-making logic for content validation and the system-wide response to operational failures.
1. Validation Scoring Formula (Blocker #6)
Agent E (Validation) and Agent F (Numerical) calculate a combined Confidence Score (0-100%). A score of 95% or higher is required for auto-publication to the main branch.
1.1 Weighted Categories
| Category | Weight | Must Pass? | Criteria |
|---|---|---|---|
| Numerical (ESI) | 40% | YES | Stats (HP, Slots, PG/CPU) must match ESI ±0%. |
| Structural | 20% | YES | Valid YAML, required fields present, correct URL path. |
| Relational | 20% | NO | Internal links resolve, TypeIDs are valid. |
| Semantic | 20% | NO | Prose description matches structured data intent. |
1.2 The "Must Pass" Rule
If any "Must Pass" category fails (score < 100% in that category), the total Confidence Score is immediately capped at 0%, regardless of other categories. This prevents LLM hallucinations from overriding official game data.
1.3 Scoring Logic
Total Score = (ESI * 0.4) + (Struct * 0.2) + (Relat * 0.2) + (Semant * 0.2)
2. Failure Handling Matrix (Blocker #8)
The system distinguishes between transient infrastructure issues and content quality failures.
2.1 Tiered Response Matrix
| Error Type | Tier | Initial Action | Max Retries | Escalation |
|---|---|---|---|---|
| API Timeout / 5xx | 1 | Exponential Backoff (1m, 5m, 15m) | 3 | Tier 3 Alert |
| Validation Fail (70-94%) | 2 | Regeneration with Error Feedback | 2 | Tier 3 Alert |
| Validation Fail (<70%) | 2 | Immediate Rejection | 1 | Tier 3 Alert |
| Auth Failure / 401 | 3 | Halt Pipeline | 0 | Critical System Alert |
| Git Conflict | 3 | Halt Pipeline | 0 | Critical System Alert |
2.2 Feedback Loop (Tier 2)
When a page fails validation with a score of 70-94%, Agent E sends a "Correction Request" back to the source agent (A, B, or C) containing:
- The specific field that failed.
- The expected value (from ESI or Schema).
- The current value produced.
2.3 Critical System Alerting
Tier 3 failures trigger a webhook notification to the system administrator. The LangGraph state is persisted as a "Suspended" checkpoint, allowing for manual inspection and resume via LangSmith.
3. Rate Limiting Policy (Blocker #11)
To ensure stability and prevent IP bans, the following global limits are enforced at the transport layer:
| Target | Rate Limit | Burst |
|---|---|---|
| ESI API | 20 req / sec | 50 |
| MediaWiki API | 2 req / sec | 5 |
| Wiki.js API | 5 req / sec | 10 |
| LLM APIs | Per Provider Tier | N/A |