61 lines
2.8 KiB
Markdown
61 lines
2.8 KiB
Markdown
# Quality Control & Operations Protocol
|
|
|
|
This document defines the automated decision-making logic for content validation and the system-wide response to operational failures.
|
|
|
|
## 1. Validation Scoring Formula (Blocker #6)
|
|
|
|
Agent E (Validation) and Agent F (Numerical) calculate a combined **Confidence Score (0-100%)**. A score of **95% or higher** is required for auto-publication to the `main` branch.
|
|
|
|
### 1.1 Weighted Categories
|
|
|
|
| Category | Weight | Must Pass? | Criteria |
|
|
|----------|--------|------------|----------|
|
|
| **Numerical (ESI)** | 40% | **YES** | Stats (HP, Slots, PG/CPU) must match ESI ±0%. |
|
|
| **Structural** | 20% | **YES** | Valid YAML, required fields present, correct URL path. |
|
|
| **Relational** | 20% | NO | Internal links resolve, TypeIDs are valid. |
|
|
| **Semantic** | 20% | NO | Prose description matches structured data intent. |
|
|
|
|
### 1.2 The "Must Pass" Rule
|
|
If any **"Must Pass"** category fails (score < 100% in that category), the total Confidence Score is immediately capped at **0%**, regardless of other categories. This prevents LLM hallucinations from overriding official game data.
|
|
|
|
### 1.3 Scoring Logic
|
|
`Total Score = (ESI * 0.4) + (Struct * 0.2) + (Relat * 0.2) + (Semant * 0.2)`
|
|
|
|
---
|
|
|
|
## 2. Failure Handling Matrix (Blocker #8)
|
|
|
|
The system distinguishes between transient infrastructure issues and content quality failures.
|
|
|
|
### 2.1 Tiered Response Matrix
|
|
|
|
| Error Type | Tier | Initial Action | Max Retries | Escalation |
|
|
|------------|------|----------------|-------------|------------|
|
|
| **API Timeout / 5xx** | 1 | Exponential Backoff (1m, 5m, 15m) | 3 | Tier 3 Alert |
|
|
| **Validation Fail (70-94%)** | 2 | Regeneration with Error Feedback | 2 | Tier 3 Alert |
|
|
| **Validation Fail (<70%)** | 2 | Immediate Rejection | 1 | Tier 3 Alert |
|
|
| **Auth Failure / 401** | 3 | Halt Pipeline | 0 | **Critical System Alert** |
|
|
| **Git Conflict** | 3 | Halt Pipeline | 0 | **Critical System Alert** |
|
|
|
|
### 2.2 Feedback Loop (Tier 2)
|
|
When a page fails validation with a score of 70-94%, Agent E sends a "Correction Request" back to the source agent (A, B, or C) containing:
|
|
1. The specific field that failed.
|
|
2. The expected value (from ESI or Schema).
|
|
3. The current value produced.
|
|
|
|
### 2.3 Critical System Alerting
|
|
Tier 3 failures trigger a webhook notification to the system administrator. The LangGraph state is persisted as a "Suspended" checkpoint, allowing for manual inspection and resume via LangSmith.
|
|
|
|
---
|
|
|
|
## 3. Rate Limiting Policy (Blocker #11)
|
|
|
|
To ensure stability and prevent IP bans, the following global limits are enforced at the transport layer:
|
|
|
|
| Target | Rate Limit | Burst |
|
|
|--------|------------|-------|
|
|
| **ESI API** | 20 req / sec | 50 |
|
|
| **MediaWiki API** | 2 req / sec | 5 |
|
|
| **Wiki.js API** | 5 req / sec | 10 |
|
|
| **LLM APIs** | Per Provider Tier | N/A |
|