stack/docs/orchestrator-learnings.json

{
  "project": "mosaic-stack",
  "milestone": "M6-AgentOrchestration",
  "created_at": "2026-02-05T20:00:00Z",
  "learnings": [
    {
      "task_id": "MS-SEC-001",
      "task_type": "AUTH_ADD",
      "estimate_k": 15,
      "actual_k": 0.3,
      "variance_pct": -98,
      "characteristics": {
        "file_count": 1,
        "keywords": ["authentication", "orchestrator API", "ApiKeyGuard"]
      },
      "analysis": "CRITICAL VARIANCE - Investigate. Possible causes: (1) Auth already existed, (2) Task was trivial decorator addition, (3) Reporting error. Need to verify task completion quality.",
      "flags": ["CRITICAL", "NEEDS_INVESTIGATION"],
      "captured_at": "2026-02-05T15:30:00Z"
    },
    {
      "task_id": "MS-SEC-003",
      "task_type": "ERROR_HANDLING",
      "estimate_k": 8,
      "actual_k": 18.5,
      "variance_pct": 131,
      "characteristics": {
        "file_count": 4,
        "keywords": ["secret scanner", "error state", "scan result type", "Zod schema"]
      },
      "analysis": "CRITICAL VARIANCE - Task required adding new fields to existing type, updating all callers, modifying error messages, comprehensive error path tests. Type interface changes cascade through codebase.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T16:42:00Z"
    },
    {
      "task_id": "MS-SEC-006",
      "task_type": "CONFIG_DEFAULT_CHANGE",
      "estimate_k": 10,
      "actual_k": 18,
      "variance_pct": 80,
      "characteristics": {
        "file_count": 3,
        "keywords": ["Docker sandbox", "default enabled", "security warning", "config test"]
      },
      "analysis": "Underestimated test coverage needed. New config test file (8 tests) + security warning tests (2 tests) required more tokens than simple default flip.",
      "flags": [],
      "captured_at": "2026-02-05T16:05:00Z"
    },
    {
      "task_id": "MS-SEC-010",
      "task_type": "INPUT_VALIDATION",
      "estimate_k": 5,
      "actual_k": 8.5,
      "variance_pct": 70,
      "characteristics": {
        "file_count": 2,
        "keywords": ["OAuth callback", "error sanitization", "allowlist", "encodeURIComponent"]
      },
      "analysis": "Underestimated allowlist complexity. Required 18 OAuth 2.0/OIDC error codes, URL encoding for all params, and 5 comprehensive security tests.",
      "flags": [],
      "captured_at": "2026-02-05T16:36:00Z"
    },
    {
      "task_id": "MS-SEC-011",
      "task_type": "CONFIG_EXTERNALIZATION",
      "estimate_k": 8,
      "actual_k": 15,
      "variance_pct": 87.5,
      "characteristics": {
        "file_count": 2,
        "keywords": ["OIDC", "federation", "env vars", "trailing slash normalization"]
      },
      "analysis": "Underestimated integration complexity. Required reusing auth.config OIDC vars, handling trailing slash differences between auth config and JWT validation, adding fail-fast logic, and 5 new tests.",
      "flags": [],
      "captured_at": "2026-02-05T16:45:00Z"
    },
    {
      "task_id": "MS-SEC-012",
      "task_type": "BUG_FIX_SIMPLE",
      "estimate_k": 3,
      "actual_k": 12.5,
      "variance_pct": 317,
      "characteristics": {
        "file_count": 2,
        "keywords": ["boolean logic", "nullish coalescing", "ReactFlow", "handleDeleteSelected"]
      },
      "analysis": "CRITICAL VARIANCE - Estimate was for simple operator change (?? to ||), but task expanded to add 13 comprehensive tests covering all boolean logic scenarios. 'Simple fix' tasks with untested code should include test addition in estimate.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T16:55:00Z"
    },
    {
      "task_id": "MS-HIGH-001",
      "task_type": "NULLABLE_REFACTOR",
      "estimate_k": 8,
      "actual_k": 12.5,
      "variance_pct": 56,
      "characteristics": {
        "file_count": 2,
        "keywords": ["OpenAI", "nullable client", "embedding service", "graceful degradation"]
      },
      "analysis": "Making a service client nullable requires updating all call sites with null checks and adding tests for the unconfigured path. Estimate should include caller updates.",
      "flags": [],
      "captured_at": "2026-02-05T17:27:00Z"
    },
    {
      "task_id": "MS-HIGH-004",
      "task_type": "OBSERVABILITY_ADD",
      "estimate_k": 10,
      "actual_k": 22,
      "variance_pct": 120,
      "characteristics": {
        "file_count": 2,
        "keywords": ["rate limiter", "fallback", "health check", "degraded mode"]
      },
      "analysis": "CRITICAL VARIANCE - Adding observability to a service requires: (1) tracking state variables, (2) new methods for status exposure, (3) integration with health check system, (4) comprehensive test coverage for all states. Estimate 2x for 'add health check' tasks.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T18:02:00Z"
    },
    {
      "task_id": "MS-HIGH-006",
      "task_type": "RATE_LIMITING_ADD",
      "estimate_k": 8,
      "actual_k": 25,
      "variance_pct": 213,
      "characteristics": {
        "file_count": 3,
        "keywords": ["rate limiting", "catch-all route", "IP extraction", "X-Forwarded-For"]
      },
      "analysis": "CRITICAL VARIANCE - Adding rate limiting requires: (1) understanding existing throttle infrastructure, (2) IP extraction helpers for proxy setups, (3) new test file for rate limit behavior, (4) Retry-After header testing. Estimate 3x for rate limiting tasks.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T18:22:00Z"
    },
    {
      "task_id": "MS-HIGH-007",
      "task_type": "CONFIG_VALIDATION",
      "estimate_k": 5,
      "actual_k": 18,
      "variance_pct": 260,
      "characteristics": {
        "file_count": 4,
        "keywords": ["UUID validation", "federation", "startup validation", "config file"]
      },
      "analysis": "CRITICAL VARIANCE - 'Simple validation' tasks expand to: (1) new config module/file, (2) validation function with edge cases, (3) module init hook integration, (4) updating callers to use new config getter, (5) 18 comprehensive tests. Estimate 3-4x for config validation tasks.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T18:35:00Z"
    },
    {
      "task_id": "MS-HIGH-008",
      "task_type": "SECURITY_REFACTOR",
      "estimate_k": 12,
      "actual_k": 25,
      "variance_pct": 108,
      "characteristics": {
        "file_count": 5,
        "keywords": ["CSRF", "fetch replacement", "API client", "FormData upload"]
      },
      "analysis": "CRITICAL VARIANCE - Routing fetch() through API client required: (1) adding new apiPostFormData() method for FormData, (2) finding additional calls not in original finding, (3) updating test mocks to handle CSRF fetches, (4) handling different Content-Type scenarios. Multi-file refactors expand beyond listed files.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T18:50:00Z"
    },
    {
      "task_id": "MS-HIGH-009",
      "task_type": "FEATURE_GATING",
      "estimate_k": 10,
      "actual_k": 30,
      "variance_pct": 200,
      "characteristics": {
        "file_count": 6,
        "keywords": ["NODE_ENV", "mock data", "Coming Soon component", "environment check"]
      },
      "analysis": "CRITICAL VARIANCE - Feature gating requires: (1) creating reusable placeholder component, (2) tests for the component, (3) updating multiple pages, (4) environment-specific logic in each page. Creating reusable UI components adds significant overhead.",
      "flags": ["CRITICAL"],
      "captured_at": "2026-02-05T19:05:00Z"
    }
  ],
  "phase_summaries": [],
  "proposed_adjustments": [
    {
      "category": "AUTH_ADD",
      "current_heuristic": "15-25K",
      "proposed_heuristic": "NO CHANGE NEEDED",
      "confidence": "HIGH",
      "evidence": ["MS-SEC-001"],
      "notes": "Investigation complete: -98% variance was REPORTING ANOMALY, not estimation error. Actual implementation was 276 lines (guard + tests + docs). Token usage reporting may have bug. Heuristic is accurate."
    }
  ],
  "investigation_queue": [
    {
      "task_id": "MS-SEC-001",
      "question": "Did this task actually add authentication, or was auth already present?",
      "priority": "HIGH",
      "status": "CLOSED",
      "resolution": "LEGITIMATE COMPLETION - Implementation verified: OrchestratorApiKeyGuard with 82 lines of guard code, 169 lines of tests, 6 files changed, 276 total lines. The 0.3K token usage was a REPORTING ANOMALY, not incomplete work.",
      "verified_at": "2026-02-05T20:30:00Z"
    }
  ]
}