115 lines
4.4 KiB
Markdown
115 lines
4.4 KiB
Markdown
# QA & Testing Guide
|
|
|
|
## Before Starting
|
|
|
|
1. Check assigned issue: `~/.config/mosaic/rails/git/issue-list.sh -a @me`
|
|
2. Create scratchpad: `docs/scratchpads/{issue-number}-{short-name}.md`
|
|
3. Review `docs/PRD.md` or `docs/PRD.json` as the requirements source.
|
|
4. Review acceptance criteria and affected change surfaces.
|
|
|
|
## Testing Policy (Hard Rules)
|
|
|
|
1. Situational testing is the PRIMARY validation gate.
|
|
2. Baseline testing is REQUIRED for all software changes.
|
|
3. TDD is risk-based and REQUIRED only for defined high-risk change types.
|
|
4. Tests MUST validate requirements and behavior, not only internal implementation details.
|
|
|
|
## Priority Order
|
|
|
|
1. Situational tests: prove requirements and real behavior on changed surfaces.
|
|
2. Baseline tests: lint/type/unit/integration safety checks.
|
|
3. TDD discipline: applied where risk justifies test-first workflow.
|
|
|
|
## Risk-Based TDD Requirement
|
|
|
|
| Change Type | TDD Requirement | Required Action |
|
|
|---|---|---|
|
|
| Bug fix | REQUIRED | Write a failing reproducer test first, then fix. |
|
|
| Security/auth/permission logic | REQUIRED | Write failing security/permission-path test first. |
|
|
| Critical business logic or data mutation rules | REQUIRED | Write failing rule/invariant test first. |
|
|
| API behavior regression | REQUIRED | Write failing contract/behavior test first. |
|
|
| Low-risk UI copy/style/layout | OPTIONAL | Add verification tests as appropriate; TDD recommended, not mandatory. |
|
|
| Mechanical refactor with unchanged behavior | OPTIONAL | Ensure regression/smoke coverage and situational evidence. |
|
|
|
|
If TDD is not required and skipped, record rationale in scratchpad.
|
|
If TDD is required and skipped, task is NOT complete.
|
|
|
|
## Baseline Test Requirements
|
|
|
|
For all software changes, run baseline checks applicable to the repo:
|
|
|
|
1. lint/static checks
|
|
2. type checks
|
|
3. unit tests for changed logic
|
|
4. integration tests for changed boundaries
|
|
|
|
## Situational Testing Matrix (Primary Gate)
|
|
|
|
| Change Surface | Required Situational Tests |
|
|
|---|---|
|
|
| Authentication/authorization | auth failure-path tests, permission boundary tests, token/session validation |
|
|
| Database schema/migrations | migration up/down validation, rollback safety, data integrity checks |
|
|
| API contract changes | backward compatibility checks, consumer-impact tests, contract tests |
|
|
| Frontend/UI workflow changes | end-to-end flow tests, accessibility sanity checks, state transition checks |
|
|
| CI/CD or deployment changes | pipeline execution validation, artifact integrity checks, rollback path check |
|
|
| Security-sensitive logic | abuse-case tests, input validation fuzzing/sanitization checks |
|
|
| Performance-critical path | baseline comparison, regression threshold checks |
|
|
|
|
## Coverage Requirements
|
|
|
|
### Minimum Standards
|
|
|
|
- Overall Coverage: 85% minimum
|
|
- Critical Paths: 95% minimum (auth, payments, data mutations)
|
|
- New Code: 90% minimum
|
|
|
|
Coverage is necessary but NOT sufficient. Passing coverage does not replace situational verification.
|
|
|
|
## Requirements-to-Evidence Mapping (Mandatory)
|
|
|
|
Before completion, map each acceptance criterion to concrete evidence.
|
|
Acceptance criteria MUST come from the active PRD.
|
|
|
|
Template:
|
|
|
|
```markdown
|
|
| Acceptance Criterion | Verification Method | Evidence |
|
|
|---|---|---|
|
|
| AC-1: ... | Situational test / baseline test / manual verification | command + result |
|
|
| AC-2: ... | ... | ... |
|
|
```
|
|
|
|
## Test Quality Rules
|
|
|
|
1. Test behavior and outcomes, not private implementation details.
|
|
2. Include failure-path and edge-case assertions for changed behavior.
|
|
3. Keep tests deterministic; no new flaky tests.
|
|
4. Keep tests isolated; no dependency on execution order.
|
|
|
|
## Anti-Gaming Rules
|
|
|
|
1. Do NOT stop at "tests pass" if acceptance criteria are not verified.
|
|
2. Do NOT write narrow tests that only satisfy assertions while missing real workflow behavior.
|
|
3. Do NOT claim completion without situational evidence for impacted surfaces.
|
|
|
|
## Reporting
|
|
|
|
QA report MUST include:
|
|
|
|
1. baseline tests run and outcomes
|
|
2. situational tests run and outcomes
|
|
3. TDD usage decision (required/applied or optional/skipped with rationale)
|
|
4. acceptance-criteria-to-evidence mapping
|
|
5. coverage results
|
|
6. residual risk notes
|
|
|
|
## Before Completing
|
|
|
|
1. Baseline tests pass.
|
|
2. Required situational tests pass.
|
|
3. TDD obligations met for required change types.
|
|
4. Acceptance criteria mapped to evidence.
|
|
5. No flaky tests introduced.
|
|
6. CI pipeline passes (if available).
|
|
7. Scratchpad updated with results.
|