format: apply repo prettier (3.8.1) to the folded skills tree
963 markdown files reformatted with the repository's pinned prettier so pnpm format:check covers the folded tree like every other repo file. The formatter's embedded-language pass also normalized code fences (TS semicolons, closed HTML tags in examples, lowercased CSS hex colors, one renumbered list that skipped an index). Alphanumeric token deltas vs the fold commit were audited file-by-file; all are formatter-equivalent markup normalizations plus the four sanitized skills.
This commit is contained in:
@@ -24,20 +24,24 @@ Before designing a test, understand:
|
||||
## Core Principles
|
||||
|
||||
### 1. Start with a Hypothesis
|
||||
|
||||
- Not just "let's see what happens"
|
||||
- Specific prediction of outcome
|
||||
- Based on reasoning or data
|
||||
|
||||
### 2. Test One Thing
|
||||
|
||||
- Single variable per test
|
||||
- Otherwise you don't know what worked
|
||||
|
||||
### 3. Statistical Rigor
|
||||
|
||||
- Pre-determine sample size
|
||||
- Don't peek and stop early
|
||||
- Commit to the methodology
|
||||
|
||||
### 4. Measure What Matters
|
||||
|
||||
- Primary metric tied to business value
|
||||
- Secondary metrics for context
|
||||
- Guardrail metrics to prevent harm
|
||||
@@ -66,12 +70,12 @@ We'll know this is true when [metrics].
|
||||
|
||||
## Test Types
|
||||
|
||||
| Type | Description | Traffic Needed |
|
||||
|------|-------------|----------------|
|
||||
| A/B | Two versions, single change | Moderate |
|
||||
| A/B/n | Multiple variants | Higher |
|
||||
| MVT | Multiple changes in combinations | Very high |
|
||||
| Split URL | Different URLs for variants | Moderate |
|
||||
| Type | Description | Traffic Needed |
|
||||
| --------- | -------------------------------- | -------------- |
|
||||
| A/B | Two versions, single change | Moderate |
|
||||
| A/B/n | Multiple variants | Higher |
|
||||
| MVT | Multiple changes in combinations | Very high |
|
||||
| Split URL | Different URLs for variants | Moderate |
|
||||
|
||||
---
|
||||
|
||||
@@ -79,14 +83,15 @@ We'll know this is true when [metrics].
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| Baseline | 10% Lift | 20% Lift | 50% Lift |
|
||||
|----------|----------|----------|----------|
|
||||
| 1% | 150k/variant | 39k/variant | 6k/variant |
|
||||
| 3% | 47k/variant | 12k/variant | 2k/variant |
|
||||
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
|
||||
| 10% | 12k/variant | 3k/variant | 550/variant |
|
||||
| Baseline | 10% Lift | 20% Lift | 50% Lift |
|
||||
| -------- | ------------ | ----------- | ------------ |
|
||||
| 1% | 150k/variant | 39k/variant | 6k/variant |
|
||||
| 3% | 47k/variant | 12k/variant | 2k/variant |
|
||||
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
|
||||
| 10% | 12k/variant | 3k/variant | 550/variant |
|
||||
|
||||
**Calculators:**
|
||||
|
||||
- [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html)
|
||||
- [Optimizely's](https://www.optimizely.com/sample-size-calculator/)
|
||||
|
||||
@@ -97,19 +102,23 @@ We'll know this is true when [metrics].
|
||||
## Metrics Selection
|
||||
|
||||
### Primary Metric
|
||||
|
||||
- Single metric that matters most
|
||||
- Directly tied to hypothesis
|
||||
- What you'll use to call the test
|
||||
|
||||
### Secondary Metrics
|
||||
|
||||
- Support primary metric interpretation
|
||||
- Explain why/how the change worked
|
||||
|
||||
### Guardrail Metrics
|
||||
|
||||
- Things that shouldn't get worse
|
||||
- Stop test if significantly negative
|
||||
|
||||
### Example: Pricing Page Test
|
||||
|
||||
- **Primary**: Plan selection rate
|
||||
- **Secondary**: Time on page, plan distribution
|
||||
- **Guardrail**: Support tickets, refund rate
|
||||
@@ -120,14 +129,15 @@ We'll know this is true when [metrics].
|
||||
|
||||
### What to Vary
|
||||
|
||||
| Category | Examples |
|
||||
|----------|----------|
|
||||
| Headlines/Copy | Message angle, value prop, specificity, tone |
|
||||
| Visual Design | Layout, color, images, hierarchy |
|
||||
| CTA | Button copy, size, placement, number |
|
||||
| Content | Information included, order, amount, social proof |
|
||||
| Category | Examples |
|
||||
| -------------- | ------------------------------------------------- |
|
||||
| Headlines/Copy | Message angle, value prop, specificity, tone |
|
||||
| Visual Design | Layout, color, images, hierarchy |
|
||||
| CTA | Button copy, size, placement, number |
|
||||
| Content | Information included, order, amount, social proof |
|
||||
|
||||
### Best Practices
|
||||
|
||||
- Single, meaningful change
|
||||
- Bold enough to make a difference
|
||||
- True to the hypothesis
|
||||
@@ -136,13 +146,14 @@ We'll know this is true when [metrics].
|
||||
|
||||
## Traffic Allocation
|
||||
|
||||
| Approach | Split | When to Use |
|
||||
|----------|-------|-------------|
|
||||
| Standard | 50/50 | Default for A/B |
|
||||
| Conservative | 90/10, 80/20 | Limit risk of bad variant |
|
||||
| Ramping | Start small, increase | Technical risk mitigation |
|
||||
| Approach | Split | When to Use |
|
||||
| ------------ | --------------------- | ------------------------- |
|
||||
| Standard | 50/50 | Default for A/B |
|
||||
| Conservative | 90/10, 80/20 | Limit risk of bad variant |
|
||||
| Ramping | Start small, increase | Technical risk mitigation |
|
||||
|
||||
**Considerations:**
|
||||
|
||||
- Consistency: Users see same variant on return
|
||||
- Balanced exposure across time of day/week
|
||||
|
||||
@@ -151,11 +162,13 @@ We'll know this is true when [metrics].
|
||||
## Implementation
|
||||
|
||||
### Client-Side
|
||||
|
||||
- JavaScript modifies page after load
|
||||
- Quick to implement, can cause flicker
|
||||
- Tools: PostHog, Optimizely, VWO
|
||||
|
||||
### Server-Side
|
||||
|
||||
- Variant determined before render
|
||||
- No flicker, requires dev work
|
||||
- Tools: PostHog, LaunchDarkly, Split
|
||||
@@ -165,6 +178,7 @@ We'll know this is true when [metrics].
|
||||
## Running the Test
|
||||
|
||||
### Pre-Launch Checklist
|
||||
|
||||
- [ ] Hypothesis documented
|
||||
- [ ] Primary metric defined
|
||||
- [ ] Sample size calculated
|
||||
@@ -175,16 +189,19 @@ We'll know this is true when [metrics].
|
||||
### During the Test
|
||||
|
||||
**DO:**
|
||||
|
||||
- Monitor for technical issues
|
||||
- Check segment quality
|
||||
- Document external factors
|
||||
|
||||
**DON'T:**
|
||||
|
||||
- Peek at results and stop early
|
||||
- Make changes to variants
|
||||
- Add traffic from new sources
|
||||
|
||||
### The Peeking Problem
|
||||
|
||||
Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.
|
||||
|
||||
---
|
||||
@@ -192,6 +209,7 @@ Looking at results before reaching sample size and stopping early leads to false
|
||||
## Analyzing Results
|
||||
|
||||
### Statistical Significance
|
||||
|
||||
- 95% confidence = p-value < 0.05
|
||||
- Means <5% chance result is random
|
||||
- Not a guarantee—just a threshold
|
||||
@@ -207,18 +225,19 @@ Looking at results before reaching sample size and stopping early leads to false
|
||||
|
||||
### Interpreting Results
|
||||
|
||||
| Result | Conclusion |
|
||||
|--------|------------|
|
||||
| Significant winner | Implement variant |
|
||||
| Significant loser | Keep control, learn why |
|
||||
| Result | Conclusion |
|
||||
| ------------------------- | -------------------------------- |
|
||||
| Significant winner | Implement variant |
|
||||
| Significant loser | Keep control, learn why |
|
||||
| No significant difference | Need more traffic or bolder test |
|
||||
| Mixed signals | Dig deeper, maybe segment |
|
||||
| Mixed signals | Dig deeper, maybe segment |
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
Document every test with:
|
||||
|
||||
- Hypothesis
|
||||
- Variants (with screenshots)
|
||||
- Results (sample, metrics, significance)
|
||||
@@ -231,16 +250,19 @@ Document every test with:
|
||||
## Common Mistakes
|
||||
|
||||
### Test Design
|
||||
|
||||
- Testing too small a change (undetectable)
|
||||
- Testing too many things (can't isolate)
|
||||
- No clear hypothesis
|
||||
|
||||
### Execution
|
||||
|
||||
- Stopping early
|
||||
- Changing things mid-test
|
||||
- Not checking implementation
|
||||
|
||||
### Analysis
|
||||
|
||||
- Ignoring confidence intervals
|
||||
- Cherry-picking segments
|
||||
- Over-interpreting inconclusive results
|
||||
|
||||
Reference in New Issue
Block a user