format: apply repo prettier (3.8.1) to the folded skills tree
963 markdown files reformatted with the repository's pinned prettier so pnpm format:check covers the folded tree like every other repo file. The formatter's embedded-language pass also normalized code fences (TS semicolons, closed HTML tags in examples, lowercased CSS hex colors, one renumbered list that skipped an index). Alphanumeric token deltas vs the fold commit were audited file-by-file; all are formatter-equivalent markup normalizations plus the four sanitized skills.
This commit is contained in:
fargo
@@ -9,6 +9,7 @@ This document provides guidance on creating comprehensive evaluations for MCP se
|
||||
## Quick Reference
|
||||
|
||||
### Evaluation Requirements
|
||||
|
||||
- Create 10 human-readable questions
|
||||
- Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
|
||||
- Each question requires multiple tool calls (potentially dozens)
|
||||
@@ -16,6 +17,7 @@ This document provides guidance on creating comprehensive evaluations for MCP se
|
||||
- Answers must be STABLE (won't change over time)
|
||||
|
||||
### Output Format
|
||||
|
||||
```xml
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
@@ -34,6 +36,7 @@ The measure of quality of an MCP server is NOT how well or comprehensively the s
|
||||
## Evaluation Overview
|
||||
|
||||
Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
|
||||
|
||||
- Realistic
|
||||
- Clear and concise
|
||||
- Unambiguous
|
||||
@@ -176,6 +179,7 @@ Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DE
|
||||
### Step 1: Documentation Inspection
|
||||
|
||||
Read the documentation of the target API to understand:
|
||||
|
||||
- Available endpoints and functionality
|
||||
- If ambiguity exists, fetch additional information from the web
|
||||
- Parallelize this step AS MUCH AS POSSIBLE
|
||||
@@ -184,6 +188,7 @@ Read the documentation of the target API to understand:
|
||||
### Step 2: Tool Inspection
|
||||
|
||||
List the tools available in the MCP server:
|
||||
|
||||
- Inspect the MCP server directly
|
||||
- Understand input/output schemas, docstrings, and descriptions
|
||||
- WITHOUT calling the tools themselves at this stage
|
||||
@@ -191,6 +196,7 @@ List the tools available in the MCP server:
|
||||
### Step 3: Developing Understanding
|
||||
|
||||
Repeat steps 1 & 2 until you have a good understanding:
|
||||
|
||||
- Iterate multiple times
|
||||
- Think about the kinds of tasks you want to create
|
||||
- Refine your understanding
|
||||
@@ -200,6 +206,7 @@ Repeat steps 1 & 2 until you have a good understanding:
|
||||
### Step 4: Read-Only Content Inspection
|
||||
|
||||
After understanding the API and tools, USE the MCP server tools:
|
||||
|
||||
- Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY
|
||||
- Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions
|
||||
- Should NOT call any tools that modify state
|
||||
@@ -214,6 +221,7 @@ After understanding the API and tools, USE the MCP server tools:
|
||||
### Step 5: Task Generation
|
||||
|
||||
After inspecting the content, create 10 human-readable questions:
|
||||
|
||||
- An LLM should be able to answer these with the MCP server
|
||||
- Follow all question and answer guidelines above
|
||||
|
||||
@@ -247,6 +255,7 @@ Each QA pair consists of a question and an answer. The output should be an XML f
|
||||
### Good Questions
|
||||
|
||||
**Example 1: Multi-hop question requiring deep exploration (GitHub MCP)**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
|
||||
@@ -255,6 +264,7 @@ Each QA pair consists of a question and an answer. The output should be an XML f
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
|
||||
- Requires multiple searches to find archived repositories
|
||||
- Needs to identify which had the most forks before archival
|
||||
- Requires examining repository details for the language
|
||||
@@ -262,6 +272,7 @@ This question is good because:
|
||||
- Based on historical (closed) data that won't change
|
||||
|
||||
**Example 2: Requires understanding context without keyword matching (Project Management MCP)**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
|
||||
@@ -270,6 +281,7 @@ This question is good because:
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
|
||||
- Doesn't use specific project name ("initiative focused on improving customer onboarding")
|
||||
- Requires finding completed projects from specific timeframe
|
||||
- Needs to identify the project lead and their role
|
||||
@@ -278,6 +290,7 @@ This question is good because:
|
||||
- Based on completed work (won't change)
|
||||
|
||||
**Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
|
||||
@@ -286,6 +299,7 @@ This question is good because:
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
|
||||
- Requires filtering bugs by date, priority, and status
|
||||
- Needs to group by assignee and calculate resolution rates
|
||||
- Requires understanding timestamps to determine 48-hour windows
|
||||
@@ -294,6 +308,7 @@ This question is good because:
|
||||
- Based on historical data from specific time period
|
||||
|
||||
**Example 4: Requires synthesis across multiple data types (CRM MCP)**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
|
||||
@@ -302,6 +317,7 @@ This question is good because:
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
|
||||
- Requires understanding subscription tier changes
|
||||
- Needs to identify upgrade events in specific timeframe
|
||||
- Requires comparing contract values
|
||||
@@ -312,6 +328,7 @@ This question is good because:
|
||||
### Poor Questions
|
||||
|
||||
**Example 1: Answer changes over time**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>How many open issues are currently assigned to the engineering team?</question>
|
||||
@@ -320,11 +337,13 @@ This question is good because:
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
|
||||
- The answer will change as issues are created, closed, or reassigned
|
||||
- Not based on stable/stationary data
|
||||
- Relies on "current state" which is dynamic
|
||||
|
||||
**Example 2: Too easy with keyword search**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
|
||||
@@ -333,11 +352,13 @@ This question is poor because:
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
|
||||
- Can be solved with a straightforward keyword search for exact title
|
||||
- Doesn't require deep exploration or understanding
|
||||
- No synthesis or analysis needed
|
||||
|
||||
**Example 3: Ambiguous answer format**
|
||||
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>List all the repositories that have Python as their primary language.</question>
|
||||
@@ -346,6 +367,7 @@ This question is poor because:
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
|
||||
- Answer is a list that could be returned in any order
|
||||
- Difficult to verify with direct string comparison
|
||||
- LLM might format differently (JSON array, comma-separated, newline-separated)
|
||||
@@ -388,6 +410,7 @@ After creating your evaluation file, you can use the provided evaluation harness
|
||||
```
|
||||
|
||||
Or install manually:
|
||||
|
||||
```bash
|
||||
pip install anthropic mcp
|
||||
```
|
||||
@@ -420,6 +443,7 @@ Evaluation files use XML format with `<qa_pair>` elements:
|
||||
The evaluation script (`scripts/evaluation.py`) supports three transport types:
|
||||
|
||||
**Important:**
|
||||
|
||||
- **stdio transport**: The evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
|
||||
- **sse/http transports**: You must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
|
||||
|
||||
@@ -436,6 +460,7 @@ python scripts/evaluation.py \
|
||||
```
|
||||
|
||||
With environment variables:
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t stdio \
|
||||
@@ -580,6 +605,7 @@ python scripts/evaluation.py \
|
||||
### Connection Errors
|
||||
|
||||
If you get connection errors:
|
||||
|
||||
- **STDIO**: Verify the command and arguments are correct
|
||||
- **SSE/HTTP**: Check the URL is accessible and headers are correct
|
||||
- Ensure any required API keys are set in environment variables or headers
|
||||
@@ -587,6 +613,7 @@ If you get connection errors:
|
||||
### Low Accuracy
|
||||
|
||||
If many evaluations fail:
|
||||
|
||||
- Review the agent's feedback for each task
|
||||
- Check if tool descriptions are clear and comprehensive
|
||||
- Verify input parameters are well-documented
|
||||
@@ -596,7 +623,8 @@ If many evaluations fail:
|
||||
### Timeout Issues
|
||||
|
||||
If tasks are timing out:
|
||||
|
||||
- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
|
||||
- Check if tools are returning too much data
|
||||
- Verify pagination is working correctly
|
||||
- Consider simplifying complex questions
|
||||
- Consider simplifying complex questions
|
||||
|
||||
Reference in New Issue
Block a user