Prompt Engineer Toolkit

Systematic toolkit for designing, testing, and optimizing LLM prompts with the rigor of software engineering — including prompt patterns, evaluation frameworks, and iteration workflows.

When to Use

Use this toolkit when:

• Designing system prompts for production LLM applications
• Optimizing prompt performance on specific tasks
• Building reusable prompt templates across projects
• Need systematic evaluation of prompt quality

Use simpler approaches when:

• One-off queries that don't need optimization
• Simple completions where the default behavior is sufficient
• Tasks where fine-tuning would be more effective than prompt engineering

Quick Start

Structured Prompt Template

Copy
# Role
You are a [specific role] with expertise in [domain].

# Context
[Background information the model needs to understand the task]

# Task
[Clear, specific instruction about what to produce]

# Constraints
- [Constraint 1: format, length, style]
- [Constraint 2: what to include/exclude]
- [Constraint 3: tone and audience]

# Examples
## Input:
[Example input]

## Output:
[Example output demonstrating desired format and quality]

# Output Format
[Explicit description of expected output structure]

Prompt Evaluation Framework

Copy
import json
from dataclasses import dataclass

@dataclass
class PromptTest:
    input: str
    expected_contains: list[str]
    expected_format: str  # "json", "markdown", "plain"
    max_length: int = 1000

def evaluate_prompt(prompt_template, test_cases, llm_fn):
    results = []
    for test in test_cases:
        response = llm_fn(prompt_template.format(input=test.input))

        score = 0
        # Check content requirements
        for expected in test.expected_contains:
            if expected.lower() in response.lower():
                score += 1

        # Check format compliance
        if test.expected_format == "json":
            try:
                json.loads(response)
                score += 2
            except json.JSONDecodeError:
                pass

        # Check length constraint
        if len(response) <= test.max_length:
            score += 1

        results.append({
            "input": test.input,
            "score": score,
            "max_score": len(test.expected_contains) + 3,
            "response_length": len(response)
        })

    avg_score = sum(r["score"] for r in results) / sum(r["max_score"] for r in results)
    return {"results": results, "average_score": avg_score}

Core Concepts

Prompt Design Patterns

Pattern	Purpose	Example
Role Assignment	Set expertise context	"You are a senior security auditor"
Few-Shot	Teach by example	2-5 input/output pairs
Chain of Thought	Improve reasoning	"Think step by step before answering"
Output Structuring	Control format	"Respond in JSON with fields: ..."
Constraint Setting	Limit behavior	"Do not include opinions or speculation"
Self-Consistency	Improve reliability	Generate multiple responses, take majority

Iteration Workflow

1. Draft → Write initial prompt
2. Test → Run against 10+ diverse test cases
3. Analyze → Identify failure patterns
4. Refine → Add constraints, examples, or clarifications
5. Evaluate → Compare v1 vs v2 on same test suite
6. Deploy → Use the version with higher eval score

Prompt Optimization Techniques

Reduce ambiguity:

Bad:  "Summarize this text"
Good: "Write a 3-sentence summary of this text focusing on the key technical decisions. Use present tense."

Add format constraints:

Bad:  "List the pros and cons"
Good: "List exactly 3 pros and 3 cons as bullet points. Each point should be one sentence."

Use delimiters for inputs:

Good: "Analyze the code between <code> and </code> tags:\n<code>{user_code}</code>"

Configuration

Parameter	Description
role	Expertise persona for the model
context	Background information and domain knowledge
task	Clear instruction for what to produce
constraints	Behavioral limitations and requirements
examples	Few-shot input/output demonstrations
output_format	Expected response structure
temperature	Creativity vs determinism (0.0-1.0)

Best Practices

1. Be specific, not vague — "Write a 200-word technical summary" beats "Summarize this"
2. Show, don't just tell — few-shot examples are more effective than long instructions
3. Test with adversarial inputs — edge cases, ambiguous queries, and out-of-scope requests
4. Version your prompts — treat prompts as code with git tracking and changelogs
5. Evaluate quantitatively — use scoring functions, not gut feel, to compare prompt versions
6. Separate concerns — system prompt (role/context) vs user prompt (task/input)

Common Issues

Model ignores formatting instructions: Move format instructions to the end of the prompt (recency bias). Add an explicit example showing the exact output format. Use XML tags or delimiters to structure the expected output.

Inconsistent outputs across runs: Set temperature to 0 for deterministic results. Add more few-shot examples to anchor behavior. Use output validation to retry on format violations.

Prompt too long, hitting token limits: Move static context to system message with prefix caching. Compress few-shot examples — keep the most representative 2-3. Split complex tasks into sequential prompts.