Architect LLMs Helper

An autonomous agent that helps software architects evaluate, integrate, and manage multiple LLM providers — handling model comparison, fallback chains, provider switching, and unified API abstraction.

When to Use This Agent

Choose Architect LLMs Helper when:

• You need to evaluate multiple LLM providers (OpenAI, Anthropic, Google, open-source) for your use case
• You want a unified abstraction layer that supports provider switching without code changes
• You need failover chains so your application survives provider outages
• Cost comparison across providers is needed with real benchmarking data

Consider alternatives when:

• You are committed to a single provider and need deep integration (use provider-specific docs)
• You need model training or fine-tuning guidance (use an ML engineer agent)
• Your LLM usage is trivial (single API call, no fallback needed)

Quick Start

Copy
# .claude/agents/llms-helper.yml
name: architect-llms-helper
description: Evaluate and manage multiple LLM providers
agent_prompt: |
  You are an LLM Integration Architect. Help teams:

  1. Compare LLM providers on quality, cost, latency, and features
  2. Design unified API abstraction layers
  3. Implement fallback chains and load balancing
  4. Set up provider-agnostic prompt templates
  5. Monitor cross-provider performance and cost
  6. Plan migration between providers

  Prioritize: reliability > quality > cost > latency.

Example invocation:

Copy
claude "Compare Claude, GPT-4, and Gemini for our code review automation. Needs function calling, 100K+ context, and <$0.01 per review."

Sample comparison output:

LLM Provider Comparison — Code Review Automation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

                 Claude Sonnet  GPT-4o       Gemini 1.5 Pro
Context Window   200K tokens    128K tokens  1M tokens
Tool Use         Excellent      Excellent    Good
Code Quality     95/100         92/100       88/100
Cost/Review      $0.006         $0.008       $0.004
Latency (p95)    2.1s           2.8s         3.2s
Rate Limits      4K RPM         500 RPM      360 RPM

Recommendation: Claude Sonnet (primary) + Gemini Pro (fallback)
  - Best code quality score for primary reviews
  - Gemini as fallback for cost savings and outage resilience
  - Combined: 99.9% availability with <$0.007 avg cost

Core Concepts

Provider Abstraction Layer

Copy
// Unified LLM client with automatic fallback
interface LLMProvider {
  name: string;
  generate(params: GenerateParams): Promise<GenerateResult>;
  supportsFeature(feature: string): boolean;
  estimateCost(inputTokens: number, outputTokens: number): number;
}

class LLMRouter {
  private providers: LLMProvider[];
  private metrics: MetricsCollector;

  constructor(config: RouterConfig) {
    this.providers = config.providers.map(p => createProvider(p));
  }

  async generate(params: GenerateParams): Promise<GenerateResult> {
    const eligible = this.providers.filter(p =>
      params.requiredFeatures?.every(f => p.supportsFeature(f)) ?? true
    );

    for (const provider of eligible) {
      try {
        const start = Date.now();
        const result = await provider.generate(params);
        this.metrics.record(provider.name, Date.now() - start, 'success');
        return result;
      } catch (error) {
        this.metrics.record(provider.name, 0, 'failure');
        console.warn(`${provider.name} failed, trying next: ${error.message}`);
      }
    }

    throw new Error('All LLM providers failed');
  }
}

Cost Monitoring Dashboard

Copy
// Track per-provider costs in real-time
const costTracker = {
  providers: {
    claude: { inputCostPer1K: 0.003, outputCostPer1K: 0.015 },
    gpt4o: { inputCostPer1K: 0.005, outputCostPer1K: 0.015 },
    gemini: { inputCostPer1K: 0.00125, outputCostPer1K: 0.005 }
  },

  calculateCost(provider, inputTokens, outputTokens) {
    const config = this.providers[provider];
    return (inputTokens / 1000 * config.inputCostPer1K) +
           (outputTokens / 1000 * config.outputCostPer1K);
  },

  monthlyProjection(dailyRequests, avgInputTokens, avgOutputTokens) {
    return Object.entries(this.providers).map(([name, config]) => ({
      provider: name,
      dailyCost: this.calculateCost(name, dailyRequests * avgInputTokens, dailyRequests * avgOutputTokens),
      monthlyCost: this.calculateCost(name, dailyRequests * avgInputTokens * 30, dailyRequests * avgOutputTokens * 30)
    }));
  }
};

Configuration

Option	Type	Default	Description
primaryProvider	string	"claude"	Primary LLM provider
fallbackProviders	string[]	["gpt-4o", "gemini"]	Ordered fallback chain
maxRetries	number	2	Retries per provider before fallback
costCeiling	number	0.05	Max cost per request in USD
latencyTarget	number	5000	Max acceptable latency in ms
featureRequirements	string[]	[]	Required features: tool_use, vision, long_context

Best Practices

1. Abstract the provider, not the prompt — Different models respond differently to the same prompt. Keep a unified API layer for routing and fallback, but maintain provider-specific prompt templates that are optimized for each model's strengths and instruction-following patterns.

2. Monitor quality metrics per provider, not just uptime — Track answer quality scores (via automated evaluation or user feedback) per provider. A provider with 99.9% uptime but declining quality is worse than one with 99.5% uptime and consistent quality. Set quality alerts alongside availability alerts.

3. Negotiate enterprise rates before scaling — Per-token pricing drops significantly with committed usage. Once you exceed $1K/month with any provider, contact their sales team for volume discounts. The savings often exceed 30-50% compared to pay-as-you-go pricing.

4. Test failover chains regularly — Do not wait for a real outage to discover your fallback does not work. Run monthly failover drills by temporarily blocking the primary provider and verifying the fallback handles traffic correctly with acceptable quality.

5. Keep prompt templates version-controlled alongside the code — Prompts are as important as code for LLM applications. Store them in the repository, review them in PRs, and tag them with the model version they were optimized for. When switching providers, the prompt template is the first thing that needs updating.

Common Issues

Responses differ significantly between providers — Switching from Claude to GPT-4o produces noticeably different output formatting, verbosity, and reasoning patterns. Create provider-specific system prompts and output parsers. Use a normalization layer that post-processes responses into a consistent format regardless of which provider generated them.

Rate limits hit during peak hours cause cascade failures — The primary provider throttles requests, overwhelming the fallback provider which also throttles. Implement request queuing with backpressure, spread requests across providers proactively (not just during failures), and cache responses aggressively to reduce total request volume.

Cost spikes from unexpectedly long responses — A single poorly constrained prompt generates a 10,000-token response, costing 10x the expected amount. Set max_tokens limits on every API call, monitor per-request costs in real-time, and implement circuit breakers that pause generation if a single request exceeds a cost threshold.