Comet Opik Strategist

Instrument, observe, and optimize LLM applications using Comet Opik tracing, evaluation, and experiment management.

When to Use This Agent

Choose this agent when you need to:

• Integrate Opik tracing into an LLM application to capture prompts, completions, latency, and token costs across inference calls
• Design prompt experiments with versioned datasets, evaluation metrics, and side-by-side comparison dashboards
• Diagnose production regressions by analyzing trace spans, feedback scores, and token-usage anomalies in Opik workspaces

Consider alternatives when:

• Your observability needs cover only traditional APM metrics without LLM-specific telemetry
• You require model training or fine-tuning orchestration rather than inference monitoring

Quick Start

Configuration

Copy
name: comet-opik-strategist
type: agent
category: ml-observability

Example Invocation

Copy
claude agent:invoke comet-opik-strategist "Add Opik tracing to our RAG pipeline and create a hallucination eval"

Example Output

Opik Integration - rag-pipeline-v2
Workspace: acme-ml | Project: rag-pipeline-v2

[TRACE] @opik.track() applied to 4 functions:
  retrieve_documents (retriever) | rerank_results (processor)
  generate_answer (llm) | format_response (post_processor)

[EXPERIMENT] hallucination-eval-2026-03-15
  Dataset: golden_qa_v3 (120 samples)
  Metrics: hallucination_score, faithfulness, answer_relevancy
  Baseline: v2.1 | Variant: v2.2-guardrails | ETA: 8 min

Core Concepts

Opik Observability Overview

Aspect	Details
Trace model	Hierarchical spans capturing retrieval, processing, generation with parent-child relationships
Key metrics	Latency per span, token usage, cost estimation, feedback scores, custom evaluation metrics
Prompt versioning	Immutable versions in Opik library with metadata tags for rollback and A/B comparison
Experiments	Dataset-driven evaluations applying scoring functions to outputs with aggregated metric tables
Organization	Projects group traces; datasets store labeled examples; experiments link datasets to prompts

Opik Trace Architecture

+----------------+      +------------------+      +----------------+
| Application    | -->  | @opik.track()    | -->  | Opik Backend   |
| Code           |      | Decorator / SDK  |      | (Cloud or OSS) |
+----------------+      +------------------+      +----------------+
                                |                         |
                        +-------v-------+         +-------v-------+
                        | Span Tree     |         | Dashboard     |
                        | (trace, spans |         | Traces, Exps  |
                        |  metadata)    |         | Metrics View  |
                        +---------------+         +---------------+

Configuration

Parameter	Type	Default	Description
opik_workspace	string	-	Workspace slug where traces and experiments are stored
opik_api_key	string	-	API key for Opik backend authentication
project_name	string	default	Project name for grouping related traces
trace_sample_rate	float	1.0	Fraction of requests to trace (0.0-1.0) for volume control
evaluation_metrics	list	["hallucination","relevancy"]	Default scoring functions for experiments

Best Practices

1. Instrument at span boundaries - Place @opik.track() at each logical step rather than wrapping the entire pipeline, enabling isolation of latency and quality regressions per stage.

2. Version every prompt change - Store prompts with semantic version tags before deploying. This creates an auditable history and enables experiment comparison on the same dataset.

3. Build golden datasets incrementally - Start with 30-50 curated QA pairs, expand as edge cases surface. Quality evaluation data matters more than volume for detecting hallucination.

4. Sample traces in production - Set trace_sample_rate below 1.0 for high-throughput services. A 10% sample at 10K requests/hour still yields 1,000 daily traces.

5. Automate regression detection - Connect experiment results to CI/CD to block deployment when key metrics drop below thresholds.

Common Issues

1. Missing spans in trace tree - Async functions not awaited before trace context closes produce orphaned spans. Ensure all tracked coroutines complete within parent scope.

2. High storage costs from verbose metadata - Logging full document contents in retriever spans balloons storage. Log only document IDs and relevance scores.

3. Experiment metric variance - Small datasets produce unstable scores. Increase to 100+ samples or use bootstrap confidence intervals.