Specialist Data Ally

An autonomous data analysis agent that helps explore, clean, transform, and visualize datasets, combining statistical analysis with practical data engineering to turn raw data into actionable insights.

When to Use This Agent

Choose Data Ally when:

• Exploring unfamiliar datasets to understand structure and quality
• Cleaning messy data with missing values, duplicates, and inconsistencies
• Building data transformation pipelines for analytics or ML preparation
• Creating statistical summaries and visualizations for stakeholder reports
• Performing ad-hoc queries and aggregations across large datasets

Consider alternatives when:

• Building production ML models (use an AI engineer agent)
• Designing data warehouse architecture (use a data architect agent)
• Creating interactive dashboards (use a BI tool like Metabase or Tableau)

Quick Start

Copy
# .claude/agents/specialist-data-ally.yml
name: Data Ally
model: claude-sonnet-4-20250514
tools:
  - Read
  - Write
  - Bash
  - Glob
  - Grep
prompt: |
  You are a data analysis expert. Help explore, clean, transform,
  and visualize datasets. Use pandas, numpy, and matplotlib/seaborn
  for analysis. Always start by understanding the data shape,
  types, and quality before performing analysis.

Example invocation:

Copy
claude --agent specialist-data-ally "Analyze the sales data in
  data/sales_2024.csv - find top products, seasonal trends,
  and customer segments, then create a summary report"

Core Concepts

Data Exploration Workflow

Copy
import pandas as pd

# Step 1: Load and inspect
df = pd.read_csv('data.csv')
print(f"Shape: {df.shape}")
print(f"Columns: {df.dtypes}")
print(f"Missing: {df.isnull().sum()}")
print(f"Duplicates: {df.duplicated().sum()}")

# Step 2: Statistical summary
print(df.describe(include='all'))

# Step 3: Distribution checks
for col in df.select_dtypes(include='number'):
    print(f"{col}: skew={df[col].skew():.2f}, "
          f"kurtosis={df[col].kurtosis():.2f}")

# Step 4: Relationship exploration
print(df.corr(numeric_only=True))

Data Quality Checks

Check	Method	Action
Missing values	df.isnull().sum()	Impute, drop, or flag
Duplicates	df.duplicated()	Remove or investigate
Type mismatches	df.dtypes	Cast or parse
Outliers	IQR or Z-score	Cap, remove, or investigate
Cardinality	df.nunique()	Encode or group rare values
Date parsing	pd.to_datetime()	Standardize formats
String cleaning	.str.strip().lower()	Normalize text

Transformation Patterns

Copy
# Chained transformation pipeline
result = (
    df
    .pipe(remove_duplicates)
    .pipe(handle_missing_values)
    .pipe(normalize_dates, col='created_at')
    .pipe(encode_categoricals, cols=['region', 'product_type'])
    .pipe(create_features, target='revenue')
    .pipe(validate_output)
)

Configuration

Parameter	Description	Default
max_rows_display	Rows to show in output previews	20
missing_threshold	Drop columns above this % missing	50%
outlier_method	Outlier detection strategy	IQR
date_format	Default date parsing format	Auto-detect
encoding	File encoding assumption	UTF-8
sample_size	Rows to sample for large datasets	10,000
visualization_lib	Charting library preference	Matplotlib

Best Practices

1. Always inspect before transforming. Run shape, dtypes, describe(), and head() before writing any transformation code. Assumptions about data formats, ranges, and completeness are the most common source of silent errors. A column named "price" might contain strings with currency symbols, and a "date" column might have mixed formats across rows.

2. Create reproducible transformation pipelines. Use function-based transformations that can be chained with .pipe() rather than ad-hoc cell-by-cell notebook mutations. Each function should take a DataFrame and return a DataFrame. This pattern makes pipelines testable, reusable, and easy to debug by inserting inspection points between steps.

3. Handle missing data intentionally, not by default. Before dropping or imputing missing values, understand why they're missing. Data missing completely at random can be imputed safely. Data missing because of a systematic reason (users who didn't complete a form) carries information that imputation destroys. Document your missing data strategy and its assumptions.

4. Validate outputs against expected constraints. After transformations, check that results make sense: are all dates in the expected range? Are prices positive? Does the row count match expectations? Add assertion checks to your pipeline that catch data quality issues before they propagate to downstream consumers.

5. Sample large datasets for exploration, use full data for final analysis. Working with a 10,000-row sample lets you iterate quickly on transformations and visualizations. Once the pipeline is correct, run it on the full dataset. This approach saves time without sacrificing accuracy, as long as the sample is representative.

Common Issues

Memory errors when loading large CSV files. Use chunked reading with pd.read_csv(chunksize=N) to process large files in pieces. Specify column dtypes explicitly to reduce memory usage—a column of integers stored as float64 uses twice the memory necessary. For datasets larger than available RAM, consider DuckDB or Polars, which handle out-of-core processing natively.

Inconsistent results between notebook runs. This happens when cells are executed out of order or when in-place modifications create hidden state. Use the .pipe() pattern to build deterministic transformation chains. Restart the kernel and run all cells sequentially before sharing results. Better yet, move finalized analysis into Python scripts that can be run reproducibly.

Visualizations don't communicate the intended message. Choose the chart type based on what you're showing: trends over time (line chart), comparisons (bar chart), distributions (histogram or box plot), relationships (scatter plot), or composition (stacked bar or pie). Label axes clearly, use consistent color schemes, and include annotations that highlight the key insight rather than forcing the viewer to discover it themselves.