Power Simulation Calibrator

Calibrate simulation models against real-world outcomes with systematic validation, accuracy scoring, and continuous improvement recommendations.

When to Use This Command

Run this command when...

• You have a simulation model producing outputs that diverge from observed reality and need to identify which parameters require adjustment
• You want to establish a formal validation framework that scores your model's accuracy against historical data
• Your simulation needs periodic recalibration as the underlying system evolves and new outcome data becomes available

Do NOT use this command when...

• You are building a simulation from scratch -- use digital-twin-auto or monte-carlo-simulator-runner first
• Your model has no historical outcome data to validate against

Quick Start

Copy
# .claude/commands/power-simulation-calibrator.md
# Calibrate simulation accuracy
Calibrate simulation: $ARGUMENTS

Copy
# Run the command
claude "power-simulation-calibrator demand forecasting model with Q1-Q3 actuals showing 15% overestimation"

Expected output:
- Parameter sensitivity ranking for calibration priority
- Recommended parameter adjustments with magnitudes
- Before/after accuracy comparison
- Validation scores (MAPE, RMSE, correlation)
- Recalibration schedule recommendation

Core Concepts

Concept	Description
Calibration Target	The accuracy metric being optimized (MAPE, RMSE, R-squared)
Parameter Sensitivity	Ranking of which model parameters most affect prediction accuracy
Validation Dataset	Historical outcomes used to score model predictions
Drift Detection	Identifying when model accuracy degrades over time
Improvement Cycle	Iterative loop of adjust, validate, measure, repeat

Calibration Workflow:

  Simulation Model + Actuals
       |
  [Compare Predictions vs Outcomes]
       |
  [Calculate Error Metrics]
       |
  [Rank Parameter Sensitivity]
       |
  [Adjust Top Parameters]
       |
  [Re-validate]----> Meets Target?
       |                  |
      No               Yes
       |                  |
  [Next Iteration]   Deploy + Monitor

Configuration

Parameter	Default	Description
Accuracy Target	80-95%	Desired prediction accuracy level based on use case
Validation Split	70/30	Ratio of training to holdout data for validation
Max Iterations	10	Maximum calibration adjustment rounds before reporting
Error Metric	MAPE	Primary accuracy metric (MAPE, RMSE, MAE, R-squared)
Drift Threshold	5% degradation	Accuracy drop that triggers recalibration alert

Best Practices

1. Provide actual outcome data -- the calibrator is only as good as the validation dataset. Include real numbers, dates, and measurement conditions
2. Specify your accuracy target -- "mission-critical 95%" calibration differs from "exploratory 70%" in how aggressively parameters are tuned
3. Hold out recent data -- use the most recent period as a validation set rather than random sampling to test the model's forward-looking accuracy
4. Calibrate one subsystem at a time -- adjusting everything simultaneously makes it impossible to attribute improvement to specific changes
5. Schedule recalibration -- set a cadence (monthly, quarterly) based on how fast the underlying system changes, and re-run this command each cycle

Common Issues

1. Accuracy improves on training data but not holdout -- the model is overfitting to historical noise. Reduce parameter flexibility or increase holdout size
2. Multiple parameters compensate for each other -- this indicates structural model issues. Review whether the simulation architecture correctly represents the real system
3. Diminishing returns on iterations -- if accuracy plateaus below your target, the model structure itself may need revision rather than parameter tuning