Assess
Check Grafana alerts, Jenkins build status, and the CubeAPM service inventory.
grafana alert rule list -o jsonAn agent workspace wiring read-only observability, CI/CD, and infrastructure CLIs into one isolated credential environment so a coding agent can investigate incidents, run RCAs, and understand production behavior.
read-only, always
# Assess alerts and failed delivery jobs grafana alert rule list -o json jenkins job list --recursive --status FAILURE -o json # Follow the affected service into traces cubeapm traces search --service checkout --status error --last 1h -o json # Check whether the failure lines up with cluster events kubectl get events -n checkout --sort-by=.lastTimestamp | tail -30 # Correlate the timestamps and save the RCA under incidents/
start broad, then narrow
Check Grafana alerts, Jenkins build status, and the CubeAPM service inventory.
grafana alert rule list -o jsonSearch error traces and related logs, then open the trace waterfall for the affected service.
cubeapm traces search --status error --last 1hCompare error rate, latency, and service health across the incident window.
cubeapm metrics query 'error_rate' --last 1hReview deploy annotations, recent Jenkins builds, build logs, merged pull requests, and releases.
jenkins build list checkout/deploy --limit 5Trace error propagation through the dependency graph and check the supporting infrastructure.
cubeapm traces get <trace-id> -o jsonMatch deployment timestamps with error spikes, Grafana annotations, and cluster events.
kubectl get events -n checkout --sort-by=.lastTimestampA read-only toolbelt for a coding agent that investigates like your best on-call.
Move from an alert to traces, metrics, logs, deployment history, and infrastructure state with a disciplined RCA workflow.
Match error and latency windows with builds, releases, cluster events, queue lag, cache health, and database evidence.
Production-only conventions, read-only roles, session safeguards, command allowlists, and approval prompts keep investigations read-shaped.
Use twelve wired CLIs plus optional Elasticsearch access without switching credential contexts or stitching together separate workspaces.
Run the same workspace under Claude Code, Codex CLI, OpenCode, or another agent runtime that follows AGENTS.md.
direnv pins CLI profiles and environment credentials inside the clone, keeping production access separate from global configuration.
Wire together twelve read-only CLIs in one isolated credential environment so a coding agent can correlate signals across systems in one session.
correlate the timestamps
grafana alert rule list -o jsonEstablish the active alert window.
jenkins job list --recursive --status FAILURE -o jsonCompare build start and completion times.
cubeapm metrics query 'rate(http_requests_total{status=~"5.."}[5m])' -o jsonLocate the error spike.
kubectl get events -n checkout --sort-by=.lastTimestamp | tail -30Check whether the failure lines up with cluster events.