Observability
Read alerts, traces, metrics, logs, cloud signals and Elasticsearch evidence through reckon run.
Start with Grafana and CubeAPM. Add AWS when application telemetry is thin or cannot explain an infrastructure symptom, and Elasticsearch when a service's logs live there instead of CubeAPM.
Grafana: alerts and change markers
reckon run grafana -- alert rule list -o json
reckon run grafana -- alert silence list -o json
reckon run grafana -- annotation list --tags deploy --from <time> --to <time> -o json
reckon run grafana -- datasource list -o jsonAlert rules show what is monitored and at which thresholds; they are configuration, not firing state. Silences explain missing notifications, deploy annotations give correlation points, and the datasource list shows where each panel's data comes from when a gap may be an observability problem.
CubeAPM: services, traces, logs and metrics
CubeAPM usually holds every environment in one server, so every read carries
the active environment's label (PROD below; RECKON.md lists yours). Reckon
adds --env to traces search and traces services when it is missing,
refuses metrics and logs queries that do not name the label, and refuses any
command that mentions another environment's label.
reckon run cubeapm -- traces services --env PROD --last 1h -o json
reckon run cubeapm -- traces search --service <svc> --env PROD --status error --last 1h -o json
reckon run cubeapm -- traces get <trace-id> --last 2h -o json
reckon run cubeapm -- logs query 'error' --service <svc> --stream '{env="PROD"}' --last 1h -o json
reckon run cubeapm -- logs hits --query '_stream:{env="PROD"} AND level:error' --last 6h --step 15m -o jsonCross-check the symptom with PromQL, with the label in every selector, and map how the failure moves between services:
reckon run cubeapm -- metrics query-range 'sum(rate(http_requests_total{env="PROD",status=~"5.."}[5m]))' --last 1h --step 60s -o json
reckon run cubeapm -- metrics label-values service --match '{env="PROD"}' -o json
reckon run cubeapm -- logs field-names --last 1h -o json
reckon run cubeapm -- traces dependencies --last 1h -o jsonMetric names and label keys depend on your instrumentation: some families use
cube.environment instead of env. Check with metrics label-values and
logs field-names before you trust an empty result, and record what you
learn in knowledge/.
AWS: infrastructure corroboration
reckon run aws -- cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=<lb> --start-time <ISO> --end-time <ISO> --period 60 --statistics Sum --output json
reckon run aws -- sqs get-queue-attributes --queue-url <url> --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --output json
reckon run aws -- ecs describe-services --cluster <cluster> --services <service> --output json
reckon run aws -- logs filter-log-events --log-group-name <group> --start-time <epoch-ms> --end-time <epoch-ms> --filter-pattern '"ERROR"' --output jsonThese reads confirm load balancer 5xx responses, queue depth, ECS service
events, or CloudWatch errors in the same window. Message age is the AWS/SQS
CloudWatch metric ApproximateAgeOfOldestMessage. Reckon allows only
read-shaped AWS commands.
Elasticsearch: log stores
reckon run elasticsearch -- cluster health -o json
reckon run elasticsearch -- index list --pattern '*<svc>*' -o json
reckon run elasticsearch --stdin -- search query <index> -f - --size 20 --sort '@timestamp:desc' -o json < query.json
reckon run elasticsearch -- search sql --query 'SELECT "@timestamp", message FROM "<index>" ORDER BY "@timestamp" DESC LIMIT 20' -o jsonPass the Query DSL body on standard input with --stdin and -f -, and bound
it with a @timestamp range. --from is a document offset, not a time.