Operations

Message Broker operator runbooks.

Message Broker operator runbooks for private beta health, metrics, audit, diagnostics, storage inspection, quota rejection, replication status boundaries, and support details handoff.

Current availability

Private Beta. Public beta is scheduled for .

Private BetaConfirm First

Runbook Snapshot

Runbooks
8
First checks
13
Triage entries
14
Documentation status
Source-scoped

Runbook Summary

Message Broker operator runbook coverage
RunbookSeverityScopeFirst checkSource scope
Startup and health smokeInformationalLocal broker process, /health, storage readiness, auth postureConfirm the listener, node id, storage readiness, release metadata, and configured auth mode.documented source scope
Metrics exposure checkInformational/metrics and metrics exporter configurationConfirm HTTP metrics exposure before treating metrics_disabled as an incident.documented source scope
Audit query and export checkInformational/v1/audit/events and /v1/audit/exportConfirm protected-route auth, tenant filtering, page size, and payload-free audit output.documented source scope
Diagnostics bundle collectionInformationalLocal or remote mq diagnostics collectCollect payload-free diagnostics and confirm --include-payloads is not used.documented source scope
Storage inspection triageInformationalRetention dry-run, tiering readiness, catalog, quarantine routesUse read-only storage administration routes before changing stream or runtime state.documented source scope
Quota rejection triageBoundary checkquota_exceeded responses on publish or consumer-group creationCapture status, error body, tenant or stream scope, and configured quota context.documented source scope
Replication status boundaryInformationalReplication status, worker poll, pause, resume, target applyTreat route output as operational signal only, not proof of cross-site outcomes.documented source scope
Support details handoffInformationalRedacted route output, diagnostics bundle, config, and logsCapture evidence before mutation and redact sensitive data before sharing.documented source scope

Local Startup

Replace the example source path, then start the local private beta cluster in one terminal. The command creates a unique workspace for cluster data, logs, PID files, and diagnostics. Keep the printed values for the check terminal and retain the workspace if its evidence is still needed.

cd /path/to/Service.MessageBroker
export BROKER_SOURCE_DIR="$PWD"
export BROKER_RUNBOOK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/message-broker-runbook-XXXXXX")"
export BROKER_CLUSTER_DIR="$BROKER_RUNBOOK_DIR/cluster"
printf '%s
' 'message-broker-operator-runbook-v1' >   "$BROKER_RUNBOOK_DIR/.stornamics-message-broker-runbook"
printf 'Message Broker runbook workspace: %s
' "$BROKER_RUNBOOK_DIR"
printf 'Run these in the check terminal:
export BROKER_SOURCE_DIR=%q
export BROKER_RUNBOOK_DIR=%q
'   "$BROKER_SOURCE_DIR" "$BROKER_RUNBOOK_DIR"
STORNAMICS_LOCAL_CLUSTER_DIR="$BROKER_CLUSTER_DIR"   bash Support/Start-LocalCluster.sh

First Checks And Cleanup

Configure check terminal

First check

# First set BROKER_SOURCE_DIR and BROKER_RUNBOOK_DIR to the values
# printed by the startup terminal.
: "${BROKER_SOURCE_DIR:?Set BROKER_SOURCE_DIR to the printed source directory.}"
: "${BROKER_RUNBOOK_DIR:?Set BROKER_RUNBOOK_DIR to the printed runbook workspace.}"
export BROKER_CLUSTER_DIR="$BROKER_RUNBOOK_DIR/cluster"
export BROKER_DIAG_FILE="$BROKER_RUNBOOK_DIR/diagnostics.json"
test -f "$BROKER_RUNBOOK_DIR/.stornamics-message-broker-runbook"
test -d "$BROKER_CLUSTER_DIR"
export BROKER_ENDPOINT=http://127.0.0.1:9101
export BROKER_STREAM=source record.ops
export BROKER_GROUP=source record-workers
export BROKER_CONSUMER=source record-reader

Expected signal

The printed source and runbook workspace are accepted, and endpoint, stream, consumer, cluster, and diagnostics values are set.

Capture: Source revision, endpoint, stream, consumer group, runbook workspace, cluster path, and diagnostics path.

Create stream

First check

curl -i -X PUT \
  -H 'Content-Type: application/json' \
  --data '{}' \
  "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM"

Expected signal

201 Created or 200 OK with stream metadata for the scoped smoke stream.

Capture: Status, stream name, response body, and route.

Publish smoke record

First check

curl -i -X POST \
  -H 'Content-Type: application/json' \
  --data '{"message_id":"source record-1","payload":"operator-smoke","durability_mode":"local-append"}' \
  "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM/records"

Expected signal

201 Created with offset, next offset, message id, and payload byte count.

Capture: Status, offset, message id, payload byte count, and stream name.

Health

First check

curl -i "$BROKER_ENDPOINT/health"

Expected signal

200 OK JSON with status, service, node id, storage status, auth mode, and release metadata.

Capture: Status, selected node, auth mode, storage status, and timestamp.

Metrics

First check

curl -i "$BROKER_ENDPOINT/metrics"

Expected signal

200 OK text when HTTP metrics exposure is enabled; 404 metrics_disabled means this runtime did not expose metrics.

Capture: Status, response body header, startup metrics flag, and exporter configuration.

Audit events

First check

curl -i "$BROKER_ENDPOINT/v1/audit/events?start_offset=0&max_records=25"

Expected signal

200 OK JSON with parsed audit events and record metadata when auth permits the administrator route.

Capture: Status, page parameters, event count, and redacted route names.

Audit export

First check

curl -i "$BROKER_ENDPOINT/v1/audit/export?start_offset=0&max_records=25"

Expected signal

200 OK application/x-ndjson with one payload-free audit record per line when auth permits the route.

Capture: Status, content type, record count, and redacted line sample.

Diagnostics

First check

mq cli --target "$BROKER_ENDPOINT" diagnostics collect \
  --bundle-id source record-local \
  --output "$BROKER_DIAG_FILE"

Expected signal

Diagnostics bundle JSON is written without payload contents.

Capture: Bundle path, schema/version field, redaction summary, and command status.

Retention dry-run

First check

curl -i "$BROKER_ENDPOINT/streams/$BROKER_STREAM/retention/dry-run?max_bytes=1048576"

Expected signal

200 OK with a non-mutating retention plan.

Capture: Stream, policy input, planned action summary, and timestamp.

Tiering readiness

First check

curl -i "$BROKER_ENDPOINT/streams/$BROKER_STREAM/tiering/readiness?max_age_ms=86400000"

Expected signal

200 OK with payload-free tiering readiness.

Capture: Readiness fields, stream, policy input, and timestamp.

Replay read

First check

curl -i "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM/records?start_offset=0&max_records=10&consumer_group=$BROKER_GROUP&consumer_id=$BROKER_CONSUMER"

Expected signal

200 OK with records, diagnostics, next offset, and end-of-stream signal.

Capture: Status, returned record count, consumer group, and redacted message ids.

Replication status

First check

curl -i "$BROKER_ENDPOINT/v1/replication/status"

Expected signal

200 OK with scheduler, transport, worker, checkpoint, target, or control status fields where configured.

Capture: Status, configured binding source, route body, and operator note.

Stop cluster and clean local smoke workspace

Cleanup

# This permanently deletes the printed workspace, including cluster data,
# logs, and the diagnostics bundle. Retain it instead if evidence is still needed.
: "${BROKER_SOURCE_DIR:?Set BROKER_SOURCE_DIR to the printed source directory.}"
: "${BROKER_RUNBOOK_DIR:?Set BROKER_RUNBOOK_DIR to the printed runbook workspace.}"
: "${BROKER_CLUSTER_DIR:?Set BROKER_CLUSTER_DIR from BROKER_RUNBOOK_DIR.}"
: "${BROKER_DIAG_FILE:?Set BROKER_DIAG_FILE from BROKER_RUNBOOK_DIR.}"
expected_prefix="${TMPDIR:-/tmp}/message-broker-runbook-"
case "$BROKER_RUNBOOK_DIR" in
  "$expected_prefix"*) ;;
  *)
    printf 'Refusing to remove unexpected Message Broker workspace: %s
'       "$BROKER_RUNBOOK_DIR" >&2
    exit 1
    ;;
esac
if [ "$BROKER_CLUSTER_DIR" != "$BROKER_RUNBOOK_DIR/cluster" ] ||    [ "$BROKER_DIAG_FILE" != "$BROKER_RUNBOOK_DIR/diagnostics.json" ]; then
  printf 'Refusing Message Broker cleanup with mismatched runbook paths.
' >&2
  exit 1
fi
if [ ! -f "$BROKER_RUNBOOK_DIR/.stornamics-message-broker-runbook" ] ||    ! grep -qx 'message-broker-operator-runbook-v1'      "$BROKER_RUNBOOK_DIR/.stornamics-message-broker-runbook"; then
  printf 'Refusing Message Broker cleanup without the runbook ownership marker: %s
'     "$BROKER_RUNBOOK_DIR" >&2
  exit 1
fi
if [ ! -f "$BROKER_SOURCE_DIR/Support/Stop-LocalCluster.sh" ]; then
  printf 'Cannot find the Message Broker stop helper under: %s
'     "$BROKER_SOURCE_DIR" >&2
  exit 1
fi
if ! STORNAMICS_LOCAL_CLUSTER_DIR="$BROKER_CLUSTER_DIR"   bash "$BROKER_SOURCE_DIR/Support/Stop-LocalCluster.sh"; then
  printf 'Message Broker shutdown failed; retaining workspace: %s
'     "$BROKER_RUNBOOK_DIR" >&2
  exit 1
fi
for node_number in 1 2 3; do
  node_id="local-$node_number"
  port=$((9100 + node_number))
  pid_file="$BROKER_CLUSTER_DIR/pids/node-$node_number.pid"
  if [ -e "$pid_file" ]; then
    printf 'Message Broker shutdown left a PID file; retaining workspace: %s
'       "$pid_file" >&2
    exit 1
  fi
  response="$(curl -sS --max-time 1     "http://127.0.0.1:$port/health" 2>&1 || true)"
  if [[ "$response" == *'"service":"stornamics-message-broker"'* &&         "$response" == *""node_id":"$node_id""* ]]; then
    printf 'Message Broker node still responds on port %s; retaining workspace.
'       "$port" >&2
    exit 1
  fi
done
rm -rf -- "$BROKER_RUNBOOK_DIR"

Expected signal

The product stop helper stops all three recorded nodes, shutdown verification passes, and only the guarded runbook workspace is removed.

Capture: Stop-helper output and the retained workspace path before cleanup; do not run removal while evidence is still needed.

Metrics Triage

Message Broker metrics triage
SymptomLikely causeFirst checkRemediation
/metrics returns 404Broker was not started with HTTP metrics exposure.Check startup flags for --metrics-exposure http.Restart local broker with HTTP metrics exposure before opening a metrics incident.
Metrics route returns 401Protected auth mode is enabled and no acceptable credential exists.Confirm security.auth_mode in /health and the route auth reference.Supply the accepted API key or trusted upstream identity header for the configured local mode.
Metrics output is incompleteScrape happened before activity or exporter update completed.Publish/replay one smoke record and retry the scrape.Capture the before/after scrape and command sequence; avoid claiming metric completeness.
Metrics labels look too broadTenant or stream label policy needs review.Compare labels with the route/auth compatibility matrix and limits and compatibility guidance.Keep tenant and quota wording verification-required until security and operations support teams sign.

Audit Triage

Message Broker audit triage
SymptomLikely causeFirst checkRemediation
Audit route returns 401/403Route requires administrator access outside insecure-local mode.Confirm auth mode, API key, mTLS trust-file, or trusted-header setup.Retry with valid local credentials or route through the configured trusted upstream.
Audit route returns no recordsNo audited activity happened in the selected offset window.Run the health, stream create, publish, replay, or queue check again.Adjust start_offset and max_records; do not infer audit loss from an empty early page.
Audit export cannot be parsedClient treated NDJSON as a single JSON document.Check Content-Type and inspect one line at a time.Parse each line independently and include a redacted sample in the evidence package.
Audit storage errorAudit storage cannot be read or decoded by the runtime route.Capture status, error body, stream/runtime context, and service log lines.Escalate with the evidence package before restarting or deleting local state.

Storage Pressure And Quota Triage

Message Broker storage and quota triage
SymptomLikely causeFirst checkRemediation
quota_exceededTenant publish or consumer-group quota rejected the request.Capture status 429, error body, stream, consumer group, and quota config.Reduce the request shape or adjust local quota config; do not classify expected quota rejection as an outage.
write_not_allowedStorage admission rejected cursor, queue, or quarantine mutation.Check route, stream, consumer group, and storage-admission message.Preserve evidence and retry only after confirming storage state and accepted limits.
stream_not_foundStream was not created in the active runtime data directory.List streams with GET /v1/streams or recreate the scoped smoke stream.Create the stream or point the client at the intended node/data directory.
Retention dry-run errorInvalid policy shape or missing stream.Check max_bytes, max_age_ms, and stream name.Correct the dry-run query; do not perform deletion or compaction because these routes are read-only.
Tiering readiness errorInvalid tiering query, missing stream, or manifest generation issue.Capture status, route body, stream name, and policy query.Keep the finding as storage inspection evidence; do not claim archive movement or S3 upload.
Queue persistence errorDelivery state could not be stored after a queue mutation.Capture route, delivery id, subscription id, and error body.Stop mutating the delivery until support reviews the evidence package.

Replication Status Boundary

Replication routes are useful operational signals, but they are not proof of cross-site durability, failover, recovery, availability, or topology outcomes.

Message Broker replication status checks
CheckCommandExpected signalBoundary
Statuscurl -i "$BROKER_ENDPOINT/v1/replication/status"Route returns configured scheduler, transport, checkpoint, target, worker, or operator-control fields.Use as route-level status only.
Worker pollcurl -i -X POST "$BROKER_ENDPOINT/v1/replication/workers/poll"200 OK poll report when bindings are configured, or 409 replication_not_configured when they are not.Missing binding is configuration state, not evidence of failed replication outcome.
Pause workercurl -i -X POST \ -H 'Content-Type: application/json' \ --data '{"stream":"source record.ops","source_site":"site-a","target_site":"site-b","reason":"operator check"}' \ "$BROKER_ENDPOINT/v1/replication/workers/pause"200 OK control status when the named worker exists, or 404 replication_worker_not_found.Control status does not prove target application, RPO, RTO, failover, or recovery behavior.
Target apply error triageReview /v1/replication/target/apply error body from the affected request.Bounded errors such as stream mismatch, policy mismatch, gap, duplicate conflict, or payload limit.Attach target-environment evidence before stronger replication wording can be accepted.
  1. Capture ScopeRecord route, status, body, binding source, stream, sites, tenant scope, and timestamp.
  2. Link Matrix RowLink the replication status or target-apply row before interpreting the response.
  3. Name Review LevelState whether evidence is route-level, binding-level, target-apply-level, or topology-level.
  4. Keep Claims In ReviewCustomer-facing outcome claims stay verification-required until operations and product the appropriate team accepts them.

Support Review Package

Message Broker support review package
IncludeSourceRedact or exclude
Health responseGET /healthPublic hostname, environment-specific endpoint names, and unrelated headers.
Metrics scrape or disabled responseGET /metricsTenant identifiers, customer stream names, and labels that reveal private topology.
Audit events or export sample/v1/audit/events and /v1/audit/exportAPI keys, bearer tokens, trusted headers, payloads, tenant identifiers, personal data, and secrets.
Diagnostics bundlemq diagnostics collectPayload contents, --include-payloads, tokens, certificates, private keys, customer IDs, and PII.
Storage inspection responsesRetention dry-run, tiering readiness, catalog, quarantine routesPayloads, proprietary stream names, unrelated directory listings, and unreviewed archive claims.
Quota rejection source scopequota_exceeded response and runtime configurationCustomer tenant identifiers, private stream names, and raw payload content.
Replication status or worker outputReplication status, worker poll, pause, resume, target applySecrets, target credentials, customer topology details, and outcome promises.
Compatibility, limits, and limits linksdocumented source scopeRoadmap promises or unreviewed customer commitments.

Safety Rules

Keep Replication Bounded

Replication routes are route-level or binding-level evidence unless a reviewed target-environment packet proves stronger outcomes.

Keep Diagnostics Payload-Free

Diagnostics bundles should not include customer payloads. Reject or remove payload collection before support handoff.

Keep Quotas Out Of Outage Language

Expected quota rejection is admission control evidence. Treat it as an incident only when configured limits or error behavior are wrong.

Source scope

Source scope map
Runbook areaSource scopeStatus
Health, metrics, audit, storage routesdocumented source scopeSource-scoped
CLI diagnostics and remote managementdocumented source scopeSource-scoped
Route, auth, replication, and limit boundariesdocumented source scopeSource-scoped
Message Broker operator runbook artifactdocumented source scope and /products/message-broker/operations/operator-runbooks/Source-scoped

Internal public documentation boundaries track this Message Broker runbook page and its support details handoff requirements.

Safe Next Steps

  • Available

    Read HTTP reference

    Use route shapes, permissions, errors, page sizes, metrics, audit, storage, and replication details when triaging broker requests.

  • Available

    Read CLI and SDK reference

    Check lifecycle flags, diagnostics collection, local/remote CLI limits, and SDK entry points.

  • Available

    Compare route and auth scope

    Separate private beta route behavior from auth, replication, queue, CLI, SDK, and Kafka protocol boundaries.

  • Available

    Review Message Broker limits

    Use SMB public documentation boundaries before strengthening runbook, support, security, replication, or compatibility wording.