Operations

Controlled BetaVerification Required

Message Broker operator runbooks.

Message Broker operator runbooks for controlled-beta health, metrics, audit, diagnostics, storage inspection, quota rejection, replication status boundaries, and support evidence handoff.

Current availability

Private Beta. Public beta is scheduled for October 5, 2026.

Runbook Snapshot

Runbooks: 8
First checks: 12
Triage entries: 14
Review state: Pending

Runbook Summary

Message Broker operator runbook coverage
Runbook	Severity	Scope	First check	Evidence
Startup and health smoke	P2 to P4	Local broker process, /health, storage readiness, auth posture	Confirm the listener, node id, storage readiness, release metadata, and configured auth mode.	SMB-EVID-002, SMB-EVID-008
Metrics exposure check	P2 to P4	/metrics and metrics exporter configuration	Confirm HTTP metrics exposure before treating metrics_disabled as an incident.	SMB-EVID-002, DOCS-031
Audit query and export check	P2 to P4	/v1/audit/events and /v1/audit/export	Confirm protected-route auth, tenant filtering, page size, and payload-free audit output.	SMB-EVID-002, DOCS-031
Diagnostics bundle collection	P2 to P4	Local or remote mq diagnostics collect	Collect payload-free diagnostics and confirm --include-payloads is not used.	SMB-EVID-009, DOCS-032
Storage inspection triage	P2 to P4	Retention dry-run, tiering readiness, catalog, quarantine routes	Use read-only storage administration routes before changing stream or runtime state.	SMB-EVID-008, SMB-EVID-010
Quota rejection triage	P3 to P4	quota_exceeded responses on publish or consumer-group creation	Capture status, error body, tenant or stream scope, and configured quota context.	SMB-EVID-008, SMB-EVID-010
Replication status boundary	P2 to P4	Replication status, worker poll, pause, resume, target apply	Treat route output as operational signal only, not proof of cross-site outcomes.	SMB-EVID-005, SMB-EVID-010
Support evidence handoff	P2 to P4	Redacted route output, diagnostics bundle, config, and logs	Capture evidence before mutation and redact sensitive data before sharing.	DOCS-049, DOCS-051

Local Startup

Start the local controlled-beta cluster in one terminal, then run the first checks in a second terminal.

cd /path/to/Service.MessageBroker
export BROKER_CLUSTER_DIR="${TMPDIR:-/tmp}/message-broker-docs-051-cluster"
STORNAMICS_LOCAL_CLUSTER_DIR="$BROKER_CLUSTER_DIR" \
  bash Support/Start-LocalCluster.sh

First Checks

Configure check terminal

First check

export BROKER_ENDPOINT=http://127.0.0.1:9101
export BROKER_STREAM=docs-051.ops
export BROKER_GROUP=docs-051-workers
export BROKER_CONSUMER=docs-051-reader
export BROKER_DIAG_FILE="${TMPDIR:-/tmp}/message-broker-docs-051-diagnostics.json"

Expected signal

Endpoint, stream, group, consumer, and diagnostics path are set.

Capture: Endpoint value, stream name, consumer group, and diagnostics path.

Create stream

First check

curl -i -X PUT \
  -H 'Content-Type: application/json' \
  --data '{}' \
  "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM"

Expected signal

201 Created or 200 OK with stream metadata for the scoped smoke stream.

Capture: Status, stream name, response body, and route.

Publish smoke record

First check

curl -i -X POST \
  -H 'Content-Type: application/json' \
  --data '{"message_id":"docs-051-1","payload":"operator-smoke","durability_mode":"local-append"}' \
  "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM/records"

Expected signal

201 Created with offset, next offset, message id, and payload byte count.

Capture: Status, offset, message id, payload byte count, and stream name.

Health

First check

curl -i "$BROKER_ENDPOINT/health"

Expected signal

200 OK JSON with status, service, node id, storage status, auth mode, and release metadata.

Capture: Status, selected node, auth mode, storage status, and timestamp.

Metrics

First check

curl -i "$BROKER_ENDPOINT/metrics"

Expected signal

200 OK text when HTTP metrics exposure is enabled; 404 metrics_disabled means this runtime did not expose metrics.

Capture: Status, response body header, startup metrics flag, and exporter configuration.

Audit events

First check

curl -i "$BROKER_ENDPOINT/v1/audit/events?start_offset=0&max_records=25"

Expected signal

200 OK JSON with parsed audit events and record metadata when auth permits the administrator route.

Capture: Status, page parameters, event count, and redacted route names.

Audit export

First check

curl -i "$BROKER_ENDPOINT/v1/audit/export?start_offset=0&max_records=25"

Expected signal

200 OK application/x-ndjson with one payload-free audit record per line when auth permits the route.

Capture: Status, content type, record count, and redacted line sample.

Diagnostics

First check

mq cli --target "$BROKER_ENDPOINT" diagnostics collect \
  --bundle-id docs-051-local \
  --output "$BROKER_DIAG_FILE"

Expected signal

Diagnostics bundle JSON is written without payload contents.

Capture: Bundle path, schema/version field, redaction summary, and command status.

Retention dry-run

First check

curl -i "$BROKER_ENDPOINT/streams/$BROKER_STREAM/retention/dry-run?max_bytes=1048576"

Expected signal

200 OK with a non-mutating retention plan.

Capture: Stream, policy input, planned action summary, and timestamp.

Tiering readiness

First check

curl -i "$BROKER_ENDPOINT/streams/$BROKER_STREAM/tiering/readiness?max_age_ms=86400000"

Expected signal

200 OK with payload-free tiering readiness.

Capture: Readiness fields, stream, policy input, and timestamp.

Replay read

First check

curl -i "$BROKER_ENDPOINT/v1/streams/$BROKER_STREAM/records?start_offset=0&max_records=10&consumer_group=$BROKER_GROUP&consumer_id=$BROKER_CONSUMER"

Expected signal

200 OK with records, diagnostics, next offset, and end-of-stream signal.

Capture: Status, returned record count, consumer group, and redacted message ids.

Replication status

First check

curl -i "$BROKER_ENDPOINT/v1/replication/status"

Expected signal

200 OK with scheduler, transport, worker, checkpoint, target, or control status fields where configured.

Capture: Status, configured binding source, route body, and reviewer note.

Metrics Triage

Message Broker metrics triage
Symptom	Likely cause	First check	Remediation
/metrics returns 404	Broker was not started with HTTP metrics exposure.	Check startup flags for --metrics-exposure http.	Restart local broker with HTTP metrics exposure before opening a metrics incident.
Metrics route returns 401	Protected auth mode is enabled and no acceptable credential exists.	Confirm security.auth_mode in /health and the route auth reference.	Supply the accepted API key or trusted upstream identity header for the configured local mode.
Metrics output is incomplete	Scrape happened before activity or exporter update completed.	Publish/replay one smoke record and retry the scrape.	Capture the before/after scrape and command sequence; avoid claiming metric completeness.
Metrics labels look too broad	Tenant or stream label policy needs review.	Compare labels with the route/auth compatibility matrix and evidence index.	Keep tenant and quota wording verification-required until security and operations reviewers sign.

Audit Triage

Message Broker audit triage
Symptom	Likely cause	First check	Remediation
Audit route returns 401/403	Route requires administrator access outside insecure-local mode.	Confirm auth mode, API key, mTLS trust-file, or trusted-header setup.	Retry with valid local credentials or route through the configured trusted upstream.
Audit route returns no records	No audited activity happened in the selected offset window.	Run the health, stream create, publish, replay, or queue check again.	Adjust start_offset and max_records; do not infer audit loss from an empty early page.
Audit export cannot be parsed	Client treated NDJSON as a single JSON document.	Check Content-Type and inspect one line at a time.	Parse each line independently and include a redacted sample in the evidence package.
Audit storage error	Audit storage cannot be read or decoded by the runtime route.	Capture status, error body, stream/runtime context, and service log lines.	Escalate with the evidence package before restarting or deleting local state.

Storage Pressure And Quota Triage

Message Broker storage and quota triage
Symptom	Likely cause	First check	Remediation
quota_exceeded	Tenant publish or consumer-group quota rejected the request.	Capture status 429, error body, stream, consumer group, and quota config.	Reduce the request shape or adjust local quota config; do not classify expected quota rejection as an outage.
write_not_allowed	Storage admission rejected cursor, queue, or quarantine mutation.	Check route, stream, consumer group, and storage-admission message.	Preserve evidence and retry only after confirming storage state and reviewer-approved limits.
stream_not_found	Stream was not created in the active runtime data directory.	List streams with GET /v1/streams or recreate the scoped smoke stream.	Create the stream or point the client at the intended node/data directory.
Retention dry-run error	Invalid policy shape or missing stream.	Check max_bytes, max_age_ms, and stream name.	Correct the dry-run query; do not perform deletion or compaction because these routes are read-only.
Tiering readiness error	Invalid tiering query, missing stream, or manifest generation issue.	Capture status, route body, stream name, and policy query.	Keep the finding as storage inspection evidence; do not claim archive movement or S3 upload.
Queue persistence error	Delivery state could not be stored after a queue mutation.	Capture route, delivery id, subscription id, and error body.	Stop mutating the delivery until support reviews the evidence package.

Replication Status Boundary

Replication routes are useful operational signals, but they are not proof of cross-site durability, failover, recovery, availability, or topology outcomes.

Message Broker replication status checks
Check	Command	Expected signal	Boundary
Status	`curl -i "$BROKER_ENDPOINT/v1/replication/status"`	Route returns configured scheduler, transport, checkpoint, target, worker, or operator-control fields.	Use as route-level status only.
Worker poll	`curl -i -X POST "$BROKER_ENDPOINT/v1/replication/workers/poll"`	200 OK poll report when bindings are configured, or 409 replication_not_configured when they are not.	Missing binding is configuration state, not evidence of failed replication outcome.
Pause worker	`curl -i -X POST \ -H 'Content-Type: application/json' \ --data '{"stream":"docs-051.ops","source_site":"site-a","target_site":"site-b","reason":"operator check"}' \ "$BROKER_ENDPOINT/v1/replication/workers/pause"`	200 OK control status when the named worker exists, or 404 replication_worker_not_found.	Control status does not prove target application, RPO, RTO, failover, or recovery behavior.
Target apply error triage	`Review /v1/replication/target/apply error body from the affected request.`	Bounded errors such as stream mismatch, policy mismatch, gap, duplicate conflict, or payload limit.	Attach target-environment evidence before stronger replication wording can be accepted.

Capture ScopeRecord route, status, body, binding source, stream, sites, tenant scope, and timestamp.
Link Matrix RowLink the replication status or target-apply row before interpreting the response.
Name Evidence LevelState whether evidence is route-level, binding-level, target-apply-level, or topology-level.
Keep Claims In ReviewCustomer-facing outcome claims stay verification-required until operations and product reviewers accept them.

Support Evidence

Message Broker support evidence package
Include	Source	Redact or exclude
Health response	GET /health	Public hostname, environment-specific endpoint names, and unrelated headers.
Metrics scrape or disabled response	GET /metrics	Tenant identifiers, customer stream names, and labels that reveal private topology.
Audit events or export sample	/v1/audit/events and /v1/audit/export	API keys, bearer tokens, trusted headers, payloads, tenant identifiers, personal data, and secrets.
Diagnostics bundle	mq diagnostics collect	Payload contents, --include-payloads, tokens, certificates, private keys, customer IDs, and PII.
Storage inspection responses	Retention dry-run, tiering readiness, catalog, quarantine routes	Payloads, proprietary stream names, unrelated directory listings, and unreviewed archive claims.
Quota rejection evidence	quota_exceeded response and runtime configuration	Customer tenant identifiers, private stream names, and raw payload content.
Replication status or worker output	Replication status, worker poll, pause, resume, target apply	Secrets, target credentials, customer topology details, and outcome promises.
Compatibility, limits, and evidence links	DOCS-040, DOCS-042, SMB-EVID-005, SMB-EVID-010	Roadmap promises or unreviewed customer commitments.

Safety Rules

Keep Replication Bounded

Replication routes are route-level or binding-level evidence unless a reviewed target-environment packet proves stronger outcomes.

Keep Diagnostics Payload-Free

Diagnostics bundles should not include customer payloads. Reject or remove payload collection before support handoff.

Keep Quotas Out Of Outage Language

Expected quota rejection is admission control evidence. Treat it as an incident only when configured limits or error behavior are wrong.

Evidence

DOCS-051 evidence map
Runbook area	Evidence	Status
Health, metrics, audit, storage routes	DOCS-031, SMB-EVID-002, SMB-EVID-008	pending
CLI diagnostics and remote management	DOCS-032, SMB-EVID-009	pending
Route, auth, replication, and limit boundaries	DOCS-040, DOCS-042, SMB-EVID-005, SMB-EVID-010, SMB-EVID-011	pending
Message Broker operator runbook artifact	DOCS-051 and /products/message-broker/operations/operator-runbooks/	pending

Internal review records track this Message Broker runbook page and its evidence handoff requirements.

Safe Next Steps

Available: DOCS-031
Read HTTP reference
Use route shapes, permissions, errors, page sizes, metrics, audit, storage, and replication details when triaging broker requests.
Available: DOCS-032
Read CLI and SDK reference
Check lifecycle flags, diagnostics collection, local/remote CLI limits, and SDK entry points.
Available: DOCS-040
Compare route and auth scope
Separate controlled-beta route behavior from auth, replication, queue, CLI, SDK, and Kafka protocol boundaries.
Available
Review Message Broker evidence
Use SMB evidence records before strengthening runbook, support, security, replication, or compatibility wording.