Operations standard

Verification Required

Runbook template.

Use this standard to draft operator runbooks with symptoms, first checks, diagnosis, remediation, escalation, evidence capture, limits, and reviewer routing that are clear before a responder needs them.

Template Snapshot

Required sections
9
Severity levels
4
Reviewer lanes
5
Review state
Routed

Required Sections

Runbook section standard
SectionPurposeApproval check
SummaryState the incident shape, affected product surface, severity rule, and first safe action.Owner confirms scope and product boundary.
SymptomsList user-facing and operator-facing signals with source, severity, and evidence to capture.Operations reviewer confirms signals are actionable.
Prerequisites and safetyName access, environment, privacy, data sensitivity, and commands that must not be run first.Operations and support reviewers confirm safety boundaries.
First checksProvide read-only or low-risk checks before remediation.Docs QA reviewer confirms commands and expected output are inspectable.
DiagnosisMap findings to likely causes and evidence records.Product SME confirms the cause map is not stronger than current evidence.
RemediationGive stepwise remediation, verification, rollback, and follow-up actions.Operations reviewer confirms steps are safe for the stated scope.
EscalationName escalation triggers, owners, evidence bundle contents, and redaction rules.Support reviewer confirms handoff expectations.
EvidenceTrack source, test, fixture, release note, support record, or manual review evidence.Evidence owner keeps status and last checked date current.
Limits and next stepsState unsupported modes, unreviewed claims, and related runbooks or limits pages.Docs lead confirms the runbook does not overclaim.

Severity Model

Initial severity and response guidance
SeverityUse whenRequired action
P1Broad customer-visible service interruption, data admission stopped, or unsafe remediation risk.Escalate immediately, capture evidence before mutation, and keep support informed.
P2A product workflow, tenant, stream, bucket, dataset, or operator path is blocked for a subset of users.Run first checks, capture evidence, and escalate if no safe remediation exists.
P3Degraded, intermittent, or repeatable symptom with a known workaround or limited scope.Diagnose, remediate when safe, and file follow-up evidence.
P4Question, validation gap, maintenance preparation, or docs-only correction.Link the owning page and update evidence before stronger guidance is published.

Approval Route

Runbook reviewer routing
ReviewerReviewsStatus values
Product SMEProduct behavior, supported surface, known limits, and maturity.pending, approved, changes-requested, deferred
Operations reviewerFirst checks, remediation safety, rollback, and verification.pending, approved, changes-requested, deferred
Support reviewerEscalation triggers, customer handoff, support bundle contents, and exclusions.pending, approved, changes-requested, deferred
Docs QA reviewerCommands, expected output, evidence records, and repeatability.pending, approved, changes-requested, deferred
Security or privacy reviewerAuth, secrets, tenant data, logs, traces, support bundles, and redaction rules when present.pending, approved, changes-requested, deferred

Evidence Standard

Runbook evidence record types
Evidence typeUse forRequired fields
SourceRoute handler, configuration, runtime code, script, or README.Path, owner, date checked, status, and what the source does not prove.
TestSmoke test, diagnostic script, automation output, or fixture-backed check.Command, target environment, expected result, owner, date checked, and covered behavior.
Manual reviewReviewer-run procedure, incident review, or support acceptance.Reviewer, date, scope, decision, follow-up, and limits.
Support recordSupport bundle, redacted log excerpt, incident note, or customer handoff artifact.Redaction status, owner, sensitive fields excluded, date, and retention boundary.

Safety Rules

Start With Low-Risk Checks

Prefer read-only or low-risk checks unless the runbook explains why a mutating step is the safest first action.

Capture Before Mutation

Capture evidence before a step can overwrite logs, state, queues, segments, caches, WAL files, or diagnostics.

Redact Support Evidence

Keep secrets, bearer tokens, customer payloads, tenant identifiers, personal data, and proprietary traces out of support bundles unless a reviewed redaction path exists.

Evidence

Internal standards and templates define runbook structure, evidence expectations, and handoff requirements. Public pages should show the reviewed runbook pattern without publishing private source locations.

Safe Next Steps

  • Available: DOCS-050

    Use ObjectDB operator runbooks

    Use this template for health checks, platform capabilities, S3 gateway limits, local adapter caveats, and support evidence.

  • Available: DOCS-051

    Use Message Broker operator runbooks

    Use this template for health, metrics, audit, diagnostics, storage pressure, quota rejection, replication status, and support evidence.

  • Available: DOCS-052

    Use LogDB operator runbooks

    Use the same structure for OTLP admission, WAL, segment bundle, query, and BYOC preflight troubleshooting.

  • Available: DOCS-053

    Use Concordia validation runbooks

    Record what each Cache, Gateway, Storage, TSDB, and active-service validation proves and does not prove.

  • Available: DOCS-054

    Use observability and audit guide

    Route health, metrics, audit, logs, diagnostics, support bundle evidence, and privacy-safe handoff through cross-product guidance.

  • Available: DOCS-055

    Use troubleshooting index

    Route symptom-first checks to the right product runbook before drafting new remediation guidance.

  • Available: DOCS-056

    Use support bundle and privacy page

    Apply bundle contents, exclusions, redaction expectations, approval triggers, and privacy review boundaries.