Start With Evidence
Capture the first failing status, reason code, route, script, and runtime context before remediation changes the signal.
Operations index
Verification RequiredTroubleshooting index.
A central, symptom-first index for ObjectDB, Message Broker, LogDB, Concordia, and cross-product evidence handoff. Use it to pick the right runbook before changing local state or broadening a support claim.
Index Snapshot
- Entries
- 29
- Categories
- 5
- Products
- 5
- Review state
- Routed
Triage Flow
- Match SymptomUse the category and product before running a command or changing local state.
- Capture First FailureRecord the first status, reason code, script, route, and evidence owner.
- Open Product RunbookUse the linked runbook for exact commands, expected output, cleanup, and limits.
- Redact Before HandoffMinimize credentials, payloads, tenant data, object keys, traces, and topology details.
Startup And Health
| Product | Symptom | First check | Likely cause | Next action | Evidence | Runbook |
|---|---|---|---|---|---|---|
| ObjectDB | Health check cannot connect. | Compare OBJECTDB_BIND_ADDRESS with OBJECTDB_ENDPOINT and check the service terminal. | Service is not running, still compiling, or bound to another address. | Start the service on an unprivileged loopback port and retry /health. | DOCS-050, ODB-EVID-007 | ObjectDB operator runbooks |
| ObjectDB | Startup platform probe fails. | Read service stderr and verify OBJECTDB_DATA_DIR. | Data directory cannot be created, written, flushed, renamed, or inspected. | Choose a writable local path and capture stderr plus capability context before retrying. | DOCS-050, DOCS-030, ODB-EVID-008 | ObjectDB operator runbooks |
| Message Broker | Broker health reports storage not ready. | Capture GET /health, selected node id, data directory, and storage status. | Local storage path, startup flags, or recovery state needs review. | Preserve health output and use the Message Broker startup runbook before changing local state. | DOCS-051, SMB-EVID-002, SMB-EVID-008 | Message Broker operator runbooks |
| LogDB | Health route cannot be reached. | Confirm LOGDB_BIND_ADDRESS, service terminal state, and /health URL. | Service is not running, still compiling, exited early, or bound to another address. | Restart on a loopback bind and capture the first startup error before retrying. | DOCS-052, LDB-EVID-001, LDB-EVID-009 | LogDB operator runbooks |
| Concordia | Cache or TSDB health cannot connect. | Check the service log, bind address, and product-specific /health URL. | Cargo is still compiling, service exited, or endpoint values drifted. | Wait for startup, correct the endpoint, or preserve the early exit for Concordia review. | DOCS-053, CONC-EVID-003, CONC-EVID-009 | Concordia validation runbooks |
| Concordia | Gateway health stays blocked. | Capture alpha_write_blockers, startup args, and health response. | One alpha-ready flag or fixture input is missing. | Restart with the documented alpha-ready command and keep accepted writes diagnostic-only. | DOCS-053, CONC-EVID-010, CONC-EVID-011 | Concordia validation runbooks |
Auth And Admission
| Product | Symptom | First check | Likely cause | Next action | Evidence | Runbook |
|---|---|---|---|---|---|---|
| ObjectDB | S3 route returns AccessDenied. | Capture request path, headers used, operation header, and error XML. | Local request is missing expected credential shape or the operation is outside the active profile. | Retry only with scoped local credentials and link the S3 Core compatibility or limits row. | DOCS-050, DOCS-038, DOCS-042 | ObjectDB operator runbooks |
| Message Broker | Protected route returns 401 or 403. | Confirm security.auth_mode in /health and the route/auth compatibility row. | API key, mTLS source, or trusted-header setup is missing or mismatched. | Retry with valid local credentials or the configured trusted upstream path. | DOCS-051, DOCS-040 | Message Broker operator runbooks |
| Message Broker | Publish or consumer group returns quota_exceeded. | Capture status 429, error body, tenant or stream scope, and quota config. | Tenant publish or consumer-group quota rejected the request. | Reduce request shape or adjust reviewed local quota config; do not classify expected rejection as an outage. | DOCS-051, SMB-EVID-008, SMB-EVID-010 | Message Broker operator runbooks |
| LogDB | Ingest or query returns missing_tenant. | Capture headers and route without payload contents. | x-logdb-tenant was omitted or empty. | Send the expected tenant header for local checks and keep broader tenant wording in security review. | DOCS-052, LDB-EVID-002, LDB-EVID-003 | LogDB operator runbooks |
| LogDB | Ingest returns invalid_credentials or tenant_mismatch. | Compare bearer token, tenant header, and OTLP resource tenant claim. | Token does not match local policy or resource tenant differs from authenticated tenant. | Align local credentials and tenant attributes; preserve the rejected body shape for security review. | DOCS-052, LDB-EVID-002, LDB-EVID-003 | LogDB operator runbooks |
| Concordia | Protected route stays rejected. | Capture authorization header shape and missing_authorization or invalid_bearer_token body. | Local bearer token is missing or does not match the documented token for that binary. | Use the product-specific local token and keep the result scoped to local admission. | DOCS-053, CONC-EVID-004, CONC-EVID-009 | Concordia validation runbooks |
Data Visibility And State
| Product | Symptom | First check | Likely cause | Next action | Evidence | Runbook |
|---|---|---|---|---|---|---|
| ObjectDB | NoSuchBucket or NoSuchKey. | Confirm bucket/key values and list current objects. | Bucket or key was not created in the current data directory. | Create the bucket/object again or point at the intended data directory. | DOCS-050, ODB-EVID-006, ODB-EVID-007 | ObjectDB operator runbooks |
| ObjectDB | BucketNotEmpty during cleanup. | Run ListObjectsV2 with the smoke prefix. | Visible current objects remain in the bucket. | Delete current smoke objects before deleting the bucket; avoid shared data cleanup. | DOCS-050, ODB-EVID-006 | ObjectDB operator runbooks |
| Message Broker | Replay returns no records. | Check stream name, start offset, max records, consumer group, and publish result. | Stream was not created, endpoint points at another node, or offset window is empty. | Recreate the scoped smoke stream or query the exact offset produced by publish. | DOCS-051, DOCS-031 | Message Broker operator runbooks |
| Message Broker | Queue delivery state cannot be stored. | Capture route, delivery id, subscription id, and error body. | Queue persistence path or storage admission rejected the mutation. | Stop mutating the delivery and route evidence to support review. | DOCS-051, SMB-EVID-008 | Message Broker operator runbooks |
| LogDB | Query returns zero matches. | Re-run ingest, re-extract record ID, and check dataset. | Dataset, record ID, trace ID, time filter, or data directory differs. | Query the exact dataset and identifier produced by the smoke ingest. | DOCS-052, LDB-EVID-004, LDB-EVID-008 | LogDB operator runbooks |
| Concordia | TSDB recent query returns no series. | Compare tenant, dataset, series name, tags, and append payload. | Query identity fields do not match appended samples. | Reuse the exact identity fields from the tutorial or focused script. | DOCS-053, CONC-EVID-009 | Concordia validation runbooks |
Limits And Unsupported Behavior
| Product | Symptom | First check | Likely cause | Next action | Evidence | Runbook |
|---|---|---|---|---|---|---|
| ObjectDB | Unsupported S3 request returns NotImplemented. | Capture status, x-objectdb-s3-operation, error code, path, query string, and request ID. | Request is outside the active s3-core-v1 profile. | Use a supported S3 Core route or record an unsupported-behavior follow-up. | DOCS-050, DOCS-038, DOCS-042 | ObjectDB operator runbooks |
| Message Broker | /metrics returns 404 metrics_disabled. | Check startup flags for --metrics-exposure http. | Broker was not started with HTTP metrics exposure. | Restart local broker with HTTP metrics exposure only if metrics are needed for the check. | DOCS-051, DOCS-031 | Message Broker operator runbooks |
| Message Broker | Replication worker route returns replication_not_configured. | Capture binding source, route body, stream, source site, target site, and timestamp. | Replication bindings are not configured for the local run. | Treat as configuration state; do not present it as a failed replication outcome. | DOCS-051, DOCS-040 | Message Broker operator runbooks |
| LogDB | Metrics or profiles are sent to LogDB. | Check route and content against the compatibility matrix. | Metrics and profiles are outside the documented LogDB route surface. | Use logs/traces routes or track metrics/profile need as future compatibility work. | DOCS-052, DOCS-039, DOCS-042 | LogDB operator runbooks |
| LogDB | S3 query read fails. | Confirm LOGDB_QUERY_STORE=s3 and redacted LOGDB_QUERY_S3_* values. | S3 query-read config, region, bucket, prefix, or credentials failed. | Treat as query-read configuration evidence only; do not infer ingest publication behavior. | DOCS-052, LDB-EVID-009, LDB-EVID-010 | LogDB operator runbooks |
| Concordia | Gateway accepted output is overclaimed. | Check committed:false, visible:false, and diagnostic_only:true. | Diagnostic flags were omitted from evidence or summary wording. | Keep the evidence diagnostic; do not claim quorum, visible reads, or cross-node routing. | DOCS-053, CONC-EVID-010, CONC-EVID-011 | Concordia validation runbooks |
Evidence And Handoff
| Product | Symptom | First check | Likely cause | Next action | Evidence | Runbook |
|---|---|---|---|---|---|---|
| ObjectDB | Support evidence includes sensitive object data. | Inspect S3 response headers, XML errors, object key, payload sample, and data-directory context. | Evidence capture included payload contents or sensitive object names. | Minimize to status, operation header, request ID, and known-limits link before sharing. | DOCS-050, DOCS-054 | ObjectDB operator runbooks |
| Message Broker | Diagnostics bundle includes payloads. | Confirm mq diagnostics collect was run without --include-payloads. | Payload inclusion was requested or a bundle shape drifted. | Regenerate a payload-free bundle and route bundle contents through support and privacy review. | DOCS-051, DOCS-054 | Message Broker operator runbooks |
| LogDB | Support package contains raw telemetry. | Inspect query, OTLP, segment, and WAL evidence before sharing. | Raw payloads, trace/span IDs, dataset names, or record IDs were captured without minimization. | Replace with shape-only query evidence, counts, indexes, and redacted IDs when needed. | DOCS-052, DOCS-054 | LogDB operator runbooks |
| Concordia | Protocol or fixture evidence is treated as live behavior. | Check fixture name, message family, decoded field summary, parser result, and compatibility row. | Registry allocation or fixture bytes were confused with runtime support. | Keep future frame IDs labeled future until parser, serializer, listener, and reviewer evidence exist. | DOCS-053, DOCS-041 | Concordia validation runbooks |
| Cross-product | Claim-sensitive wording appears in a troubleshooting note. | Run docs claim lint and inspect nearby limits or evidence. | Troubleshooting text strengthened support, availability, security, performance, scale, or customer-commitment wording. | Rewrite to observed behavior and route through DOCS-007 and DOCS-011 before publishing. | DOCS-007, DOCS-011, DOCS-054 | Observability and audit guide |
Product Runbook Links
| Product | Troubleshooting source | Runbook route | Limits route |
|---|---|---|---|
| ObjectDB | docs/operations/DOCS-050-objectdb-operator-runbooks.md | ObjectDB operator runbooks | ObjectDB limits |
| Message Broker | docs/operations/DOCS-051-message-broker-operator-runbooks.md | Message Broker operator runbooks | Message Broker limits |
| LogDB | docs/operations/DOCS-052-logdb-operator-runbooks.md | LogDB operator runbooks | LogDB limits |
| Concordia | docs/operations/DOCS-053-concordia-validation-runbooks.md | Concordia validation runbooks | Concordia limits |
| Cross-product | docs/operations/DOCS-054-observability-audit-guide.md | Observability and audit guide | Errors and limits catalog |
Troubleshooting Guardrails
Unsupported Is Not Outage
Known unsupported behavior stays a limits entry unless a documented supported route or workflow is also failing.
Use Product Owners
The central index routes symptoms; product runbooks own exact commands, cleanup, escalation, and reviewer decisions.
Evidence
| Index area | Evidence | Status |
|---|---|---|
| Startup, health, auth, state, limits, and evidence rows | DOCS-050 through DOCS-054 and DOCS-036 | pending |
| ObjectDB troubleshooting entries | DOCS-050, DOCS-038, and DOCS-042 | pending |
| Message Broker troubleshooting entries | DOCS-051, DOCS-031, DOCS-040, and DOCS-042 | pending |
| LogDB troubleshooting entries | DOCS-052, DOCS-033, DOCS-039, and DOCS-042 | pending |
| Concordia troubleshooting entries | DOCS-053, DOCS-035, DOCS-041, and DOCS-042 | pending |
| Sensitive troubleshooting and support handoff guidance | DOCS-007, DOCS-011, and DOCS-054 | pending |
Limits
- This index points to first checks and next actions. Product runbooks own exact commands, expected output, cleanup, remediation, escalation, and evidence requirements.
- Entries should not be used to claim support coverage, incident response timing, data retention, security posture, privacy approval, availability, performance, scale, production deployment readiness, or customer commitments.
- Unsupported behavior should stay separate from incidents unless a documented supported path is also failing.
- Support evidence must be minimized and redacted before sharing outside the reviewer group.
Safe Next Steps
Available: DOCS-054
Use observability and audit guideClassify health, metrics, audit, logs, diagnostics, and support-bundle evidence before escalation.
Available: DOCS-036
Use shared errors and limits catalogCheck cross-product HTTP status, retry, pressure, unsupported, and local-preview guidance.
Available: DOCS-056
Prepare support bundle and privacy pageDefine support bundle contents, exclusions, redaction expectations, and privacy review boundaries.
