Operations

LiveVerification Required

LogDB operator runbooks.

LogDB operator runbooks for OTLP admission failures, WAL and data-directory issues, segment bundle checks, query troubleshooting, S3 query-read boundaries, BYOC preflight, and support evidence handoff.

Current availability

Private Beta. Public beta is scheduled for November 2, 2026.

Runbook Snapshot

Runbooks: 8
First checks: 11
Triage entries: 22
Review state: Pending

Runbook Summary

LogDB operator runbook coverage
Runbook	Severity	Scope	First check	Evidence
Startup and health smoke	P2 to P4	Local process, LOGDB_BIND_ADDRESS, /health	Confirm the listener, service terminal, bind address, and "Healthy" response.	LDB-EVID-001, LDB-EVID-009
OTLP admission failure triage	P2 to P4	Tenant headers, bearer token, tenant claims, dataset policy	Capture status, error kind, headers used, resource tenant claim, dataset, and retryable flag.	LDB-EVID-002, LDB-EVID-003
WAL and data-directory triage	P2 to P4	LOGDB_DATA_DIR, local WAL, replicated-local WAL settings	Confirm the data directory is disposable or approved and the WAL mode matches runtime intent.	LDB-EVID-005, LDB-EVID-009
Segment bundle check	P2 to P4	Local segment publication and query-read source	Verify query can read expected retained records before inspecting filesystem state.	LDB-EVID-004, LDB-EVID-005
Query troubleshooting	P2 to P4	POST /v1/query, filters, page limits, deadlines, datasets	Capture query body shape, page limit, filter, status, matched records, scanned bundles, indexes.	LDB-EVID-004, LDB-EVID-008
S3 query-read distinction	P2 to P4	LOGDB_QUERY_STORE=s3 and LOGDB_QUERY_S3_*	Confirm S3 is selected only for query reads and does not imply ingest publication to S3.	LDB-EVID-009, LDB-EVID-010
BYOC handoff preflight	P2 to P4	Terraform output, Helm values, retained resources	Run offline preflight against terraform output JSON before treating handoff values as usable.	LDB-EVID-009, LDB-EVID-010
Support evidence handoff	P2 to P4	Redacted route output, config, query results, logs, preflight	Capture evidence before mutation and redact sensitive data before sharing.	DOCS-049, DOCS-052

Local Startup

Start LogDB in one terminal, then run the first checks in a second terminal.

cd /path/to/logdb/LogDB
export LOGDB_BIND_ADDRESS=127.0.0.1:4318
export LOGDB_DATA_DIR="${TMPDIR:-/tmp}/logdb-docs-052"
mkdir -p "$LOGDB_DATA_DIR"
cargo run

First Checks

Configure check terminal

First check

export LOGDB_ENDPOINT=http://127.0.0.1:4318
export LOGDB_TENANT=local
export LOGDB_TOKEN=local-dev-token
export LOGDB_DATASET=checkout
export LOGDB_TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736

Expected signal

Endpoint, tenant, token, dataset, and trace ID are set.

Capture: Endpoint value, tenant, dataset, and local policy note.

Health

First check

curl -i "$LOGDB_ENDPOINT/health"

Expected signal

200 OK with JSON string "Healthy".

Capture: Status, bind address, service terminal state, and timestamp.

Ingest smoke log

First check

LOG_RESPONSE="$(curl -s -X POST "$LOGDB_ENDPOINT/v1/otlp/logs" \
  -H 'content-type: application/json' \
  -H "x-logdb-tenant: $LOGDB_TENANT" \
  -H "authorization: Bearer $LOGDB_TOKEN" \
  --data '{"resourceLogs":[{"resource":{"attributes":[{"key":"tenant_id","value":{"stringValue":"local"}},{"key":"service.name","value":{"stringValue":"checkout"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"1783947600000000000","body":{"stringValue":"operator smoke"}}]}]}]}')"

Expected signal

Accepted JSON with accepted_records: 1 and at least one logdb record ID.

Capture: Status body, accepted record count, tenant, dataset, and redacted record ID if sensitive.

Extract record ID

First check

LOG_RECORD_ID="$(printf '%s' "$LOG_RESPONSE" | sed -nE 's/.*"record_ids":\["([^"]+)".*/\1/p')"
printf '%s\n' "$LOG_RECORD_ID"

Expected signal

A deterministic logdb record ID is available for query checks.

Capture: Record ID shape and extraction command.

Query by record ID

First check

curl -i -X POST "$LOGDB_ENDPOINT/v1/query" \
  -H 'content-type: application/json' \
  -H "x-logdb-tenant: $LOGDB_TENANT" \
  -H "authorization: Bearer $LOGDB_TOKEN" \
  --data "{\"dataset\":\"$LOGDB_DATASET\",\"filter\":{\"record_id\":\"$LOG_RECORD_ID\"},\"page\":{\"limit\":10}}"

Expected signal

200 OK with matched_records: 1 and used_indexes containing record_id.

Capture: Query shape, status, matched count, used indexes, and scanned bundle count.

Tenant rejection

First check

curl -i -X POST "$LOGDB_ENDPOINT/v1/query" \
  -H 'content-type: application/json' \
  --data '{"dataset":"checkout"}'

Expected signal

Rejected JSON with missing_tenant.

Capture: Status, error kind, route, and omitted header note.

Page limit check

First check

curl -i -X POST "$LOGDB_ENDPOINT/v1/query" \
  -H 'content-type: application/json' \
  -H "x-logdb-tenant: $LOGDB_TENANT" \
  -H "authorization: Bearer $LOGDB_TOKEN" \
  --data "{\"dataset\":\"$LOGDB_DATASET\",\"page\":{\"limit\":501}}"

Expected signal

Rejected query response because the maximum query page limit is 500.

Capture: Status, limit value, error body, and reference link.

Data directory

First check

printf '%s\n' "$LOGDB_DATA_DIR"

Expected signal

Path is disposable for local smoke checks or explicitly approved for the environment.

Capture: Configured value and operator note.

Local query source

First check

find "$LOGDB_DATA_DIR" -maxdepth 3 -type f \( -name 'manifest.json' -o -name '*.parquet' \)

Expected signal

Segment bundle files appear only after local publication has flushed retained records.

Capture: File names, parent directory, timestamp, and note that payload contents stay out.

S3 query config

First check

printf '%s\n' "${LOGDB_QUERY_STORE:-local}" "${LOGDB_QUERY_S3_BUCKET:-}" "${LOGDB_QUERY_S3_PREFIX:-}" "${LOGDB_QUERY_S3_REGION:-}"

Expected signal

Local is default; S3 values affect query reads only when LOGDB_QUERY_STORE=s3.

Capture: Selected query store, redacted bucket/prefix/region, and boundary note.

BYOC preflight

First check

cd /path/to/logdb
cargo test control::byoc_preflight

Expected signal

Preflight tests pass for Terraform output, Helm values, retained resources, and uninstall notes.

Capture: Test command, result, source revision, and redacted handoff sample if available.

OTLP Admission Failure Triage

LogDB OTLP admission triage
Symptom	Likely cause	First check	Remediation
missing_tenant	x-logdb-tenant was omitted or empty.	Capture headers and route.	Send the expected tenant header for local checks; keep broader tenant wording in security review.
missing_credentials	Bearer token was omitted, malformed, or not forwarded.	Confirm authorization: Bearer local-dev-token for local policy checks.	Retry with local credentials or the configured environment credential path.
unknown_tenant	Tenant is not present in the active tenant policy.	Compare header tenant with policy owner notes.	Use a configured tenant or route to product/security review before documenting a new tenant path.
invalid_credentials	Token does not match the active local policy.	Confirm token source and whether proxy/header rewriting changed casing.	Rotate to the approved local token or capture the failed credential path for security review.
tenant_mismatch	OTLP resource tenant claim differs from the authenticated tenant.	Compare tenant_id or service.logdb.tenant_id with x-logdb-tenant.	Align resource attributes and tenant header; preserve the rejected body for review.
dataset_not_authorized	Credential is not allowed for the requested dataset.	Capture dataset, tenant, and route without payload contents.	Use an allowed dataset or route the policy change through security and product reviewers.
record_quota_exceeded	Tenant record window is exhausted.	Capture accepted record count, route, tenant, retryable flag, and config.	Reduce batch size or adjust reviewed local policy; do not present quota rejection as an outage.
byte_quota_exceeded	Tenant byte window or gateway byte budget is exhausted.	Capture byte size, batch shape, gateway config, and retryable flag.	Retry with smaller batches or reviewed budget changes.
concurrency_exceeded	Tenant or gateway concurrent request budget is saturated.	Capture in-flight config and retryable response.	Back off client concurrency and retain evidence for capacity review.
Unsupported media or encoding	Content type or content-encoding is outside the accepted set.	Check content-type, content-encoding, and route family.	Use JSON compatibility routes for JSON or native protobuf routes for application/x-protobuf.
Malformed or empty OTLP batch	Payload shape is invalid or contains no records.	Save a redacted shape-only sample and response error kind.	Fix request shape; avoid sharing raw payloads in the support package.

WAL And Data-Directory Triage

LogDB WAL and data-directory triage
Symptom	Likely cause	First check	Remediation
Service cannot create data dir	LOGDB_DATA_DIR points at an unwritable or invalid location.	Check path, parent permissions, and startup stderr.	Choose a writable local path and capture the failed path before retrying.
WAL replay or repair fails	Local WAL files cannot be read, decoded, repaired, or opened.	Capture startup stderr and the configured data directory.	Preserve the data directory for review; do not delete shared state before support accepts evidence.
Durable ingest returns 503	Retryable publication or WAL condition failed.	Capture route, retryable flag, tenant, dataset, and service log lines.	Retry after preserving evidence; escalate if the same local WAL condition repeats.
Replicated-local startup fails	Replica paths or write quorum cannot satisfy validation.	Check LOGDB_WAL_DURABILITY_MODE, replica paths, and write quorum.	Correct replica path/quorum settings; keep replicated-local wording verification-required.
Cleanup target looks unsafe	Data directory points at root, home, or a shared product directory.	Compare cleanup command with LOGDB_DATA_DIR.	Stop cleanup and get owner approval before deletion.

Segment Bundle And Query Triage

LogDB segment and query triage
Symptom	Likely cause	First check	Remediation
Query returns zero matches	Dataset, record ID, trace ID, time filter, or data directory differs.	Re-run ingest, re-extract record ID, and check dataset.	Query the exact dataset and identifier produced by the smoke ingest.
Query page is rejected	page.limit exceeds 500 or cursor shape is invalid.	Capture query body and error body.	Use a page limit at or below 500 and pass the returned cursor exactly.
Query deadline expires	deadline_ms is too small for the requested scan.	Capture deadline, scanned bundles, and filter shape.	Increase the deadline for local diagnostics or use exact ID filters where possible.
Segment scan or index error	Segment bundle, manifest, Parquet file, or sidecar index cannot read.	Capture scanned_bundles, used_indexes, error body, and service logs.	Preserve local segment files and avoid destructive cleanup until review.
S3 query returns read errors	S3 query-read config, AWS region, bucket, prefix, or credentials fail.	Confirm LOGDB_QUERY_STORE=s3 and redacted LOGDB_QUERY_S3_* values.	Treat as query-read configuration only; do not infer ingest publication behavior.
Metrics or profiles are sent	Signal family is outside the documented LogDB route surface.	Check route and content against the compatibility matrix.	Use logs/traces routes or track metrics/profile need as future compatibility work.

S3 Query-Read Boundary

LOGDB_QUERY_STORE=s3 changes only the query read path for retained segment bundles. Endpoint JSON, OTLP, and JSON compatibility ingest continue through local data and WAL unless a separate reviewed publication reference says otherwise.

LogDB S3 query-read checks
Check	Expected signal	Boundary
Query store selected	LOGDB_QUERY_STORE=s3 plus bucket, optional prefix, and optional region.	S3 is a query-read store selection, not ingest publication.
Local default	Unset or non-s3 LOGDB_QUERY_STORE uses local segment bundles.	Local query success does not prove S3 read behavior.
Region override	Non-empty LOGDB_QUERY_S3_REGION when an override is supplied.	Region config does not imply BYOC readiness.
Query error	Runtime query read or index failures return rejected query JSON.	Capture query-read evidence separately from WAL ingest evidence.
Publication wording	No active LogDB docs claim live ingest publication to S3 from query-read config.	Keep customer-owned bucket writes future or verification-required until evidence is accepted.

BYOC Handoff Preflight

Use BYOC preflight only for offline handoff validation. It does not prove a deployed customer environment, support posture, or customer-owned S3 publication path.

LogDB BYOC preflight checks
Check	Command or source	Expected signal	Boundary
Terraform output	terraform output -json from deploy/aws/byoc/terraform	Output contains telemetry bucket, prefix, KMS key ARN, role ARN, and values.	Redact account IDs, bucket names, KMS ARNs, IAM roles, and tags before sharing.
Offline preflight tests	cargo test control::byoc_preflight	Tests pass for normalized handoff values and invalid-output rejection.	Test evidence is offline validation, not deployment readiness.
Layout validation	./scripts/validate-byoc-layout.sh	BYOC Terraform, Helm, and docs layout checks pass where available.	Layout validation does not prove a cluster or image exists.
Helm values	helm_values from the preflight result	Values line up with runtime bind, gRPC, data dir, AWS, bucket, prefix, KMS.	Values are handoff metadata until install automation and environment evidence exist.
Retained resources	retained_customer_data_resources and uninstall_notes	Retained-resource wording and prevent-destroy notes are present.	Do not provide destructive cleanup instructions for customer data.

Support Evidence

LogDB support evidence package
Include	Source	Redact or exclude
Health response and bind settings	GET /health, LOGDB_BIND_ADDRESS	Public hostnames, private ports when sensitive, and unrelated headers.
OTLP admission response	/v1/logs, /v1/traces, /v1/otlp/logs, /v1/otlp/traces	Bearer tokens, tenant identifiers, dataset names, raw payloads, trace/span IDs when customer-identifying.
Query request and response shape	POST /v1/query	Raw payloads, customer identifiers, sensitive record IDs, personal data, and proprietary trace contents.
Data-directory and WAL context	LOGDB_DATA_DIR, startup stderr, WAL mode env vars	Directory listings that expose unrelated customer data, secrets, and shared-state cleanup commands.
Segment bundle metadata	Manifest names, file counts, scanned bundle count, indexes	Parquet payload contents, raw telemetry, private object keys, and unreviewed retention claims.
S3 query-read configuration	LOGDB_QUERY_STORE, LOGDB_QUERY_S3_*	AWS credentials, account IDs, bucket names, prefixes, object keys, and customer topology details.
BYOC handoff output	Terraform output, Helm values, preflight result	AWS account IDs, KMS ARNs, IAM role ARNs, bucket names, namespaces, tags, and customer-specific notes.
Compatibility, limits, evidence links	DOCS-039, DOCS-042, LDB-EVID-009, LDB-EVID-010	Roadmap promises, S3 publication claims, deployment readiness wording, and unreviewed commitments.

Safety Rules

Keep S3 Boundaries Visible

S3 query-read diagnostics are not evidence for S3 publication, customer-owned bucket writes, or BYOC readiness.

Protect Telemetry Payloads

Capture request shape, statuses, IDs, counts, indexes, and redacted metadata; do not include raw telemetry in routine diagnostics.

Preserve Shared State

Do not delete WAL files, segment bundles, shared data directories, S3 objects, or retained BYOC resources from this runbook.

Evidence

DOCS-052 evidence map
Runbook area	Evidence	Status
OTLP admission and query routes	DOCS-033, LDB-EVID-001, LDB-EVID-002, LDB-EVID-003, LDB-EVID-008	pending
WAL, data directory, S3 query-read, BYOC	DOCS-034, LDB-EVID-005, LDB-EVID-009	pending
Compatibility and known limits	DOCS-039, DOCS-042, LDB-EVID-010, LDB-EVID-011	pending
LogDB operator runbook artifact	DOCS-052 and /products/logdb/operations/operator-runbooks/	pending

Internal review records track this LogDB runbook page and its evidence handoff requirements.

Safe Next Steps

Available: DOCS-033
Read OTLP and query reference
Use route shapes, tenant headers, error mapping, query filters, page limits, and query-read boundaries.
Available: DOCS-034
Read configuration and BYOC reference
Check bind address, data directory, optional gRPC, WAL modes, S3 query-read settings, Terraform output, Helm values, and preflight.
Available: DOCS-039
Compare OTLP and query scope
Separate live ingest and local query from S3 query-read, BYOC, replay, unsupported signals, and future publication work.
Available
Review LogDB evidence
Use LDB evidence records before strengthening runbook, support, security, S3, BYOC, or readiness wording.