ally

Enablement

Logging Standards

Baseline from Strata Production Logs

A structured logging standard derived from real production telemetry in the c3_logs bucket. These patterns define the baseline for all services shipping logs to Dynatrace.

Why Structured Logging?

What we get when everyone logs the same way

Readable = Less Toil

Why: a human-readable message means an on-call engineer understands what happened without decoding fields by hand — less toil, faster MTTR, every incident

PII Redaction

Why: a fixed schema is the only thing a whitelist can reliably redact against — unstructured logs hide PII in free text where no filter can find it

Standardized Alerting

Why: one alert rule (e.g. level == "error") works across every service instead of a custom rule per team's log format

Generic Alerting + Routing

Why: consistent fields (namespace, lineOfBusiness) let one generic alert route itself to the right owner automatically

The Baseline Schema

Required fields for every log record (JSON to stderr)

Required Fields

Field	Purpose
`requestId`	Unique per-request correlation ID (UUID)
`traceId`	Distributed trace ID (may match requestId)
`funcName`	Function/module emitting the log
`message`	Human-readable summary of the event
`level`	`info` \| `warn` \| `error`
`logType`	`CONSOLE` (app) or `SYSTEM`
`region`	AWS region (`us-east-1`, `us-east-2`)
`namespace`	Service namespace (e.g. `strata-customer-prod`)

Contextual Fields

Field	When
`conversationId`	IVR/interaction flows
`strataClient`	Caller identity (GENESYS, Salesforce)
`flowName`	Business flow being executed
`lineOfBusiness`	Auto, Deposits, etc.
`metadata`	Structured payload (durations, paths, etc.)
`operation`	GraphQL operation name

ally

Real Examples

Log Levels in Production

What each level looks like from actual Strata services

INFO

Successful Operation

GraphQL operation completed — includes duration for performance tracking

{
  "requestId": "9ba47df2-8a07-4a3e-963f-f00ec1f1b443",
  "traceId": "9ba47df2-8a07-4a3e-963f-f00ec1f1b443",
  "strataClient": "Salesforce",
  "flowName": "CustomerSummary",
  "lineOfBusiness": "Deposits",
  "namespace": "strata-customer-prod",
  "funcName": "useGraphLogger.onExecute",
  "message": "GraphQL operation completed successfully",
  "metadata": {
    "queryName": "GetCustomer",
    "operationName": "getCustomerBy",
    "duration": 190,
    "hasData": true,
    "service": "ACM",
    "servicePath": "/customer/master/v6/altids/{altId}"
  },
  "logType": "CONSOLE",
  "level": "info"
}

Key pattern: Duration in ms, downstream service name, and path logged on every successful call. This is what powers latency dashboards.

WARN

Execution Warning

Service returned a non-OK status — not a crash, but needs investigation

{
  "requestId": "b4712a69-f3fe-4cb7-8b78-3a1c4d2f2de1",
  "traceId": "b4712a69-f3fe-4cb7-8b78-3a1c4d2f2de1",
  "conversationId": "08d79cb7-aecb-4b52-a3e1-7c8df6d4dc0e",
  "strataClient": "GENESYS",
  "flowName": "Dep_Identify_Call_Link",
  "lineOfBusiness": "Deposits",
  "namespace": "strata-aivr-svc-prod",
  "operation": "GetCiigRoutingFlags",
  "message": "mesh: End query/mutation: GetCiigRoutingFlags execution",
  "endpoint": "https://secure.ally.com/acs/ivr/v3/interaction1",
  "httpRequestMethod": "POST",
  "httpStatusCode": 400,
  "logType": "CONSOLE",
  "level": "warn"
}

Key pattern: Includes endpoint, httpStatusCode, and operation so we can trace exactly which downstream call degraded.

ERROR

Error with Stack Trace

AWS Secrets Manager failure — full error context preserved

{
  "funcName": "fetchSecrets",
  "message": "Error fetching secrets from Secrets Manager",
  "metadata": {
    "httpStatusCode": 400,
    "errorMessage": "Can't find the specified secret value for staging label: AWSCURRENT"
  },
  "fullError": {
    "name": "ResourceNotFoundException",
    "message": "Can't find the specified secret...",
    "stack": "ResourceNotFoundException: ...
    at de_CommandError (client-secrets-manager/dist-cjs/index.js:1000:19)
    at async fetchSecrets2 (strata-library/dist/index.js)"
  },
  "logType": "CONSOLE",
  "level": "error"
}

Key pattern: Errors include fullError with name, message, and stack. The metadata carries the HTTP status for quick filtering.

Inbound Request Logging

What gets captured when a request enters the service

Logged on Entry

method — HTTP verb (POST, GET)
url / path — request target
userAgent — client identification
startTime — ISO-8601 timestamp
application_id — calling application
traceid — distributed tracing header
requestId — per-request UUID

What is NOT Logged

authorization header value — redacted
cookie values — redacted
access_token — redacted
session_id — redacted
Full IP addresses — redacted
PII in request body — redacted

PII Rule: Never log raw auth headers, tokens, cookies, or customer identifiers. Only the last 4 chars are acceptable for debugging.

Outbound Call Pattern

Every external call logs before and after with a shared correlation ID

Before Call

{
  "requestId": "9ba47...",
  "funcName": "useGraphLogger.onExecute",
  "message": "Starting operation",
  "metadata": {
    "stage": "execute",
    "queryName": "GetCustomer",
    "service": "ACM",
    "servicePath": "/customer/master/v6/..."
  }
}

After Call

{
  "requestId": "9ba47...",
  "funcName": "useGraphLogger.onExecute",
  "message": "GraphQL operation completed",
  "metadata": {
    "duration": 190,
    "hasData": true,
    "service": "ACM"
  },
  "httpStatusCode": 200
}

Pattern: Same requestId ties the before/after pair. Log sanitized URL + service name before; URL + duration + status after. This is mandatory for all outbound calls.

Anti-Patterns to Avoid

Common mistakes found in production logs

Unstructured Messages

console.log("Error fetching user: " + err)
console.log("Request took " + ms + "ms")

Cannot be filtered, aggregated, or alerted on in Dynatrace. Forces regex parsing.

Logging Sensitive Data

logger.info("Auth header: " + req.headers.authorization)
logger.info("User SSN: " + customer.ssn)

PII/secrets in logs violate compliance. Auth header values must always be redacted.

Missing Correlation IDs

logger.error("Database connection failed")
// No requestId, no traceId
// Impossible to link to triggering request

Without IDs, errors can't be correlated to the request path that caused them.

Swallowed Errors

try {
  await callService()
} catch (e) {
  // silently continue
}

Silent failures are invisible to monitoring. Every caught error must be logged with context.

Real Example

Whitelist, Not Blacklist

From redact_pii.py — c3-genesys-metric

# only these paths survive redaction
whitelisted_fields = [
  {
    "detail-type": "v2.detail.events.conversation.{id}.attributes",
    "fields": [
      ["attributes", "Line_of_Business"],
      ["attributes", "Authenticated"],
      ["conversationId"],
    ],
  },
]

# build clean_event from scratch — nothing
# is copied unless its path is listed above
clean_event = {"detail": {"eventBody": {}}}
for item in whitelisted_fields:
  if item["detail-type"] == event["detail-type"]:
    for field_list in item["fields"]:
      # copy only the matched subfield path
      ...

Why Whitelist Wins

Blacklist fails open: a new/renamed field ships raw until someone notices and patches the blacklist — the leak already happened
Whitelist fails closed: any field not explicitly listed is silently dropped, never logged, by construction
JSON's schema flexibility is a strength for partner interoperability — they can add, rename, or nest fields anytime
That same flexibility is a liability for PII — nothing stops a partner from renaming ssn to taxId or nesting it a level deeper
Only a whitelist survives a partner's payload changing out from under you

As we onboard more partners: assume payloads will change. Whitelisting by field path degrades safely instead of leaking silently.

Cheap win before the whitelist: regex-scrub every outbound log line for SSN, credit card, and phone number patterns. Easy to write, tiny runtime cost — catches PII buried in free text that no field whitelist would ever see.

The Standard

Rules derived from production baseline

Output Format

Single JSON logger writing to stderr
Fields: time, level, msg minimum
No console.log, no log.Printf, no inline loggers
One log record = one line of JSON

Correlation

requestId on every log in the request path
traceId propagated from incoming headers
conversationId for multi-step flows

Error Handling

Try/catch ALL external calls (AWS, DB, APIs, file I/O)
Log every error with funcName, message, metadata
Include fullError (name + message + stack) on errors
Never silently swallow exceptions

PII / Secrets

Whitelist by field path, not by blacklisting bad ones — see previous slide
Never log auth header values (last 4 chars max)
Sanitize customer identifiers before logging
No tokens, passwords, or credentials in any log

Outbound Call Checklist

Required logging for every external service call

Timing	Field	Example
Before	`service`	`"ACM"`, `"CIIG"`
Before	`servicePath`	`"/customer/master/v6/altids/{altId}"`
Before	`httpRequestMethod`	`"POST"`
Before	`operation`	`"GetCustomer"`
After	`duration`	`190` (ms)
After	`httpStatusCode`	`200`, `400`, `500`
After	`hasData`	`true` / `false`
On Error	`fullError`	Object with name, message, stack

Tie with common GUID: The requestId must be the same on both the "before" and "after" log lines so they can be correlated in a single DQL query.

Verifying with DQL

How to check your logs are landing correctly in Dynatrace

Check your service logs

fetch logs, from:now()-1h
| filter dt.system.bucket == "c3_logs"
| filter contains(content, "your-namespace")
| fields timestamp, content, loglevel
| limit 10

Check error rate

fetch logs, from:now()-1h
| filter dt.system.bucket == "c3_logs"
| filter contains(content, "your-namespace")
| summarize
    total = count(),
    errors = countIf(loglevel == "ERROR")
| fieldsAdd error_pct = errors * 100.0 / total

Check latency (p95)

fetch logs, from:now()-1h
| filter dt.system.bucket == "c3_logs"
| filter contains(content, "duration")
| filter contains(content, "your-namespace")
| parse content, "JSON:payload"
| fieldsAdd dur = payload[metadata][duration]
| summarize p95 = percentile(dur, 95)

Always scope to bucket first. An unscoped fetch logs scans 48–186 GB. Adding dt.system.bucket == "c3_logs" reduces to ~3–4 GB.

ally

Quick Reference

Format

JSON to stderr, one line per record
Required: requestId, traceId, funcName, message, level
No unstructured console.log

Security

Whitelist logged fields
Never log tokens/auth/PII
Redact all sensitive headers

Outbound Calls

Log before + after with same requestId
Capture duration_ms and status_code
fullError on any failure

Baseline source: All examples pulled from live c3_logs bucket — Strata production services (strata-customer-prod, strata-aivr-svc-prod, strata-gateway-prod). These are the patterns to replicate.