UNS Console Observability¶

This document describes the observability instrumentation for UNS Console endpoints, including request metrics, error tracking, structured logging, and dashboard/query recommendations.

Overview¶

UI endpoints emit observability data through: - Request metrics: Latency, request counts, error rates per endpoint - Error metrics: Structured error tracking with codes and messages - Structured logs: All logs include request IDs for correlation - Request ID propagation: Unique IDs for request tracing

Request ID Generation¶

Every UI request receives a unique request ID for correlation across logs and metrics.

Generation¶

Format: req-{12_hex_chars} (e.g., req-a1b2c3d4e5f6)
Generated by ObservabilityMiddleware if not provided
Clients can provide their own via X-Request-ID header

Propagation¶

Request ID is set in context at request start
Available to all endpoint handlers via get_request_id()
Included in all structured log messages
Returned in X-Request-ID response header
Included in error response bodies

Example¶

# Request with auto-generated ID
curl -i http://localhost:8000/ui/deliveries
# Response header: X-Request-ID: req-a1b2c3d4e5f6

# Request with client-provided ID
curl -i -H "X-Request-ID: my-trace-123" http://localhost:8000/ui/deliveries
# Response header: X-Request-ID: my-trace-123

Metrics¶

Metric Types¶

Metric	Type	Description
`ui_request_total`	Counter	Total requests per endpoint/method/status
`ui_request_latency_ms`	Histogram	Request latency in milliseconds
`ui_request_error`	Counter	Error occurrences with error_code labels
`ui_request_in_progress`	Gauge	Currently active requests (future)

Labels¶

All metrics include: - endpoint: Normalized endpoint path (e.g., /ui/deliveries/{id}) - method: HTTP method (e.g., GET) - status_code: HTTP response status code

Error metrics additionally include: - error_code: Machine-readable error code (e.g., not_found) - error_message: Truncated human-readable message

Endpoint Normalization¶

Path parameters are normalized for consistent metrics: - /ui/deliveries/123e4567-e89b-... → /ui/deliveries/{id}

This prevents high cardinality from unique IDs.

Error Tracking¶

Error Response Format¶

All UI error responses follow this structure:

{
  "error": {
    "code": "not_found",
    "message": "Notification not found: 123e4567-e89b-...",
    "details": null
  },
  "request_id": "req-a1b2c3d4e5f6"
}

Error Codes¶

Code	HTTP Status	Description
`not_found`	404	Requested resource does not exist
`validation_error`	422	Invalid request parameters
`internal_error`	500	Unexpected server error

Degraded State Handling¶

When the invariants YAML file cannot be loaded, the /ui/invariants endpoint returns an empty response with HTTP 200 rather than failing. This graceful degradation is logged and recorded as an error metric for monitoring:

An error metric is recorded with code invariants_load_error
The error is logged at ERROR level with details
The response returns an empty items array

Example degraded response:

{
  "items": [],
  "pagination": { "next_cursor": null, "has_more": false, "total_count": 0 }
}

Monitor for these conditions via:

increase(ui_request_error{error_code="invariants_load_error"}[1h]) > 0

Error Metrics Recording¶

Errors are recorded via record_endpoint_error():

record_endpoint_error(
    endpoint="/ui/deliveries/{id}",
    error_code="not_found",
    error_message=f"Notification not found: {notification_id}",
    status_code=404,
)

Structured Logging¶

Log Format¶

All logs include request context appended as key=value pairs:

INFO - list_deliveries called [request_id=req-abc123 cursor=None limit=25 status=DELIVERED]
INFO - UI request completed [request_id=req-abc123 endpoint=/ui/deliveries method=GET status_code=200 latency_ms=12.5]

Log Levels¶

Level	Usage
INFO	Request start/end, successful operations
WARNING	4xx errors, resource not found, degraded states
ERROR	5xx errors, unexpected exceptions
DEBUG	Detailed request parameters (disabled in production)

Using Structured Logger¶

from uns.api.observability import get_structured_logger, log_endpoint_call

logger = get_structured_logger(__name__)

# Log endpoint call with parameters
log_endpoint_call(
    logger,
    "list_deliveries",
    cursor=cursor,
    limit=limit,
    status=status,
)

# Log with custom context
logger.info(
    "Processing complete",
    extra={"items_returned": 25, "has_more": True},
)

Dashboard Queries¶

Prometheus Queries¶

For production deployments exporting to Prometheus:

Request Rate by Endpoint¶

sum(rate(ui_request_total[5m])) by (endpoint)

Error Rate¶

sum(rate(ui_request_total{status_code=~"4..|5.."}[5m]))
/
sum(rate(ui_request_total[5m]))
* 100

P95 Latency by Endpoint¶

histogram_quantile(0.95,
  sum(rate(ui_request_latency_ms_bucket[5m])) by (endpoint, le)
)

Error Count by Code¶

sum(increase(ui_request_error[1h])) by (error_code)

Log Queries¶

For log aggregation systems (e.g., Loki, Elasticsearch):

Find All Logs for a Request¶

{app="uns"} |= "request_id=req-abc123"

Find Error Responses¶

{app="uns"} |~ "status_code=4[0-9][0-9]|status_code=5[0-9][0-9]"

Find Slow Requests (>100ms)¶

{app="uns"} |~ "latency_ms=([1-9][0-9]{2,}|[1-9][0-9]{3,})"

Dashboard Recommendations¶

Overview Dashboard¶

Panel	Visualization	Query
Request Rate	Time series	`sum(rate(ui_request_total[5m])) by (endpoint)`
Error Rate %	Gauge	Error rate query above
P50/P95/P99 Latency	Time series	Latency percentiles by endpoint
Active Endpoints	Stat	Count of unique endpoints with traffic

Error Analysis Dashboard¶

Panel	Visualization	Query
Error Rate by Endpoint	Bar chart	Errors grouped by endpoint
Error Distribution	Pie chart	Errors by error_code
Error Timeline	Time series	Error rate over time
Recent Errors	Table	Last 100 error log entries

Latency Dashboard¶

Panel	Visualization	Query
Latency Distribution	Heatmap	Request latency distribution
P95 by Endpoint	Bar chart	P95 latency per endpoint
Slow Request Rate	Time series	Requests >100ms over time
Latency Correlation	Scatter	Latency vs response size

Configuration¶

MetricsCollector¶

The default in-memory collector is suitable for testing:

from uns.api.observability import configure_metrics_collector, MetricsCollector

# Use default in-memory collector
collector = configure_metrics_collector()

# Or provide custom collector
custom_collector = MetricsCollector()
configure_metrics_collector(custom_collector)

Production Export¶

For production, implement a custom MetricsCollector that exports to your metrics system:

class PrometheusMetricsCollector(MetricsCollector):
    """Exports metrics to Prometheus."""

    def __init__(self):
        super().__init__()
        self.request_counter = Counter(
            'ui_request_total',
            'Total UI requests',
            ['endpoint', 'method', 'status_code']
        )
        self.latency_histogram = Histogram(
            'ui_request_latency_ms',
            'UI request latency',
            ['endpoint', 'method']
        )

    def record_request(self, endpoint, method, status_code, latency_ms, labels=None):
        super().record_request(endpoint, method, status_code, latency_ms, labels)
        self.request_counter.labels(endpoint, method, status_code).inc()
        self.latency_histogram.labels(endpoint, method).observe(latency_ms)

API Reference¶

Observability Module¶

Function	Description
`get_request_id()`	Get current request ID from context
`set_request_id(id)`	Set request ID in context
`generate_request_id()`	Generate new unique request ID
`get_metrics_collector()`	Get global metrics collector
`configure_metrics_collector(collector)`	Configure global collector
`get_structured_logger(name)`	Get structured logger instance
`log_endpoint_call(logger, name, **params)`	Log endpoint invocation
`record_endpoint_error(endpoint, code, msg, status)`	Record error metric
`create_error_response_with_request_id(code, msg, details)`	Create error dict

MetricsCollector Methods¶

Method	Description
`record_request(endpoint, method, status, latency, labels)`	Record request metric
`record_error(endpoint, method, code, msg, status, labels)`	Record error metric
`get_request_count(endpoint, method)`	Get total request count
`get_error_count(endpoint, method, status)`	Get error count
`get_latency_stats(endpoint, method)`	Get latency statistics
`get_all_metrics()`	Get all recorded metrics
`clear()`	Clear all metrics (testing)

Verifying Instrumentation¶

Testing Request ID Propagation¶

# Verify request ID is returned in response header
curl -i http://localhost:8000/ui/deliveries | grep X-Request-ID

# Verify client-provided request ID is echoed back
curl -i -H "X-Request-ID: test-123" http://localhost:8000/ui/deliveries | grep X-Request-ID
# Expected: X-Request-ID: test-123

Testing Metrics Collection¶

from uns.api.observability import get_metrics_collector

# After making requests, inspect collected metrics
collector = get_metrics_collector()

# Check request counts
print(f"Deliveries requests: {collector.get_request_count('/ui/deliveries')}")

# Check latency stats
stats = collector.get_latency_stats('/ui/deliveries')
print(f"P95 latency: {stats['p95']}ms")

# Check error counts
print(f"404 errors: {collector.get_error_count('/ui/deliveries/{id}', status_code=404)}")

Testing Structured Logging¶

# Run with debug logging enabled
LOG_LEVEL=DEBUG python -m uvicorn uns.app:app

# Make a request and observe logs
curl http://localhost:8000/ui/deliveries

# Expected log output:
# INFO - list_deliveries called [request_id=req-abc123 cursor=None limit=25]
# INFO - UI request completed [request_id=req-abc123 endpoint=/ui/deliveries method=GET status_code=200 latency_ms=12.5]

Running Observability Tests¶

# Run all observability tests
pytest tests/uns/api/test_observability.py -v

# Run specific test class
pytest tests/uns/api/test_observability.py::TestMetricsForAllEndpoints -v

UI Data Contracts - API response formats
UI Scope - UI feature boundaries
Observability - System-wide observability