Observability — Prometheus, Grafana, Datadog, and GCP Cloud Monitoring¶

Kysira exposes Prometheus metrics from ext-proc, nginx-auth-adapter, and inference, and emits structured JSON decision logs to stdout. This document covers:

• Kubernetes deployments — Prometheus scraping, Grafana, and Datadog agent discovery
• GCP Cloud Run deployments — Cloud Logging structured logs and Cloud Monitoring log-based metrics

Architecture¶

Metrics are pulled by an in-cluster agent — Kysira services do not push. Each service exposes a standard Prometheus text-format /metrics endpoint. Your observability agent (Grafana Alloy, Prometheus, Datadog Agent) scrapes it on a configurable interval and forwards the data to your backend.

┌────────────────────────────── k8s cluster ──────────────────────────────────┐
│                                                                              │
│  ┌──────────────────┐  ┌──────────────────────┐  ┌─────────────────────┐   │
│  │  kysira-ext-proc  │  │  kysira-nginx-auth-   │  │  kysira-inference   │   │
│  │  :9090/metrics   │  │  adapter              │  │  :8081/metrics      │   │
│  └────────┬─────────┘  │  :9090/metrics        │  └──────────┬──────────┘   │
│           │            └──────────┬────────────┘             │              │
│           │ scrape                │ scrape                    │ scrape       │
│           └───────────────────────┼───────────────────────────┘             │
│                           ┌───────▼─────────┐                               │
│                           │  In-cluster     │                               │
│                           │  agent          │                               │
│                           │  (DaemonSet /   │                               │
│                           │   Deployment)   │                               │
│                           └───────┬─────────┘                               │
└───────────────────────────────────┼─────────────────────────────────────────┘
                                    │ HTTPS + API key
                           ┌────────▼─────────┐
                           │  Grafana Cloud   │  ← or Datadog / self-hosted
                           │  or Datadog      │
                           └──────────────────┘

Metrics exposed¶

ext-proc and nginx-auth-adapter (kysira_extproc_*)¶

Both services use the same metric names — they perform the same WAF role in different integration models (Envoy ext_proc vs nginx auth_request).

ext-proc port: 9090, path: /metrics (separate from gRPC port 50051). nginx-auth-adapter port: 9090, path: /metrics (separate from main port 8090).

inference (kysira_inference_*)¶

Port: 8081, path: /metrics (same port as the main API).

Agent discovery mechanisms¶

There are three distinct discovery mechanisms. Which one applies depends on what you run. All three are supported — each is controlled by a separate Helm value.

1. Prometheus pod annotations (metrics.prometheusAnnotations.enabled)¶

The simplest mechanism. Vanilla Prometheus and Grafana Alloy (with the default prometheus.scrape component) discover pods that carry:

prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "9090"   # or 8081 for inference

When to use: You run Grafana Alloy or a self-managed Prometheus configured to use annotation-based pod discovery. This is the default in many self-hosted setups.

When NOT to rely on this: kube-prometheus-stack (the most common enterprise Grafana install) uses the Prometheus Operator and ignores these annotations. Use ServiceMonitors instead.

2. Prometheus Operator ServiceMonitor (metrics.serviceMonitor.enabled)¶

kube-prometheus-stack deploys a Prometheus Operator that watches ServiceMonitor CRDs. When a ServiceMonitor exists, the Operator automatically adds the target to Prometheus's scrape config without any manual prometheus.yml editing.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus   # must match Prometheus Operator's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kysira-ext-proc
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

The release: prometheus label (or whatever label your Prometheus Operator is configured to select on) must match. Set it via metrics.serviceMonitor.additionalLabels.

When to use: You run kube-prometheus-stack or any Prometheus Operator deployment.

3. Datadog Agent autodiscovery (metrics.datadog.enabled)¶

The Datadog Agent uses its own annotation format entirely. The prometheus.io/* annotations are invisible to it. Instead, it reads ad.datadoghq.com/<container-name>.checks from the pod:

ad.datadoghq.com/ext-proc.checks: |
  {
    "openmetrics": {
      "instances": [{
        "openmetrics_endpoint": "http://%%host%%:9090/metrics",
        "namespace": "kysira",
        "metrics": ["kysira_extproc_.*"]
      }]
    }
  }

%%host%% is a Datadog autodiscovery template variable that resolves to the pod IP at runtime.

When to use: You run the Datadog Agent in your cluster with the OpenMetrics check enabled.

Helm configuration¶

All three mechanisms are opt-in and independent. Enable whichever ones match your stack.

ext-proc values¶

metrics:
  enabled: true                   # master switch — exposes /metrics at all
  path: /metrics
  port: 9090

  prometheusAnnotations:
    enabled: true                 # add prometheus.io/* pod annotations

  serviceMonitor:
    enabled: false                # create a ServiceMonitor CRD (Prometheus Operator)
    interval: 30s
    additionalLabels: {}          # e.g. { release: prometheus }

  datadog:
    enabled: false                # add ad.datadoghq.com/* pod annotations
    namespace: kysira             # Datadog metric namespace prefix

inference values¶

metrics:
  enabled: true
  path: /metrics
  port: 8081                      # same port as the main API

  prometheusAnnotations:
    enabled: true

  serviceMonitor:
    enabled: false
    interval: 30s
    additionalLabels: {}

  datadog:
    enabled: false
    namespace: kysira

Enabling for kube-prometheus-stack¶

# deploy/values-<env>.yaml
kysira-ext-proc:
  metrics:
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus   # match your Prometheus Operator's serviceMonitorSelector

kysira-inference:
  metrics:
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus

Enabling for Datadog¶

# deploy/values-<env>.yaml
kysira-ext-proc:
  metrics:
    prometheusAnnotations:
      enabled: false          # Datadog ignores these, disable to keep annotations clean
    datadog:
      enabled: true

kysira-inference:
  metrics:
    prometheusAnnotations:
      enabled: false
    datadog:
      enabled: true

Customer deployment guide¶

If your cluster already has Grafana or Datadog running, Kysira will be picked up automatically once you enable the right option above. No changes to your existing observability stack are required.

Self-hosted Prometheus (no operator)¶

If you run a plain Prometheus deployment (not via the Operator), add a scrape job manually to your prometheus.yml, or use the pod annotation method. ServiceMonitors won't be picked up without the Operator.

scrape_configs:
  - job_name: kysira-ext-proc
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        regex: kysira-ext-proc
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __address__
        regex: (.+)
        replacement: ${__meta_kubernetes_pod_ip}:$1

  - job_name: kysira-inference
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        regex: kysira-inference
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __address__
        regex: (.+)
        replacement: ${__meta_kubernetes_pod_ip}:$1

Relationship to Kysira telemetry¶

The Prometheus metrics described above are for your observability stack — they stay in your cluster.

GCP Cloud Run deployment — Cloud Logging and Cloud Monitoring¶

When running on Cloud Run (the GCP Service Extensions integration described in the GCP deployment guide), the Prometheus /metrics endpoint is not scraped by anything GCP-native — GMP (Google Managed Prometheus) targets GKE Pods, not Cloud Run revisions. Instead, Kysira ships observability through Cloud Logging and Cloud Monitoring log-based metrics.

How logs reach Cloud Logging¶

kysira-ext-proc writes every decision as a single-line JSON object to stdout. Cloud Run automatically forwards stdout to Cloud Logging under the run.googleapis.com/stdout log name. No agent, no log sink configuration, and no additional permissions are required beyond the default Cloud Run service account.

Every inspected request produces one log entry. The log entry includes a body_sample field (first 512 bytes of the raw request body) when:

• flagged: true — always included for flagged requests, or
• KYSIRA_LOG_BODY=true is set on the ext-proc service — included for every request regardless of score

{
  "severity": "INFO",
  "timestamp": "2026-06-16T12:34:56.789Z",
  "message": "decision",
  "score": 0.971,
  "flagged": true,
  "killed": true,
  "action": "active_kill",
  "mode": "active",
  "latency_ms": 38,
  "method": "POST",
  "path": "/api/chat",
  "detector": "sqli",
  "reason": "OR 1=1",
  "body_sample": "{\"prompt\":\"' OR 1=1--\"}",
  "body_truncated": false
}

body_truncated: true means the body exceeded 512 bytes and was cut.

Set KYSIRA_LOG_BODY=true during initial rollout to validate what the model sees across all traffic. Disable it (or leave it at the default false) in steady-state to avoid writing request payloads to your log store on every request.

Redacting sensitive fields¶

The query and body_sample fields can carry secrets or PII (tokens, API keys, emails). Before logging, ext-proc replaces the values of sensitive-named parameters and fields with {redacted}, keeping the names for debugging. Redaction is applied to:

• the URL query string (query), and
• JSON and application/x-www-form-urlencoded request bodies (body_sample) — JSON is walked recursively, so nested and array fields are covered too.

A built-in default set is always redacted: token, secret, password, passwd, pwd, key, auth, session, sig, signature, code, credential, cred, email, phone, ssn, account.

To cover your own fields, set KYSIRA_REDACT_KEYS to a comma-separated list of additional substrings. Matching is case-insensitive substring, so customer_id also covers external_customer_id:

KYSIRA_REDACT_KEYS=customer_id,order_ref,patient

{
  "message": "decision",
  "path": "/api/chat",
  "query": "page=2&access_token={redacted}",
  "body_sample": "{\"prompt\":\"hello\",\"api_key\":\"{redacted}\"}"
}

Two caveats:

• Only structured values are redacted. Secrets or PII sitting in free-text bodies (e.g. text/plain), or in a value that isn't behind a sensitive-named key, are not caught.
• Matching is broad by design — the default key matches any field containing "key", and a custom entry like id would redact every *_id field. Choose reasonably specific substrings.

Cloud Logging parses the JSON and promotes each key to a queryable jsonPayload.* field. Operational events (startup, mode switches, inference errors) use severity: INFO or severity: ERROR and are queryable the same way.

Querying decision logs¶

In Cloud Logging > Log Explorer, filter to just Kysira decision events:

resource.type="cloud_run_revision"
resource.labels.service_name="kysira-ext-proc"
jsonPayload.message="decision"

To see only blocked requests:

resource.type="cloud_run_revision"
resource.labels.service_name="kysira-ext-proc"
jsonPayload.killed=true

From the CLI:

gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="kysira-ext-proc" AND jsonPayload.killed=true' \
  --project=PROJECT_ID \
  --format=json \
  --freshness=1h

Log-based metrics for dashboarding¶

Create these four metrics once in Cloud Monitoring > Log-based Metrics (or via gcloud). They turn into standard Cloud Monitoring time-series that you can graph in any dashboard.

For the kysira/killed_total and kysira/flagged_total metrics, add a label on jsonPayload.detector to break down by attack type (SQLi vs prompt injection vs XSS).

Creating kysira/killed_total with a detector label via gcloud:

gcloud logging metrics create kysira_killed_total \
  --description="Kysira requests blocked by active enforcement" \
  --log-filter='resource.type="cloud_run_revision" jsonPayload.killed=true' \
  --project=PROJECT_ID

Label extraction is done in the Console UI (Metrics > Edit > Labels > Add label > Field: jsonPayload.detector).

Cloud Monitoring dashboard¶

Once the log-based metrics exist, create a dashboard in Cloud Monitoring > Dashboards > Create Dashboard with these suggested widgets:

Alerting¶

Alert on inference failures (fail-open events) — these mean enforcement is silently disabled:

# Alert if inference error rate exceeds 1 per minute for 5 minutes
gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Kysira inference errors" \
  --condition-filter='resource.type="cloud_run_revision" AND resource.labels.service_name="kysira-ext-proc" AND jsonPayload.message="inference error"' \
  --condition-threshold-value=1 \
  --condition-threshold-duration=300s \
  --project=PROJECT_ID

You can also alert directly on the kysira/killed_total log-based metric if you want to know when blocking activity spikes above a baseline.

Exporting logs to an external destination (log sink)¶

For continuous export — to a SIEM, a Kysira-managed analytics pipeline, or long-term archival — create a Cloud Logging sink. Sinks filter log entries before they leave the project and push them to the destination you choose.

Option A: Export flagged-request logs to a GCS bucket¶

Useful for forensic archival. Logs land as JSON-newline files, partitioned by date.

# Create the sink — filter to flagged requests only to minimise volume
gcloud logging sinks create kysira-flagged-sink \
  storage.googleapis.com/YOUR_BUCKET_NAME \
  --log-filter='resource.type="cloud_run_revision" AND resource.labels.service_name="kysira-ext-proc" AND jsonPayload.flagged=true' \
  --project=PROJECT_ID

# Grant the sink's writer identity permission to write to the bucket
SINK_SA=$(gcloud logging sinks describe kysira-flagged-sink \
  --project=PROJECT_ID --format='value(writerIdentity)')

gsutil iam ch "${SINK_SA}:roles/storage.objectCreator" gs://YOUR_BUCKET_NAME

Option B: Stream to a Pub/Sub topic (SIEM / real-time pipeline)¶

Useful for feeding a SIEM or triggering downstream processing in real time.

# Create a Pub/Sub topic to receive the entries
gcloud pubsub topics create kysira-decisions --project=PROJECT_ID

# Create the sink pointing at the topic
gcloud logging sinks create kysira-decisions-sink \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/kysira-decisions \
  --log-filter='resource.type="cloud_run_revision" AND resource.labels.service_name="kysira-ext-proc" AND jsonPayload.message="decision"' \
  --project=PROJECT_ID

# Grant the sink's writer identity permission to publish to the topic
SINK_SA=$(gcloud logging sinks describe kysira-decisions-sink \
  --project=PROJECT_ID --format='value(writerIdentity)')

gcloud pubsub topics add-iam-policy-binding kysira-decisions \
  --member="${SINK_SA}" \
  --role=roles/pubsub.publisher \
  --project=PROJECT_ID

Granting Kysira read access to your logs (optional)¶

If you want Kysira support to be able to query your decision logs directly (for example, during an incident), you can grant access scoped to a log view that covers only the kysira-ext-proc service — no other logs in your project are accessible.

Replace PROJECT_ID in the commands below with your Google Cloud project ID. You can find it in the Google Cloud Console in the project selector at the top of the page, or by running gcloud config get-value project.

# Create a scoped log view filtered to the ext-proc service
gcloud logging views create kysira-decisions-view \
  --bucket=_Default \
  --location=global \
  --log-filter='resource.type="cloud_run_revision" AND resource.labels.service_name="kysira-ext-proc"' \
  --project=PROJECT_ID

# Grant Kysira's service account read access to that view only
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:kysira-support@cs-poc-uv5os9gxrjsncireus36uzd.iam.gserviceaccount.com" \
  --role="roles/logging.viewAccessor" \
  --condition="expression=resource.name==\"projects/PROJECT_ID/locations/global/buckets/_Default/views/kysira-decisions-view\",title=kysira-view-only"

Access is revocable at any time by removing the IAM binding:

gcloud projects remove-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:kysira-support@cs-poc-uv5os9gxrjsncireus36uzd.iam.gserviceaccount.com" \
  --role="roles/logging.viewAccessor"