> ## Documentation Index > Fetch the complete documentation index at: https://infisical.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring and telemetry setup > Learn how to set up monitoring and telemetry for your self-hosted Infisical instance using Grafana, Prometheus, and OpenTelemetry. Infisical provides comprehensive monitoring and telemetry capabilities to help you monitor the health, performance, and usage of your self-hosted instance. This guide covers setting up monitoring using Grafana with two different telemetry collection approaches. ## Overview Infisical exports metrics in **OpenTelemetry (OTEL) format**, which provides maximum flexibility for your monitoring infrastructure. While this guide focuses on Grafana, the OTEL format means you can easily integrate with: * **Cloud-native monitoring**: AWS CloudWatch, Google Cloud Monitoring, Azure Monitor * **Observability platforms**: Datadog, New Relic, Splunk, Dynatrace * **Custom backends**: Any system that supports OTEL ingestion * **Traditional monitoring**: Prometheus, Grafana (as covered in this guide) Infisical supports two telemetry collection methods: 1. **Pull-based (Prometheus)**: Exposes metrics on a dedicated endpoint for Prometheus to scrape 2. **Push-based (OTLP)**: Sends metrics to an OpenTelemetry Collector via OTLP protocol Both approaches provide the same metrics data in OTEL format, so you can choose the one that best fits your infrastructure and monitoring strategy. ## Prerequisites * Self-hosted Infisical instance running * Access to deploy monitoring services (Prometheus, Grafana, etc.) * Basic understanding of Prometheus and Grafana ## Setup ### Environment variables Configure the following environment variables in your Infisical backend: ```bash theme={"dark"} # Enable telemetry collection OTEL_TELEMETRY_COLLECTION_ENABLED=true # Choose export type: "prometheus" or "otlp" OTEL_EXPORT_TYPE=prometheus ``` > [tab] Pull-based Monitoring (Prometheus) > > This approach exposes metrics on port 9464 at the `/metrics` endpoint, allowing Prometheus to scrape the data. The metrics are exposed in Prometheus format but originate from OpenTelemetry instrumentation. > > ### Configuration > > **1. Enable Prometheus export in Infisical** > > ```bash theme={"dark"} > OTEL_TELEMETRY_COLLECTION_ENABLED=true > OTEL_EXPORT_TYPE=prometheus > ``` > > **2. Expose the metrics port** > > Expose the metrics port in your Infisical backend: > > * **Docker**: Expose port 9464 > * **Kubernetes**: Create a service exposing port 9464 > * **Other**: Ensure port 9464 is accessible to your monitoring stack > > **3. Create Prometheus configuration** > > Create `prometheus.yml`: > > ```yaml theme={"dark"} > global: > scrape_interval: 30s > evaluation_interval: 30s > > scrape_configs: > - job_name: "infisical" > scrape_interval: 30s > static_configs: > - targets: ["infisical-backend:9464"] # Adjust hostname/port based on your deployment > metrics_path: "/metrics" > ``` > > > [!NOTE] > > > > Replace `infisical-backend:9464` with the actual hostname and port where your Infisical backend is running. This could be: > > > > * **Docker Compose**: `infisical-backend:9464` (service name) > > * **Kubernetes**: `infisical-backend.default.svc.cluster.local:9464` (service name) > > * **Bare Metal**: `192.168.1.100:9464` (actual IP address) > > * **Cloud**: `your-infisical.example.com:9464` (domain name) > > ### Deployment options > > Once you've configured Infisical to expose metrics, you'll need to deploy Prometheus to scrape and store them. Below are examples for different deployment environments. Choose the option that matches your infrastructure. > > > [tab] Docker Compose > > > > ```yaml theme={"dark"} > > services: > > prometheus: > > image: prom/prometheus:latest > > ports: > > - "9090:9090" > > volumes: > > - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro > > command: > > - "--config.file=/etc/prometheus/prometheus.yml" > > > > grafana: > > image: grafana/grafana:latest > > ports: > > - "3000:3000" > > environment: > > - GF_SECURITY_ADMIN_USER=admin > > - GF_SECURITY_ADMIN_PASSWORD=admin > > ``` > > > [tab] Kubernetes > > > > ```yaml theme={"dark"} > > # prometheus-deployment.yaml > > apiVersion: apps/v1 > > kind: Deployment > > metadata: > > name: prometheus > > spec: > > replicas: 1 > > selector: > > matchLabels: > > app: prometheus > > template: > > metadata: > > labels: > > app: prometheus > > spec: > > containers: > > - name: prometheus > > image: prom/prometheus:latest > > ports: > > - containerPort: 9090 > > volumeMounts: > > - name: config > > mountPath: /etc/prometheus > > volumes: > > - name: config > > configMap: > > name: prometheus-config > > > > --- > > # prometheus-service.yaml > > apiVersion: v1 > > kind: Service > > metadata: > > name: prometheus > > spec: > > selector: > > app: prometheus > > ports: > > - port: 9090 > > targetPort: 9090 > > type: ClusterIP > > ``` > > > [tab] Helm > > > > ```bash theme={"dark"} > > helm repo add prometheus-community https://prometheus-community.github.io/helm-charts > > helm install prometheus prometheus-community/prometheus \ > > --set server.config.global.scrape_interval=30s \ > > --set server.config.scrape_configs[0].job_name=infisical \ > > --set server.config.scrape_configs[0].static_configs[0].targets[0]=infisical-backend:9464 > > ``` > [tab] Push-based Monitoring (OTLP) > > This approach sends metrics directly to an OpenTelemetry Collector via the OTLP protocol. This gives you the most flexibility as you can configure the collector to export to multiple backends simultaneously. > > ### Configuration > > **1. Enable OTLP export in Infisical** > > ```bash theme={"dark"} > OTEL_TELEMETRY_COLLECTION_ENABLED=true > OTEL_EXPORT_TYPE=otlp > OTEL_EXPORT_OTLP_ENDPOINT=http://otel-collector:4318/v1/metrics > OTEL_COLLECTOR_BASIC_AUTH_USERNAME=infisical > OTEL_COLLECTOR_BASIC_AUTH_PASSWORD=infisical > OTEL_OTLP_PUSH_INTERVAL=30000 > ``` > > **2. Create OpenTelemetry Collector configuration** > > Create `otel-collector-config.yaml`: > > ```yaml theme={"dark"} > extensions: > health_check: > pprof: > zpages: > basicauth/server: > htpasswd: > inline: | > your_username:your_password > > receivers: > otlp: > protocols: > http: > endpoint: 0.0.0.0:4318 > auth: > authenticator: basicauth/server > > prometheus: > config: > scrape_configs: > - job_name: otel-collector > scrape_interval: 30s > static_configs: > - targets: [infisical-backend:9464] > metric_relabel_configs: > - action: labeldrop > regex: "service_instance_id|service_name" > > processors: > batch: > > exporters: > prometheus: > endpoint: "0.0.0.0:8889" > auth: > authenticator: basicauth/server > resource_to_telemetry_conversion: > enabled: true > > service: > extensions: [basicauth/server, health_check, pprof, zpages] > pipelines: > metrics: > receivers: [otlp] > processors: [batch] > exporters: [prometheus] > ``` > > > [!WARNING] > > > > Replace `your_username:your_password` with your chosen credentials. These must match the values you set in Infisical's `OTEL_COLLECTOR_BASIC_AUTH_USERNAME` and `OTEL_COLLECTOR_BASIC_AUTH_PASSWORD` environment variables. > > **3. Create Prometheus configuration** > > Create Prometheus configuration for the collector: > > ```yaml theme={"dark"} > global: > scrape_interval: 30s > evaluation_interval: 30s > > scrape_configs: > - job_name: "otel-collector" > scrape_interval: 30s > static_configs: > - targets: ["otel-collector:8889"] # Adjust hostname/port based on your deployment > metrics_path: "/metrics" > ``` > > > [!NOTE] > > > > Replace `otel-collector:8889` with the actual hostname and port where your OpenTelemetry Collector is running. This could be: > > > > * **Docker Compose**: `otel-collector:8889` (service name) > > * **Kubernetes**: `otel-collector.default.svc.cluster.local:8889` (service name) > > * **Bare Metal**: `192.168.1.100:8889` (actual IP address) > > * **Cloud**: `your-collector.example.com:8889` (domain name) > > ### Deployment options > > After configuring Infisical and the OpenTelemetry Collector, you'll need to deploy the collector to receive metrics from Infisical. Below are examples for different deployment environments. Choose the option that matches your infrastructure. > > > [tab] Docker Compose > > > > ```yaml theme={"dark"} > > services: > > otel-collector: > > image: otel/opentelemetry-collector-contrib:latest > > ports: > > - 4318:4318 # OTLP http receiver > > - 8889:8889 # Prometheus exporter metrics > > volumes: > > - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro > > command: > > - "--config=/etc/otelcol-contrib/config.yaml" > > ``` > > > [tab] Kubernetes > > > > ```yaml theme={"dark"} > > # otel-collector-deployment.yaml > > apiVersion: apps/v1 > > kind: Deployment > > metadata: > > name: otel-collector > > spec: > > replicas: 1 > > selector: > > matchLabels: > > app: otel-collector > > template: > > metadata: > > labels: > > app: otel-collector > > spec: > > containers: > > - name: otel-collector > > image: otel/opentelemetry-collector-contrib:latest > > ports: > > - containerPort: 4318 > > - containerPort: 8889 > > volumeMounts: > > - name: config > > mountPath: /etc/otelcol-contrib > > volumes: > > - name: config > > configMap: > > name: otel-collector-config > > ``` > > > [tab] Helm > > > > ```bash theme={"dark"} > > helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts > > helm install otel-collector open-telemetry/opentelemetry-collector \ > > --set config.receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 \ > > --set config.exporters.prometheus.endpoint=0.0.0.0:8889 > > ``` ## Available metrics Infisical emits metrics on two OpenTelemetry meters simultaneously. Choose which to scrape based on your deployment scale. * **High-cardinality, per-actor meters:** the `Infisical` and `API` meters' original metrics (`infisical.http.server.request.*`, `infisical.http.server.error.count`, `infisical.secret.read.count`, `infisical.auth.attempt.count`, `infisical.kmip.operation.count`) include per-actor labels such as `user.email`, `identity.name`, `client.address`, `user_agent.original`, `organization.name`, `project.name`, `secret.path`, and `secret.name`. The `SecretSyncs`, `PkiSyncs`, and `Integrations` meters likewise carry unbounded labels such as `syncId`. Useful for self-hosted deployments where you want per-user visibility directly in Grafana. May become expensive at large scale (many users, identities, or IPs) due to label cardinality. * **`InfisicalCore` meter (bounded-cardinality):** all newer metrics (queue, local refresh, audit log, product analytics aggregation, alert dispatch, event outbox, permission cache, secret cache, rate limit, build info, `infisical.core.http.request.count`, `infisical.core.http.error.count`, authentication latency, token renewal, SSO config changes, SCIM provisioning, secret sync, rotation, and dynamic secret outcomes, and database connection pool) use **only bounded enums and static route templates** as labels (e.g. HTTP method, parameterized `http.route`, error type, outcome). A strict allowlist drops every other attribute at the SDK level, so **no tenant or actor identifiers are emitted**; no organization ID, project IDs, names, emails, IPs, or user agents. This keeps the series count independent of how many organizations, users, or identities you have. Per-org and per-actor detail is available in audit logs instead. All meters emit at the same time. If you're a self-hosted instance and find the per-actor labels useful, keep using the high-cardinality metrics. For larger or multi-tenant deployments, scrape only the `InfisicalCore` metrics to keep cardinality under control. To eliminate the cost of the high-cardinality meters entirely, set `OTEL_DROP_HIGH_CARDINALITY_METERS=true`. Infisical then stops recording to the `Infisical`, `API`, `SecretSyncs`, `PkiSyncs`, and `Integrations` meters altogether, so nothing is measured, stored, exported, or held in memory for them. The instruments still exist in code, so nothing errors. The `InfisicalCore` metrics, the Node runtime metrics, and the HTTP instrumentation metrics are unaffected and keep emitting. Defaults to `false`. > [!WARNING] > > **With the high-cardinality meters enabled and `OTEL_EXPORT_TYPE=prometheus`, memory use grows with > label cardinality for the lifetime of the process.** Labels such as `client.address`, > `user_agent.original`, `secret.path`, and `secret.name` have no natural ceiling. On a busy instance > this shows up as heap that climbs with uptime and longer garbage collection pauses. Restarting the > instance releases the memory. > > Whether that matters depends on churn rather than on how many organizations you have. If your secret > paths and names, users, source IP addresses, and client versions are all largely stable, the growth > is slow enough to ignore. It keeps going as long as any one of them keeps turning over: secrets > created, renamed, or reorganized over time; clients on dynamic or short-lived IP addresses such as CI > runners and autoscaled pods; or a fleet that upgrades often. If any of those describe your > deployment, set `OTEL_DROP_HIGH_CARDINALITY_METERS=true` and get per-actor detail from audit logs > instead. > > This tradeoff applies only to Prometheus export. With `OTEL_EXPORT_TYPE=otlp`, memory for these > meters doesn't accumulate across the process lifetime. For per-user / per-identity / per-IP / per-project breakdowns, query the audit log table. It carries `actorId`, `actorType`, `projectId`, `ip`, `userAgent` and full event detail. Metrics give you the *rate* and *latency*; audit logs give you the *who* and *where*. ### Resource attributes (every metric) Every emitted metric carries these resource-level attributes (no per-metric cardinality cost): * `service.name` — the fixed identifier for the Infisical backend service * `service.version` — the release version or git SHA of the running Infisical instance * `git.commit.sha` — the exact commit the build was produced from, when available * `deployment.environment` — the environment the instance is running in (e.g. `production`, `staging`, `development`) ### Core API metrics These metrics track all HTTP API requests to Infisical, including request counts, latency, and errors. Use these to monitor overall API health, identify performance bottlenecks, and track usage patterns across users and machine identities. > [accordion] Total API Requests > > **Metric Name**: `infisical.http.server.request.count` > > **Type**: Counter > > **Unit**: `{request}` > > **Description**: Total number of API requests to Infisical (covers both human users and machine identities) > > **Attributes**: > > * `infisical.organization.id` (string): Organization ID > * `infisical.organization.name` (string): Organization name (e.g., "Platform Engineering Team") > * `infisical.user.id` (string, optional): User ID if human user > * `infisical.user.email` (string, optional): User email (e.g., "[jane.doe@cisco.com](mailto:jane.doe@cisco.com)") > * `infisical.identity.id` (string, optional): Machine identity ID > * `infisical.identity.name` (string, optional): Machine identity name (e.g., "prod-k8s-operator") > * `infisical.auth.method` (string, optional): Auth method used > * `http.request.method` (string): HTTP method (GET, POST, PUT, DELETE) > * `http.route` (string): API endpoint route pattern > * `http.response.status_code` (int): HTTP status code > * `infisical.project.id` (string, optional): Project ID > * `infisical.project.name` (string, optional): Project name > * `user_agent.original` (string, optional): User agent string > * `client.address` (string, optional): IP address > [accordion] Request Duration > > **Metric Name**: `infisical.http.server.request.duration` > > **Type**: Histogram > > **Unit**: `s` (seconds) > > **Description**: API request latency > > **Buckets**: \[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] > > **Attributes**: > > * `infisical.organization.id` (string): Organization ID > * `infisical.organization.name` (string): Organization name > * `infisical.user.id` (string, optional): User ID if human user > * `infisical.user.email` (string, optional): User email > * `infisical.identity.id` (string, optional): Machine identity ID > * `infisical.identity.name` (string, optional): Machine identity name > * `http.request.method` (string): HTTP method > * `http.route` (string): API endpoint route pattern > * `http.response.status_code` (int): HTTP status code > * `infisical.project.id` (string, optional): Project ID > * `infisical.project.name` (string, optional): Project name > [accordion] API Errors by Actor > > **Metric Name**: `infisical.http.server.error.count` > > **Type**: Counter > > **Unit**: `{error}` > > **Description**: API errors grouped by actor (for identifying misconfigured services) > > **Attributes**: > > * `infisical.organization.id` (string): Organization ID > * `infisical.organization.name` (string): Organization name > * `infisical.user.id` (string, optional): User ID if human > * `infisical.user.email` (string, optional): User email > * `infisical.identity.id` (string, optional): Identity ID if machine > * `infisical.identity.name` (string, optional): Identity name > * `http.route` (string): API endpoint where error occurred > * `http.request.method` (string): HTTP method > * `error.type` (string): Error category/type (client\_error, server\_error, auth\_error, rate\_limit\_error, etc.) > * `infisical.project.id` (string, optional): Project ID > * `infisical.project.name` (string, optional): Project name > * `client.address` (string, optional): IP address > * `user_agent.original` (string, optional): User agent information > [accordion] Total API Requests (InfisicalCore) > > **Metric Name**: `infisical.core.http.request.count` > > **Type**: Counter > > **Unit**: `{request}` > > **Description**: Total API requests, carrying only the HTTP method, the parameterized route template, and the status code. The low-cardinality counterpart to `infisical.http.server.request.count`, so the series count stays independent of how many organizations, users, or identities you have. It still emits when `OTEL_DROP_HIGH_CARDINALITY_METERS=true`. > > For HTTP error rate per route, divide the `http.response.status_code >= 400` series by the total on this metric. Do not use `infisical.core.http.error.count` as the numerator: it counts thrown errors, and a response that sets a 4xx or 5xx status without throwing (an unmatched route, for example) never reaches it. > > **Attributes**: > > * `http.request.method` (string): HTTP method, normalized to the ten names the OpenTelemetry HTTP conventions define, with anything else reported as `_OTHER` so this metric joins against `http.server.request.duration`. The Infisical meter keep the method as sent, so use those to see an unusual method such as `PROPFIND` on its own > * `http.route` (string): Parameterized route template (e.g. `/api/v1/secrets/:secretName`), never the raw path. `unknown` for a request that matched no route. > * `http.response.status_code` (int): HTTP status code > [accordion] API Errors (InfisicalCore) > > **Metric Name**: `infisical.core.http.error.count` > > **Type**: Counter > > **Unit**: `{error}` > > **Description**: API errors with bounded error classification. The low-cardinality counterpart to `infisical.http.server.error.count`; same error volume, but labels are restricted to the InfisicalCore allowlist so the series count stays independent of org/user/identity count. Use this for alerting and dashboards at scale; use `infisical.http.server.error.count` (high-cardinality meter) when you need per-actor attribution. For per-org error breakdowns, query the audit log. > > **Attributes**: > > * `http.request.method` (string): HTTP method, normalized to the ten names the OpenTelemetry HTTP conventions define, with anything else reported as `_OTHER` so this metric joins against `http.server.request.duration`. The Infisical meter keep the method as sent, so use those to see an unusual method such as `PROPFIND` on its own > * `http.route` (string): Parameterized route template where the error occurred (e.g. `/api/v1/secrets/:secretName`), never the raw path > * `error.type` (string): Bounded error classification ### Secret operations metrics These metrics provide visibility into secret access patterns, helping you understand which secrets are being accessed, by whom, and from where. Essential for security auditing and access pattern analysis. > [accordion] Secret Read Operations > > **Metric Name**: `infisical.secret.read.count` > > **Type**: Counter > > **Unit**: `{operation}` > > **Description**: Number of secret read operations > > **Attributes**: > > * `infisical.organization.id` (string): Organization ID > * `infisical.organization.name` (string): Organization name > * `infisical.project.id` (string): Project ID > * `infisical.project.name` (string): Project name (e.g., "payment-service-secrets") > * `infisical.environment` (string): Environment (dev, staging, prod) > * `infisical.secret.path` (string): Path to secrets (e.g., "/microservice-a/database") > * `infisical.secret.name` (string, optional): Name of secret > * `infisical.user.id` (string, optional): User ID if human > * `infisical.user.email` (string, optional): User email > * `infisical.identity.id` (string, optional): Machine identity ID > * `infisical.identity.name` (string, optional): Machine identity name > * `user_agent.original` (string, optional): User agent/SDK information > * `client.address` (string, optional): IP address ### Authentication metrics These metrics track authentication attempts and outcomes, enabling you to monitor login success rates, detect potential security threats, and identify authentication issues. > [accordion] Login Attempts > > **Metric Name**: `infisical.auth.attempt.count` > > **Type**: Counter > > **Unit**: `{attempt}` > > **Description**: Authentication attempts (both successful and failed) > > **Attributes**: > > * `infisical.organization.id` (string): Organization ID > * `infisical.organization.name` (string): Organization name > * `infisical.user.id` (string, optional): User ID if human (if identifiable) > * `infisical.user.email` (string, optional): User email (if identifiable) > * `infisical.identity.id` (string, optional): Identity ID if machine (if identifiable) > * `infisical.identity.name` (string, optional): Identity name (if identifiable) > * `infisical.auth.method` (string): Authentication method attempted > * `infisical.auth.result` (string): success or failure > * `error.type` (string, optional): Reason for failure if failed (invalid\_credentials, expired\_token, invalid\_token, etc.) > * `client.address` (string): IP address > * `user_agent.original` (string, optional): User agent/client information > * `infisical.auth.attempt.username` (string, optional): Attempted username/email (if available) > [accordion] Authentication Latency (InfisicalCore) > > **Metric Name**: `infisical.auth.attempt.duration` > > **Type**: Histogram > > **Unit**: `s` (seconds) > > **Description**: Authentication attempt latency by method and result. External verifications (SAML, OIDC, Kubernetes, AWS, GCP, Azure, OCI, AliCloud, ...) include the IdP/provider network round trip, so this is what tells you "SAML logins suddenly got slow" or "Kubernetes auth verification is timing out". Covers every user and machine identity login flow, including LDAP user login. > > **Attributes** (bounded): > > * `infisical.auth.method` (string): Authentication method (`email`, `saml`, `oidc`, `google`, `github`, `gitlab`, `ldap`, `universal-auth`, `kubernetes-auth`, `aws-auth`, `gcp-auth`, `azure-auth`, `oci-auth`, `alicloud-auth`, `tls-cert-auth`, `oidc-auth`, `jwt-auth`, `ldap-auth`, `spiffe-auth`) > * `infisical.auth.result` (string): `success` or `failure` > * `error.type` (string, optional): Bounded failure classification (present on failures) > [accordion] Token Renewal (InfisicalCore) > > **Metric Name**: `infisical.auth.token.renewal.count` > > **Type**: Counter > > **Unit**: `{renewal}` > > **Description**: Machine identity access token renewal attempts by outcome. Distinct from `infisical.auth.attempt.*`, which tracks initial logins. > > **Attributes** (bounded): > > * `outcome` (string): `success` or `failure` > * `infisical.auth.method` (string, optional): Identity auth method when known > * `error.type` (string, optional): Bounded failure classification (present on failures) > [!NOTE] > > The `infisical.auth.attempt.count` counter lives on the high-cardinality `Infisical` meter, while > `infisical.auth.attempt.duration` and `infisical.auth.token.renewal.count` live on the bounded > `InfisicalCore` meter. There is no per-identity "active identity" gauge: there is no reliable > last-authenticated timestamp in the schema, so monthly/weekly active identity counts are best derived > from the `infisical.auth.attempt.*` series in your metrics backend (e.g. `count by (...)` over a time > window) rather than a snapshot gauge. ### SSO configuration metrics (InfisicalCore) These metrics track changes to SSO configuration, helping you detect unexpected reconfiguration of identity providers. > [accordion] SSO Config Changes > > **Metric Name**: `infisical.sso.config.change.count` > > **Type**: Counter > > **Unit**: `{change}` > > **Description**: SSO configuration create/update events by provider. > > **Attributes** (bounded): > > * `sso.provider` (string): `saml`, `oidc`, or `ldap` > * `sso.action` (string): `create` or `update` ### SCIM provisioning metrics (InfisicalCore) These metrics track SCIM provisioning operations (user and group lifecycle), enabling you to monitor directory-sync throughput and failures. > [accordion] SCIM Operations > > **Metric Name**: `infisical.scim.operation.count` > > **Type**: Counter > > **Unit**: `{operation}` > > **Description**: SCIM provisioning operations by type and outcome. > > **Attributes** (bounded): > > * `scim.operation` (string): `create_user`, `update_user`, `replace_user`, `delete_user`, `create_group`, `update_group`, `replace_group`, `delete_group` > * `outcome` (string): `success` or `failure` > * `error.type` (string, optional): Bounded failure classification (present on failures) > [accordion] SCIM Operation Latency > > **Metric Name**: `infisical.scim.operation.duration` > > **Type**: Histogram > > **Unit**: `s` (seconds) > > **Description**: Latency of SCIM provisioning operations. Same attributes as `infisical.scim.operation.count`. ### Database metrics (InfisicalCore) This metric provides visibility into connection pool health. Query latency is intentionally not emitted, as managed databases (for example, Amazon RDS Performance Insights) already report per-statement latency at the server. > [accordion] Connection Pool > > **Metric Name**: `infisical.db.pool.connections` > > **Type**: Observable Gauge > > **Unit**: `{connection}` > > **Description**: Knex/tarn connection pool counts, observed on each export. Watch `pending` rising with `used` saturated for pool exhaustion. > > **Attributes** (bounded): > > * `db.pool.state` (string): `used`, `free`, or `pending` ### Key management interoperability protocol metrics These metrics track Key Management Interoperability Protocol (KMIP) operations, providing visibility into key management activities including key creation, retrieval, activation, revocation, and destruction. > [accordion] KMIP Operations > > **Metric Name**: `infisical.kmip.operation.count` > > **Type**: Counter > > **Unit**: `{operation}` > > **Description**: Number of KMIP operations performed > > **Attributes**: > > * `infisical.kmip.operation.type` (string): Operation type (`create`, `get`, `get_attributes`, `activate`, `revoke`, `destroy`, `locate`, `register`) > * `infisical.organization.id` (string): Organization ID > * `infisical.project.id` (string): Project ID > * `infisical.kmip.client.id` (string): KMIP client ID performing the operation > * `infisical.kmip.object.id` (string, optional): Managed object/key ID > * `infisical.kmip.object.name` (string, optional): Managed object/key name > * `infisical.identity.id` (string, optional): Machine identity ID > * `infisical.identity.name` (string, optional): Machine identity name > * `user_agent.original` (string, optional): User agent string > * `client.address` (string, optional): Client IP address ### Integration & secret sync metrics These metrics monitor secret synchronization operations between Infisical and external systems, helping you track sync health, identify integration failures, and troubleshoot connectivity issues. > [accordion] integration\_secret\_sync\_errors > > Integration secret sync error count > > * **Labels**: `version`, `integration`, `integrationId`, `type`, `status`, `name`, `projectId` > * **Example**: Monitor integration sync failures across different services > [accordion] secret\_sync\_sync\_secrets\_errors > > Secret sync operation error count > > * **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name` > * **Example**: Track secret sync failures to external systems > [accordion] secret\_sync\_import\_secrets\_errors > > Secret import operation error count > > * **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name` > * **Example**: Monitor secret import failures > [accordion] secret\_sync\_remove\_secrets\_errors > > Secret removal operation error count > > * **Labels**: `version`, `destination`, `syncId`, `projectId`, `type`, `status`, `name` > * **Example**: Track secret removal operation failures ### Secret sync, rotation, and dynamic secret metrics (InfisicalCore) These metrics tell you whether secret syncs, secret rotations, and dynamic secret lease revocations are succeeding. Their only labels are the destination, rotation type, or provider, so they show which kind of integration is failing. To find the specific sync, rotation, or dynamic secret, check its status in Infisical or the [audit logs](https://infisical.com/docs/documentation/platform/audit-logs.md). Secret syncs and rotations retry a failed attempt automatically, and the outcome counters record every attempt. Use `attempts.exhausted` to tell a failure that Infisical will retry from one it won't: * `outcome="failure"` with `attempts.exhausted="false"`: the attempt failed, and Infisical will try again * `outcome="failure"` with `attempts.exhausted="true"`: the last attempt failed, so the sync or rotation failed The status change counters record a change only when a sync, rotation, or dynamic secret moves between working and failing. If a secret sync stays broken and fails on every run, it records one `failed` change, so the number of `failed` changes is the number of syncs that broke, not the number of failed attempts. > [accordion] Secret Sync Outcome > > **Metric Name**: `infisical.secret_sync.outcome.count` > > **Type**: Counter > > **Unit**: `{attempt}` > > **Description**: Secret sync attempts by operation and outcome. If a job can't start because another job for the same secret sync is still running, the job records a `failure` with `attempts.exhausted="true"` once Infisical stops waiting for the other job. > > **Attributes**: > > * `destination` (string): the sync destination, e.g. `aws-parameter-store`, `github`, `azure-key-vault` > * `operation` (string): `sync`, `import`, or `remove` > * `outcome` (string): `success` or `failure` > * `attempts.exhausted` (string): `"true"` or `"false"` > [accordion] Secret Sync Status Change > > **Metric Name**: `infisical.secret_sync.status_change.count` > > **Type**: Counter > > **Unit**: `{change}` > > **Description**: Secret syncs whose result changed from succeeded to failed, or from failed back to succeeded. Only the final result of a sync run counts, so retried attempts, imports, removes, and jobs that couldn't start don't record a change. If a secret sync has never succeeded, its first failure doesn't record a change. > > **Attributes**: > > * `destination` (string) > * `new_status` (string): `failed` or `healthy` > [accordion] Secret Rotation Outcome > > **Metric Name**: `infisical.secret_rotation.outcome.count` > > **Type**: Counter > > **Unit**: `{attempt}` > > **Description**: Secret rotation attempts by rotation type and outcome. If a user selects **Rotate Secrets** or calls the rotate secrets endpoint for the rotation type (such as [the one for PostgreSQL credentials](https://infisical.com/docs/api-reference/endpoints/secret-rotations/postgres-credentials/rotate-secrets.md)), the rotation runs once without retries, so its outcome always has `attempts.exhausted="true"`. > > **Attributes**: > > * `type` (string): the rotation type, e.g. `postgres-credentials`, `mysql-credentials` > * `outcome` (string): `success` or `failure` > * `attempts.exhausted` (string): `"true"` or `"false"` > [accordion] Secret Rotation Status Change > > **Metric Name**: `infisical.secret_rotation.status_change.count` > > **Type**: Counter > > **Unit**: `{change}` > > **Description**: Secret rotations whose status changed from successful to failed, or from failed back to successful. A rotation records `failed` only when its last attempt fails. It records `healthy` when a later rotation succeeds, or when a local account rotation reconciles its credentials. > > **Attributes**: > > * `type` (string) > * `new_status` (string): `failed` or `healthy` > [accordion] Dynamic Secret Revocation Outcome > > **Metric Name**: `infisical.dynamic_secret.revocation.outcome.count` > > **Type**: Counter > > **Unit**: `{revocation}` > > **Description**: Dynamic secret lease revocations by provider, trigger, and outcome. A scheduled revocation records one outcome per lease, from its last attempt. A retry records every attempt, with `attempts.exhausted="true"` on the last one. If a user force-revokes a lease and the provider rejects the revocation, Infisical deletes the lease without recording an outcome. > > **Attributes**: > > * `provider` (string): the dynamic secret provider, e.g. `sql-database`, `aws-iam`, `redis` > * `trigger` (string): one of: > * `scheduled`: the lease expired > * `retry`: Infisical retried a revocation after a scheduled or manual revocation failed > * `manual`: a user revoked the lease > * `pruning`: a user deleted the dynamic secret, and Infisical revoked its remaining leases (one outcome per lease) > * `outcome` (string): `success` or `failure` > * `attempts.exhausted` (string): `"true"` or `"false"` > [accordion] Dynamic Secret Status Change > > **Metric Name**: `infisical.dynamic_secret.status_change.count` > > **Type**: Counter > > **Unit**: `{change}` > > **Description**: Dynamic secrets whose lease revocations changed between working and failing. A dynamic secret records `failed` when a scheduled revocation fails on its last attempt, and `healthy` when any later scheduled, retried, or manual revocation succeeds. > > **Attributes**: > > * `provider` (string) > * `new_status` (string): `failed` or `healthy` > [accordion] Dynamic Secret Orphaned Lease > > **Metric Name**: `infisical.dynamic_secret.orphaned_lease.count` > > **Type**: Counter > > **Unit**: `{failure}` > > **Description**: Dynamic secret leases that Infisical stopped trying to revoke after every automatic attempt failed. Infisical records one for each lease whose last retry fails, and one for each lease it can't revoke while deleting a dynamic secret. Each one is a live credential at the provider that you need to revoke manually. > > **Attributes**: > > * `provider` (string) > [accordion] Dynamic Secret Unrecorded Credential > > **Metric Name**: `infisical.dynamic_secret.unrecorded_credential.count` > > **Type**: Counter > > **Unit**: `{credential}` > > **Description**: Credentials that a provider created for a new lease when Infisical failed to save the lease and then also failed to revoke the credential. Each one is a live credential at the provider with no lease in Infisical, so nothing will revoke it automatically. Find and remove it at the provider. > > **Attributes**: > > * `provider` (string) ### Job queue metrics (InfisicalCore) These metrics give per-queue visibility into BullMQ worker health: throughput, latency, contention, failures, and stalls. Use these to detect stuck workers, queue backlog, and which queues are failing. > [accordion] Queue Job Count > > **Metric Name**: `infisical.queue.job.count` > > **Type**: Counter > > **Unit**: `{job}` > > **Description**: Jobs processed by outcome. > > **Attributes**: > > * `queue.name` (string): e.g. `audit-log`, `secret-sync`, `secret-rotation-v2` > * `job.name` (string): BullMQ job name > * `outcome` (string): `completed` or `failed` > [accordion] Queue Job Duration > > **Metric Name**: `infisical.queue.job.duration` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Job processing duration (worker pickup to completion). Skipped on framework-level failures where `processedOn` is undefined, so the histogram is not polluted with phantom zero-duration points. > > **Attributes**: `queue.name`, `job.name`, `outcome` > [accordion] Queue Job Wait > > **Metric Name**: `infisical.queue.job.wait` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Time the job spent waiting for a worker (queue contention). Subtracts the configured `job.opts.delay` so intentional scheduling doesn't inflate percentiles. Only recorded on `completed` jobs. > > **Attributes**: `queue.name`, `job.name` > [accordion] Queue Job Failure (classified) > > **Metric Name**: `infisical.queue.job.failure.count` > > **Type**: Counter > > **Unit**: `{failure}` > > **Description**: Failures classified by error type. Alert when `attempts.exhausted="true"` — those are real failures (all retries spent), not transient errors. > > **Attributes**: > > * `queue.name`, `job.name` > * `error.type` (string): one of `validation`, `auth`, `permission`, `not_found`, `rate_limit`, `db`, `timeout`, `network`, `client_closed`, `cryptography`, `policy`, `scim`, `oidc`, `internal`, `unknown` > * `attempts.exhausted` (string): `"true"` or `"false"` > [accordion] Queue Stalled > > **Metric Name**: `infisical.queue.stalled.count` > > **Type**: Counter > > **Unit**: `{job}` > > **Description**: Stalled jobs (the worker's lock on a job expired without completing it). Strongest signal of a stuck worker, OOM, or network partition. Previously invisible. > > **Attributes**: `queue.name` > [accordion] Queue Depth > > **Metric Name**: `infisical.queue.depth` > > **Type**: Observable Gauge > > **Unit**: `{job}` > > **Description**: Current number of jobs in each queue state. The SDK invokes the callback on each scrape / push interval. > > **Attributes**: > > * `queue.name` (string) > * `queue.state` (string): `waiting`, `active`, `delayed`, `failed`, `completed`, etc. ### Local refresh metrics (InfisicalCore) Every Infisical instance runs a few background tasks on a timer to refresh settings it keeps in memory, such as the license and SSO configuration. Each instance runs its own copy, so a failing refresh means that instance is serving stale settings while the others may be fine. These metrics are labelled by task, and each instance reports its own values. > [accordion] Local Refresh Run Count > > **Metric Name**: `infisical.local_refresh.run.count` > > **Type**: Counter > > **Unit**: `{run}` > > **Description**: Refresh runs by outcome. A `skipped` run means the previous run was still in progress when the next one was due. > > **Attributes**: > > * `job.name` (string): the refresh task, e.g. `self-hosted-license-sync` > * `outcome` (string): `completed`, `failed`, or `skipped` > * `error.type` (string): only on `failed` runs, with the same values as `infisical.queue.job.failure.count` > [accordion] Local Refresh Run Duration > > **Metric Name**: `infisical.local_refresh.run.duration` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: How long each refresh run took. Not recorded for skipped runs. > > **Attributes**: `job.name`, `outcome` > [accordion] Local Refresh Consecutive Failures > > **Metric Name**: `infisical.local_refresh.consecutive_failures` > > **Type**: Observable Gauge > > **Unit**: `{run}` > > **Description**: How many runs in a row have failed for each refresh task on this instance. It returns to `0` on the next successful run, so a single failure clears itself. Alert on this gauge rather than on the failure count: a value that stays above `2` or `3` means the task keeps failing and the instance has stopped picking up changes to those settings. > > **Attributes**: `job.name` ### Audit log metrics (InfisicalCore) End-to-end audit-log pipeline observability: how many events get enqueued per event type / actor, how long persistence takes, and how many are ultimately dropped. > [accordion] Audit Log Enqueued > > **Metric Name**: `infisical.audit_log.enqueued.count` > > **Type**: Counter > > **Unit**: `{event}` > > **Description**: Audit log events enqueued to BullMQ for persistence. > > **Attributes**: > > * `audit_log.event_type` (string): e.g. `LOGIN_USER`, `CREATE_SECRET`, ... > * `audit_log.actor_type` (string): `user`, `identity`, `service` > [accordion] Audit Log Persist Duration > > **Metric Name**: `infisical.audit_log.persist.duration` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Latency from worker pickup to durable storage. > > **Attributes**: > > * `audit_log.backend` (string): `postgres` or `clickhouse` > * `audit_log.event_type` (string) > [accordion] Audit Log Dropped > > **Metric Name**: `infisical.audit_log.dropped.count` > > **Type**: Counter > > **Unit**: `{event}` > > **Description**: Audit log events that exhausted BullMQ retries and were not persisted. **Operators should alert when this is non-zero** — a dropped audit event is a compliance signal. > > **Attributes**: > > * `audit_log.event_type` (string) > * `audit_log.drop_reason` (string): `max_retries` ### Audit log stream metrics (InfisicalCore) Per-provider observability for the audit-log stream feature (Datadog, Splunk, Custom HTTP, Azure, Cribl). > [accordion] Audit Log Stream Delivery > > **Metric Name**: `infisical.audit_log_stream.delivery.count` > > **Type**: Counter > > **Unit**: `{delivery}` > > **Description**: Per-provider stream delivery attempts. > > **Attributes**: > > * `audit_log_stream.provider` (string): `datadog`, `splunk`, `custom`, `azure`, `cribl` > * `audit_log_stream.id` (string): stream config ID (on the exhausted-drop counter) > * `outcome` (string): `success` or `failure` > * `error.type` (string, only on failure): one of the closed enum values > [accordion] Audit Log Stream Delivery Duration > > **Metric Name**: `infisical.audit_log_stream.delivery.duration` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Per-provider stream delivery latency (HTTP round trip to the SIEM). > > **Attributes**: `audit_log_stream.provider`, `outcome`, `error.type` (on failure) ### Product analytics aggregation metrics (InfisicalCore) High-volume product analytics events (secret pulls, machine identity logins, certificate issuance, and similar) are buffered in Redis and rolled up by a cron every 10 minutes instead of being sent one by one. The buffer is deliberately bounded, so events that the roll-up doesn't reach in time are discarded. These metrics tell you whether that is happening. They're only emitted when product analytics is enabled (`TELEMETRY_ENABLED`), since the roll-up doesn't run otherwise. > [accordion] Aggregation Published > > **Metric Name**: `infisical.product_analytics.published.count` > > **Type**: Counter > > **Unit**: `{event}` > > **Description**: Buffered events drained from Redis and sent upstream. Use it as the denominator for the drop rate below. > > **Attributes**: > > * `product_analytics.event_type` (string): the buffered event, e.g. `secrets pulled` > [accordion] Aggregation Dropped > > **Metric Name**: `infisical.product_analytics.dropped.count` > > **Type**: Counter > > **Unit**: `{event}` > > **Description**: Buffered events discarded before they were sent. `retention` means the event aged past the 30-minute buffer window while the roll-up was behind; `unparseable` means its payload could not be read back and points at a bug rather than at capacity. > > **Attributes**: > > * `product_analytics.event_type` (string) > * `product_analytics.drop_reason` (string): `retention` or `unparseable` > [accordion] Aggregation Shard Backlog > > **Metric Name**: `infisical.product_analytics.shard.backlog` > > **Type**: Histogram > > **Unit**: `{entry}` > > **Description**: Entries left in a buffer shard after the roll-up finished with it. Zero on a healthy run. > > **Attributes**: > > * `product_analytics.event_type` (string) **What to alarm on**: a sustained `retention` drop rate, and any `unparseable` drop at all. Losing a few analytics events occasionally costs nothing, but a rate that doesn't return to zero means the roll-up is permanently behind the write rate. The backlog histogram is the early warning: it rises before drops start, and a shard sitting near 100,000 entries is also shedding the oldest events on write, invisibly to the drop counter. ### Alert dispatch metrics (InfisicalCore) Every "notify me when X happens" alert (credential expiry and anything else built on the alert module) is delivered by a dispatch job. Jobs come from two places: a scheduled scan that looks for targets that have come due, and an event the product recorded, such as an identity's authentication method changing. This metric records how each job ended, so you can tell a dropped notification apart from a job that had nothing to send. > [accordion] Alert Dispatch Outcome > > **Metric Name**: `infisical.alert.dispatch.outcome.count` > > **Type**: Counter > > **Unit**: `{job}` > > **Description**: Alert dispatch jobs by resource type and outcome. > > **Attributes**: > > * `type` (string): the alert's resource type, e.g. `identity.authentication` > * `outcome` (string): one of the closed enum values below > > **Outcomes**: > > | Outcome | Meaning | > | - | - | > | `delivery_success` | Every channel the run touched delivered. | > | `delivery_partial` | At least one channel failed, so the notification was dropped on that channel. Dedup is per channel and target, so the next run retries only the failed channels. | > | `delivery_failed` | No channel delivered — the whole run dropped. | > | `alert_not_found` | The alert was deleted between enqueue and run. | > | `alert_disabled` | The alert was disabled between enqueue and run. | > | `no_provider` | No provider is registered for the alert's resource type. A misconfiguration. | > | `no_due_targets` | Nothing to alert on: either nothing matched the alert condition this run, or the targets named by the event no longer exist. | > | `no_channels` | The alert has no enabled channels, so the run is skipped before scanning. | > | `no_recipients` | Every channel in the run was email with no resolvable recipient — the recipients left the organization, or the recipient group emptied out. | > | `all_deduped` | Targets matched, but all had already been alerted inside the dedup window. | > | `no_matching_alert` | Event path only. The event arrived and no alert subscribes to it. Normal in steady state. | > > **What to alarm on**: `delivery_failed` and `delivery_partial`. Both mean a channel could not be reached and a notification was dropped. Keep `no_recipients` out of that alarm — it is customer configuration drift, not a delivery fault, and it needs a nudge to the organization rather than a page to you. > > Track the `delivery_*` share of the total separately, as a health signal rather than an alarm: a persistently low ratio means the cron is enqueueing mostly no-op jobs and should pre-filter instead. Leave `no_matching_alert` out of that ratio. Events are recorded whether or not anyone subscribes to them, so on the event path it's the majority outcome by design. ### Event outbox metrics (InfisicalCore) Event-triggered features don't poll for changes. The product writes an event to an outbox table in the same transaction as the change itself. A relay picks pending events up every 10 seconds and hands each one to the consumers that subscribe to it. Alerts are the first consumer, under the name `alert`. These metrics cover the outbox itself rather than what a consumer does with an event; for that, see the alert dispatch metrics above. > [accordion] Oldest Pending Event Age > > **Metric Name**: `infisical.event_outbox.oldest_pending_age` > > **Type**: Observable Gauge > > **Unit**: `s` > > **Description**: Age of the oldest event that hasn't reached its consumer yet. Refreshed on each relay tick, and absent for a consumer with nothing pending. > > **Attributes**: > > * `event_outbox.consumer` (string): the consumer the events are waiting for, e.g. `alert` > [accordion] Event Delivery Lag > > **Metric Name**: `infisical.event_outbox.lag` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Seconds from the event happening to the outbox recording a result for it. The floor is the 10-second relay interval, so a p99 well above that puts the delay in delivery rather than in pickup. > > **Attributes**: > > * `event_outbox.consumer` (string) > * `event_outbox.status` (string): `delivered`, `retry` (the consumer asked for another attempt), or `failed` > [accordion] Events Exhausted > > **Metric Name**: `infisical.event_outbox.exhausted.count` > > **Type**: Counter > > **Unit**: `{event}` > > **Description**: Events given up on, because the consumer rejected the event outright, because it used up all 8 delivery attempts, or because a worker claimed it and never reported back. Each one is a notification a customer configured and didn't receive. > > **Attributes**: > > * `event_outbox.consumer` (string) **What to alarm on**: any sustained rate of exhausted events, and an `oldest_pending_age` above a small multiple of the relay interval. The two say different things. A rising age with no exhausted events means events are being recorded but not delivered, and they're still in the table waiting for the relay or its workers to come back. An exhausted event is one nothing will retry; its row stays for 30 days with the last error on it, and the delivery failures leading up to it are in the logs. ### Permission cache metrics (InfisicalCore) The CASL permission cache uses a fingerprint-based two-tier scheme. These metrics tell you whether the cache is doing its job. > [accordion] Permission Cache Lookup > > **Metric Name**: `infisical.permission_cache.lookup.count` > > **Type**: Counter > > **Unit**: `{lookup}` > > **Description**: Per-lookup branch: `marker_hit` (fast path, 0 DB reads), `fingerprint_match` (1 DB read, cached data returned), `full_refetch` (full DB re-fetch), or `fingerprint_error` (fingerprint fetch failed, bypassing cache). > > **Attributes**: > > * `cache.result` (string): `marker_hit`, `fingerprint_match`, `full_refetch`, `fingerprint_error` > [accordion] Permission Cache Fingerprint Duration > > **Metric Name**: `infisical.permission_cache.fingerprint.duration` > > **Type**: Histogram > > **Unit**: `s` > > **Description**: Time to compute the lightweight permission fingerprint (1 DB read on marker expiry). ### Secret cache metrics (InfisicalCore) The secret service caches encrypted secret payloads to avoid redundant decryption on repeated reads. These metrics tell you whether the cache is effective and whether entries are being skipped for exceeding the size cap. > [accordion] Secret Cache Access > > **Metric Name**: `infisical.secret.cache.access.count` > > **Type**: Counter > > **Unit**: `{access}` > > **Description**: Secret service-layer cache accesses by outcome. `not_modified` (client revalidation returned 304), `hit` (served from cache), or `miss` (cache empty/stale, full read performed). > > **Attributes**: > > * `cache.result` (string): `not_modified`, `hit`, `miss` > [accordion] Secret Cache Entry Bytes > > **Metric Name**: `infisical.secret.cache.entry.bytes` > > **Type**: Histogram > > **Unit**: `By` > > **Description**: Encrypted secret cache entry size computed at write time. Use this to size the cache and tune the per-entry byte cap. > [accordion] Secret Cache Oversize Skip > > **Metric Name**: `infisical.secret.cache.oversize_skip.count` > > **Type**: Counter > > **Unit**: `{skip}` > > **Description**: Secret cache writes skipped because the entry exceeded the max byte cap. A high rate means large payloads are never being cached and will always incur a full read. ### Rate limit metrics (InfisicalCore) > [accordion] Rate Limit Exceeded > > **Metric Name**: `infisical.rate_limit.exceeded.count` > > **Type**: Counter > > **Unit**: `{request}` > > **Description**: HTTP 429 responses (rate limit exceeded). Labels are intentionally bounded to `http.route` only — for per-actor breakdowns, query the audit log. > > **Attributes**: > > * `http.route` (string) > * `http.request.method` (string): HTTP method, normalized to the ten names the OpenTelemetry HTTP conventions define, with anything else reported as `_OTHER`. This matches `infisical.core.http.request.count`, so you can divide one by the other to get a 429 rate per route and method. ### Build info (InfisicalCore) > [accordion] Build Info > > **Metric Name**: `infisical.build.info` > > **Type**: Observable Gauge > > **Description**: Always emits `1`. The labels carry the deployed version, git SHA, and Node version. Use this to filter Grafana dashboards by deployed version without paying per-metric cardinality. > > **Attributes**: > > * `service.version` (string) > * `git.commit.sha` (string) > * `node.version` (string) ### Node runtime metrics (auto) Heap usage, GC pause, event loop lag, and other Node runtime metrics are auto-emitted via `@opentelemetry/instrumentation-runtime-node`. Metric names follow the OTel runtime semantic conventions (`nodejs.eventloop.delay.*`, `v8js.heap.size.*`, etc.). ### System metrics These low-level HTTP metrics are automatically collected by OpenTelemetry's instrumentation layer, providing baseline performance data for all HTTP traffic. > [!WARNING] > > Infisical now follows the stable OpenTelemetry HTTP semantic conventions, which renamed both of these metrics and changed their unit from milliseconds to seconds. > > | Old name | New name | > | - | - | > | `http.server.duration` (ms) | `http.server.request.duration` (s) | > | `http.client.duration` (ms) | `http.client.request.duration` (s) | > > Their attributes were renamed too. `http.method` became `http.request.method`, and `http.status_code` became `http.response.status_code`. > > A dashboard or alert built on the old names stops returning data once you upgrade, and a latency threshold carried across unchanged is wrong by a factor of 1000. To emit both name sets at the same time while you move dashboards over, set `OTEL_SEMCONV_STABILITY_OPT_IN=http/dup`, then unset it when you're done. That roughly doubles the volume of these two metrics for as long as it's set. > > Under `http/dup` the old metrics keep `http.method` and `http.status_code`, so your existing queries keep working. They don't get `net.host.name` back, because that label is the reason for the change: it carries the client-controlled `Host` header, and it grows a new time series for every value a client sends. > [accordion] http\_server\_request\_duration > > **Metric Name**: `http.server.request.duration` > > **Type**: Histogram > > **Unit**: `s` (seconds) > > **Description**: Inbound HTTP request duration, measured at the socket rather than at the route handler, so it also covers requests that never matched a route. > > **Attributes**: > > * `http.request.method` (string): HTTP method, normalized to a fixed set (`_OTHER` for anything unrecognized) > * `http.response.status_code` (int): HTTP status code > * `error.type` (string, optional): Present only when the response failed > > Infisical opts in to the stable OpenTelemetry HTTP semantic conventions. The older conventions labelled this metric with `net.host.name`, the raw `Host` request header, which is client-controlled and grows a new time series per value; the stable conventions omit it. `http.route` is not populated, so use `infisical.core.http.request.count` for a per-route breakdown. > [accordion] http\_client\_request\_duration > > **Metric Name**: `http.client.request.duration` > > **Type**: Histogram > > **Unit**: `s` (seconds) > > **Description**: Outbound HTTP request duration, aggregated across every destination. > > **Attributes**: > > * `http.request.method` (string): HTTP method, normalized to a fixed set > * `http.response.status_code` (int): HTTP status code > * `error.type` (string, optional): Present only when the request failed > > The destination host is deliberately omitted. Infisical calls a different set of hosts for every deployment (webhook URLs, secret sync destinations, identity providers), so labelling this metric with the destination would grow the series count without limit. There is no setting that restores it. For per-destination detail, use the secret sync and audit log stream metrics, which are labelled by provider. ## Troubleshooting > [accordion] Metrics not appearing > > If your metrics aren't showing up in Prometheus or your monitoring system, check the following: > > * Verify `OTEL_TELEMETRY_COLLECTION_ENABLED=true` is set in your Infisical environment variables > * Ensure the correct `OTEL_EXPORT_TYPE` is set (`prometheus` or `otlp`) > * Check network connectivity between Infisical and your monitoring services (Prometheus or OTLP collector) > * For pull-based monitoring: Verify port 9464 is exposed and accessible > * For push-based monitoring: Verify the OTLP endpoint URL is correct and reachable > * Check Infisical backend logs for any errors related to metrics export > [accordion] Authentication errors > > If you're experiencing authentication errors with the OpenTelemetry Collector: > > * Verify basic auth credentials in your OTLP configuration match between Infisical and the collector > * Check that `OTEL_COLLECTOR_BASIC_AUTH_USERNAME` and `OTEL_COLLECTOR_BASIC_AUTH_PASSWORD` match the credentials in your `otel-collector-config.yaml` > * Ensure the htpasswd format in the collector configuration is correct > * Test the collector endpoint manually using curl with the same credentials to verify they work This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.