From 62592be60bb1430fd2e1e8f2f78fb14b57e0ffa2 Mon Sep 17 00:00:00 2001 From: GareArc Date: Fri, 27 Feb 2026 12:32:48 -0800 Subject: [PATCH] docs(enterprise): split telemetry docs into README and data dictionary Separate background/configuration instructions from the data dictionary: - README.md: Overview, configuration, correlation model, content gating - DATA_DICTIONARY.md: Pure reference format with signals and attributes The data dictionary is now concise (465 lines vs 911) and focuses on attribute types and relationships without verbose explanations. --- api/enterprise/telemetry/DATA_DICTIONARY.md | 1041 +++++-------------- api/enterprise/telemetry/README.md | 116 +++ 2 files changed, 405 insertions(+), 752 deletions(-) create mode 100644 api/enterprise/telemetry/README.md diff --git a/api/enterprise/telemetry/DATA_DICTIONARY.md b/api/enterprise/telemetry/DATA_DICTIONARY.md index 42797d70c0..fb8f35e693 100644 --- a/api/enterprise/telemetry/DATA_DICTIONARY.md +++ b/api/enterprise/telemetry/DATA_DICTIONARY.md @@ -1,197 +1,92 @@ # Dify Enterprise Telemetry Data Dictionary -This document provides a comprehensive reference for all telemetry signals emitted by the Dify Enterprise OpenTelemetry (OTEL) exporter. It is intended for platform engineers and data scientists integrating Dify with observability stacks like Prometheus, Grafana, Jaeger, or Honeycomb. +Quick reference for all telemetry signals emitted by Dify Enterprise. For configuration and architecture details, see [README.md](./README.md). -## 1. Overview +## Resource Attributes -Dify Enterprise uses a "slim span + rich companion log" architecture to provide high-fidelity observability without overwhelming trace storage. +Attached to every signal (Span, Metric, Log). -- **Traces (Spans)**: Capture the structure, identity, and timing of high-level operations (Workflows and Nodes). -- **Structured Logs**: Provide deep context (inputs, outputs, metadata) for every event, correlated to spans via `trace_id` and `span_id`. -- **Metrics**: Provide 100% accurate counters and histograms for usage, performance, and error tracking. +| Attribute | Type | Example | +|-----------|------|---------| +| `service.name` | string | `dify` | +| `host.name` | string | `dify-api-7f8b` | -### Signal Architecture +## Traces (Spans) -```mermaid -graph TD - A[Workflow Run] -->|Span| B(dify.workflow.run) - A -->|Log| C(dify.workflow.run detail) - B ---|trace_id| C - - D[Node Execution] -->|Span| E(dify.node.execution) - D -->|Log| F(dify.node.execution detail) - E ---|span_id| F - - G[Message/Tool/etc] -->|Log| H(dify.* event) - G -->|Metric| I(dify.* counter/histogram) -``` - -## 2. Configuration - -The Enterprise OTEL exporter is configured via environment variables. - -| Variable | Description | Default | -|----------|-------------|---------| -| `ENTERPRISE_ENABLED` | Master switch for all enterprise features. | `false` | -| `ENTERPRISE_TELEMETRY_ENABLED` | Master switch for enterprise telemetry. | `false` | -| `ENTERPRISE_OTLP_ENDPOINT` | OTLP collector endpoint (e.g., `http://otel-collector:4318`). | - | -| `ENTERPRISE_OTLP_HEADERS` | Custom headers for OTLP requests (e.g., `x-scope-orgid=tenant1`). | - | -| `ENTERPRISE_OTLP_PROTOCOL` | OTLP transport protocol (`http` or `grpc`). | `http` | -| `ENTERPRISE_OTLP_API_KEY` | Bearer token for authentication. | - | -| `ENTERPRISE_INCLUDE_CONTENT` | Whether to include sensitive content (inputs/outputs) in logs. | `true` | -| `ENTERPRISE_SERVICE_NAME` | Service name reported to OTEL. | `dify` | -| `ENTERPRISE_OTEL_SAMPLING_RATE` | Sampling rate for traces (0.0 to 1.0). Metrics are always 100%. | `1.0` | - -## 3. Resource Attributes - -These attributes are attached to every signal (Span, Metric, Log) emitted by the exporter. - -| Attribute | Description | Example | -|-----------|-------------|---------| -| `service.name` | The name of the service. | `dify` | -| `host.name` | The hostname of the container/machine. | `dify-api-7f8b` | - -## 4. Traces (Spans) - -Dify emits spans for long-running or structural operations. - -### 4.1 `dify.workflow.run` - -Represents the full execution of a workflow. +### `dify.workflow.run` | Attribute | Type | Description | |-----------|------|-------------| -| `dify.trace_id` | string | The business trace ID (Workflow Run ID). | -| `dify.tenant_id` | string | Tenant identifier. | -| `dify.app_id` | string | Application identifier. | -| `dify.workflow.id` | string | Workflow definition ID. | -| `dify.workflow.run_id` | string | Unique ID for this specific run. | -| `dify.workflow.status` | string | Final status (`succeeded`, `failed`, `stopped`, etc.). | -| `dify.workflow.error` | string | Error message if the run failed. | -| `dify.workflow.elapsed_time` | float | Total execution time in seconds. | -| `dify.invoke_from` | string | Source of the trigger (`api`, `webapp`, `debug`). | -| `dify.conversation.id` | string | Conversation ID (if applicable). | -| `dify.message.id` | string | Message ID (if applicable). | -| `dify.invoked_by` | string | User ID who triggered the run. | -| `dify.parent.trace_id` | string | (Optional) Trace ID of the parent workflow. | -| `dify.parent.workflow.run_id` | string | (Optional) Run ID of the parent workflow. | -| `dify.parent.node.execution_id` | string | (Optional) Execution ID of the parent node. | -| `dify.parent.app.id` | string | (Optional) App ID of the parent workflow. | +| `dify.trace_id` | string | Business trace ID (Workflow Run ID) | +| `dify.tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.workflow.id` | string | Workflow definition ID | +| `dify.workflow.run_id` | string | Unique ID for this run | +| `dify.workflow.status` | string | `succeeded`, `failed`, `stopped`, etc. | +| `dify.workflow.error` | string | Error message if failed | +| `dify.workflow.elapsed_time` | float | Total execution time (seconds) | +| `dify.invoke_from` | string | `api`, `webapp`, `debug` | +| `dify.conversation.id` | string | Conversation ID (optional) | +| `dify.message.id` | string | Message ID (optional) | +| `dify.invoked_by` | string | User ID who triggered the run | +| `dify.parent.trace_id` | string | Parent workflow trace ID (optional) | +| `dify.parent.workflow.run_id` | string | Parent workflow run ID (optional) | +| `dify.parent.node.execution_id` | string | Parent node execution ID (optional) | +| `dify.parent.app.id` | string | Parent app ID (optional) | -### 4.2 `dify.node.execution` - -Represents the execution of a single node within a workflow. +### `dify.node.execution` | Attribute | Type | Description | |-----------|------|-------------| -| `dify.trace_id` | string | The business trace ID. | -| `dify.tenant_id` | string | Tenant identifier. | -| `dify.app_id` | string | Application identifier. | -| `dify.workflow.id` | string | Workflow definition ID. | -| `dify.workflow.run_id` | string | Workflow Run ID this node belongs to. | -| `dify.message.id` | string | Message ID (if applicable). | -| `dify.conversation.id` | string | Conversation ID (if applicable). | -| `dify.node.execution_id` | string | Unique ID for this node execution. | -| `dify.node.id` | string | Node ID in the workflow graph. | -| `dify.node.type` | string | Type of node (`llm`, `knowledge-retrieval`, `tool`, etc.). | -| `dify.node.title` | string | Display title of the node. | -| `dify.node.status` | string | Execution status (`succeeded`, `failed`). | -| `dify.node.error` | string | Error message if the node failed. | -| `dify.node.elapsed_time` | float | Execution time in seconds. | -| `dify.node.index` | int | Execution order index. | -| `dify.node.predecessor_node_id` | string | ID of the node that triggered this one. | -| `dify.node.iteration_id` | string | (Optional) ID of the iteration this node belongs to. | -| `dify.node.loop_id` | string | (Optional) ID of the loop this node belongs to. | -| `dify.node.parallel_id` | string | (Optional) ID of the parallel branch. | -| `dify.node.invoked_by` | string | User ID who triggered the execution. | +| `dify.trace_id` | string | Business trace ID | +| `dify.tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.workflow.id` | string | Workflow definition ID | +| `dify.workflow.run_id` | string | Workflow Run ID | +| `dify.message.id` | string | Message ID (optional) | +| `dify.conversation.id` | string | Conversation ID (optional) | +| `dify.node.execution_id` | string | Unique node execution ID | +| `dify.node.id` | string | Node ID in workflow graph | +| `dify.node.type` | string | Node type (see appendix) | +| `dify.node.title` | string | Display title | +| `dify.node.status` | string | `succeeded`, `failed` | +| `dify.node.error` | string | Error message if failed | +| `dify.node.elapsed_time` | float | Execution time (seconds) | +| `dify.node.index` | int | Execution order index | +| `dify.node.predecessor_node_id` | string | Triggering node ID | +| `dify.node.iteration_id` | string | Iteration ID (optional) | +| `dify.node.loop_id` | string | Loop ID (optional) | +| `dify.node.parallel_id` | string | Parallel branch ID (optional) | +| `dify.node.invoked_by` | string | User ID who triggered execution | -### 4.3 `dify.node.execution.draft` +### `dify.node.execution.draft` -Identical to `dify.node.execution`, but emitted during "Preview" or "Debug" runs of a single node. +Same attributes as `dify.node.execution`. Emitted during Preview/Debug runs. -## 5. Correlation Model - -Dify uses deterministic ID generation to ensure signals are correlated across different services and asynchronous tasks. - -### ID Generation Rules -- `trace_id`: Derived from the correlation ID (workflow_run_id or node_execution_id for drafts) using `int(UUID(correlation_id))` -- `span_id`: Derived from the source ID using `SHA256(source_id)[:8]` - -### Scenario A: Simple Workflow -A single workflow run with multiple nodes. All spans and logs share the same `trace_id` (derived from `workflow_run_id`). - -``` -trace_id = UUID(workflow_run_id) -├── [root span] dify.workflow.run (span_id = hash(workflow_run_id)) -│ ├── [child] dify.node.execution - "Start" (span_id = hash(node_exec_id_1)) -│ ├── [child] dify.node.execution - "LLM" (span_id = hash(node_exec_id_2)) -│ └── [child] dify.node.execution - "End" (span_id = hash(node_exec_id_3)) -``` - -### Scenario B: Nested Sub-Workflow -A workflow calling another workflow via a Tool or Sub-workflow node. The child workflow's spans are linked to the parent via `parent_span_id`. Both workflows share the same trace_id. - -``` -trace_id = UUID(outer_workflow_run_id) ← shared across both workflows -├── [root] dify.workflow.run (outer) (span_id = hash(outer_workflow_run_id)) -│ ├── dify.node.execution - "Start Node" -│ ├── dify.node.execution - "Tool Node" (triggers sub-workflow) -│ │ └── [child] dify.workflow.run (inner) (span_id = hash(inner_workflow_run_id)) -│ │ ├── dify.node.execution - "Inner Start" -│ │ └── dify.node.execution - "Inner End" -│ └── dify.node.execution - "End Node" -``` - -**Key attributes for nested workflows:** -- Inner workflow's `dify.parent.trace_id` = outer `workflow_run_id` -- Inner workflow's `dify.parent.node.execution_id` = tool node's `execution_id` -- Inner workflow's `dify.parent.workflow.run_id` = outer `workflow_run_id` -- Inner workflow's `dify.parent.app.id` = outer `app_id` - -### Scenario C: Draft Node Execution -A single node run in isolation (debugger/preview mode). It creates its own trace where the node span is the root. - -``` -trace_id = UUID(node_execution_id) ← own trace, NOT part of any workflow -└── dify.node.execution.draft (span_id = hash(node_execution_id)) -``` - -**Key difference:** Draft executions use `node_execution_id` as the correlation_id, so they are NOT children of any workflow trace. - -## 6. Counters +## Counters All counters are cumulative and emitted at 100% accuracy. -### 6.1 Token Counters - -⚠️ **Warning on Double-Counting**: `dify.tokens.total` at the workflow level includes all tokens from its nodes. To get the true total usage for a tenant, you **MUST** filter by `operation_type`. +### Token Counters | Metric | Unit | Description | |--------|------|-------------| -| `dify.tokens.total` | `{token}` | Total tokens consumed. | -| `dify.tokens.input` | `{token}` | Input (prompt) tokens. | -| `dify.tokens.output` | `{token}` | Output (completion) tokens. | +| `dify.tokens.total` | `{token}` | Total tokens consumed | +| `dify.tokens.input` | `{token}` | Input (prompt) tokens | +| `dify.tokens.output` | `{token}` | Output (completion) tokens | -**Labels for Token Counters:** -- `tenant_id`: Tenant identifier. -- `app_id`: Application identifier. -- `operation_type`: `workflow`, `node_execution`, `message`, `rule_generate`, `code_generate`, `structured_output`, `instruction_modify`. -- `model_provider`: LLM provider (e.g., `openai`). -- `model_name`: LLM model (e.g., `gpt-4`). -- `node_type`: Workflow node type (if `operation_type=node_execution`). +**Labels:** +- `tenant_id`, `app_id`, `operation_type`, `model_provider`, `model_name`, `node_type` (if node_execution) -**PromQL Example (Total Input Tokens per Tenant):** -```promql -sum(dify_tokens_input_total{tenant_id="my-tenant", operation_type="workflow"}) -``` +⚠️ **Warning:** `dify.tokens.total` at workflow level includes all node tokens. Filter by `operation_type` to avoid double-counting. -### 6.2 Request Counters +### Request Counters | Metric | Unit | Description | |--------|------|-------------| -| `dify.requests.total` | `{request}` | Total number of operations. | +| `dify.requests.total` | `{request}` | Total operations count | -**Labels vary by `type`:** +**Labels by type:** | `type` | Additional Labels | |--------|-------------------| @@ -206,13 +101,13 @@ sum(dify_tokens_input_total{tenant_id="my-tenant", operation_type="workflow"}) | `generate_name` | `tenant_id`, `app_id` | | `prompt_generation` | `tenant_id`, `app_id`, `operation_type`, `model_provider`, `model_name`, `status` | -### 6.3 Error Counters +### Error Counters | Metric | Unit | Description | |--------|------|-------------| -| `dify.errors.total` | `{error}` | Total number of failed operations. | +| `dify.errors.total` | `{error}` | Total failed operations | -**Labels vary by `type`:** +**Labels by type:** | `type` | Additional Labels | |--------|-------------------| @@ -223,7 +118,7 @@ sum(dify_tokens_input_total{tenant_id="my-tenant", operation_type="workflow"}) | `tool` | `tenant_id`, `app_id`, `tool_name` | | `prompt_generation` | `tenant_id`, `app_id`, `operation_type`, `model_provider`, `model_name` | -### 6.4 Other Counters +### Other Counters | Metric | Unit | Labels | |--------|------|--------| @@ -233,557 +128,274 @@ sum(dify_tokens_input_total{tenant_id="my-tenant", operation_type="workflow"}) | `dify.app.updated.total` | `{app}` | `tenant_id`, `app_id` | | `dify.app.deleted.total` | `{app}` | `tenant_id`, `app_id` | -## 7. Histograms - -Histograms measure the distribution of durations and latencies. +## Histograms | Metric | Unit | Labels | |--------|------|--------| | `dify.workflow.duration` | `s` | `tenant_id`, `app_id`, `status` | -| `dify.node.duration` | `s` | `tenant_id`, `app_id`, `node_type`, `model_provider`, `plugin_name` (if tool/knowledge) | +| `dify.node.duration` | `s` | `tenant_id`, `app_id`, `node_type`, `model_provider`, `plugin_name` | | `dify.message.duration` | `s` | `tenant_id`, `app_id`, `model_provider`, `model_name` | | `dify.message.time_to_first_token` | `s` | `tenant_id`, `app_id`, `model_provider`, `model_name` | | `dify.tool.duration` | `s` | `tenant_id`, `app_id`, `tool_name` | | `dify.prompt_generation.duration` | `s` | `tenant_id`, `app_id`, `operation_type`, `model_provider`, `model_name` | -**PromQL Example (P95 Node Duration by Type):** -```promql -histogram_quantile(0.95, sum by (le, node_type) (rate(dify_node_duration_bucket[5m]))) -``` +## Structured Logs -## 8. Structured Logs +### Span Companion Logs -Dify emits 13 types of structured logs. Logs are categorized as "Span Companion" (accompanying a span) or "Standalone" (metric-only events). +Logs that accompany spans. Signal type: `span_detail` -### 8.1 Span Companion Logs +#### `dify.workflow.run` Companion Log -These logs contain the full payload for spans, providing rich detail beyond what is captured in the span attributes. - -#### 8.1.1 `dify.workflow.run` Companion Log - -**Signal Type:** `span_detail` - -This log accompanies the `dify.workflow.run` span and includes ALL span attributes PLUS additional detail attributes. - -**Additional Attributes (beyond span attributes):** +**Common attributes:** All span attributes (see Traces section) plus: | Additional Attribute | Type | Always Present | Description | |---------------------|------|----------------|-------------| -| `dify.user.id` | string | No | User identifier (when available) | -| `gen_ai.usage.total_tokens` | int | No | Total tokens consumed by this workflow (sum of all nodes) | -| `dify.workflow.version` | string | Yes | Workflow version identifier | -| `dify.workflow.inputs` | string/JSON | Yes | Workflow input parameters (content-gated) | -| `dify.workflow.outputs` | string/JSON | Yes | Workflow output results (content-gated) | -| `dify.workflow.query` | string | No | User query text (content-gated, for chat workflows) | +| `dify.user.id` | string | No | User identifier | +| `gen_ai.usage.total_tokens` | int | No | Total tokens (sum of all nodes) | +| `dify.workflow.version` | string | Yes | Workflow version | +| `dify.workflow.inputs` | string/JSON | Yes | Input parameters (content-gated) | +| `dify.workflow.outputs` | string/JSON | Yes | Output results (content-gated) | +| `dify.workflow.query` | string | No | User query text (content-gated) | -**Common Log Attributes:** +**Event attributes:** - `dify.event.name`: `"dify.workflow.run"` - `dify.event.signal`: `"span_detail"` -- `trace_id`: Correlated OTEL trace ID (32-char hex) -- `span_id`: Correlated OTEL span ID (16-char hex) -- `tenant_id`: Tenant identifier -- `user_id`: User identifier (when available) +- `trace_id`, `span_id`, `tenant_id`, `user_id` -#### 8.1.2 `dify.node.execution` and `dify.node.execution.draft` Companion Logs +#### `dify.node.execution` and `dify.node.execution.draft` Companion Logs -**Signal Type:** `span_detail` - -These logs accompany node execution spans and include ALL span attributes PLUS additional detail attributes. - -**Additional Attributes (beyond span attributes):** +**Common attributes:** All span attributes (see Traces section) plus: | Additional Attribute | Type | Always Present | Description | |---------------------|------|----------------|-------------| -| `dify.user.id` | string | No | User identifier (when available) | -| `gen_ai.provider.name` | string | No | LLM provider name (for LLM nodes only) | -| `gen_ai.request.model` | string | No | LLM model name (for LLM nodes only) | -| `gen_ai.usage.input_tokens` | int | No | Input tokens for this node (for LLM nodes only) | -| `gen_ai.usage.output_tokens` | int | No | Output tokens for this node (for LLM nodes only) | -| `gen_ai.usage.total_tokens` | int | No | Total tokens for this node (for LLM nodes only) | -| `dify.node.total_price` | float | No | Cost in currency units (for LLM nodes only) | -| `dify.node.currency` | string | No | Currency code (for LLM nodes only) | -| `dify.node.plugin_name` | string | No | Plugin name (for tool/knowledge-retrieval nodes only) | -| `dify.node.plugin_id` | string | No | Plugin identifier (for tool/knowledge-retrieval nodes only) | -| `dify.dataset.id` | string | No | Dataset identifier (for knowledge-retrieval nodes only) | -| `dify.dataset.name` | string | No | Dataset name (for knowledge-retrieval nodes only) | -| `dify.node.inputs` | string/JSON | Yes | Node input data (content-gated) | -| `dify.node.outputs` | string/JSON | Yes | Node output data (content-gated) | -| `dify.node.process_data` | string/JSON | No | Internal processing data (content-gated) | +| `dify.user.id` | string | No | User identifier | +| `gen_ai.provider.name` | string | No | LLM provider (LLM nodes only) | +| `gen_ai.request.model` | string | No | LLM model (LLM nodes only) | +| `gen_ai.usage.input_tokens` | int | No | Input tokens (LLM nodes only) | +| `gen_ai.usage.output_tokens` | int | No | Output tokens (LLM nodes only) | +| `gen_ai.usage.total_tokens` | int | No | Total tokens (LLM nodes only) | +| `dify.node.total_price` | float | No | Cost (LLM nodes only) | +| `dify.node.currency` | string | No | Currency code (LLM nodes only) | +| `dify.node.plugin_name` | string | No | Plugin name (tool/knowledge nodes) | +| `dify.node.plugin_id` | string | No | Plugin ID (tool/knowledge nodes) | +| `dify.dataset.id` | string | No | Dataset ID (knowledge nodes) | +| `dify.dataset.name` | string | No | Dataset name (knowledge nodes) | +| `dify.node.inputs` | string/JSON | Yes | Node inputs (content-gated) | +| `dify.node.outputs` | string/JSON | Yes | Node outputs (content-gated) | +| `dify.node.process_data` | string/JSON | No | Processing data (content-gated) | -**Common Log Attributes:** +**Event attributes:** - `dify.event.name`: `"dify.node.execution"` or `"dify.node.execution.draft"` - `dify.event.signal`: `"span_detail"` -- `trace_id`: Correlated OTEL trace ID (32-char hex) -- `span_id`: Correlated OTEL span ID (16-char hex) -- `tenant_id`: Tenant identifier -- `user_id`: User identifier (when available) +- `trace_id`, `span_id`, `tenant_id`, `user_id` -### 8.2 Standalone Logs +### Standalone Logs -These logs represent events that do not have a structural span. They are emitted as `metric_only` signals. +Logs without structural spans. Signal type: `metric_only` -#### 8.2.1 `dify.message.run` +#### `dify.message.run` -**Signal Type:** `metric_only` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.message.run"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID (32-char hex) | +| `span_id` | string | OTEL span ID (16-char hex) | +| `tenant_id` | string | Tenant identifier | +| `user_id` | string | User identifier (optional) | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.conversation.id` | string | Conversation ID (optional) | +| `dify.workflow.run_id` | string | Workflow run ID (optional) | +| `dify.invoke_from` | string | `service-api`, `web-app`, `debugger`, `explore` | +| `gen_ai.provider.name` | string | LLM provider | +| `gen_ai.request.model` | string | LLM model | +| `gen_ai.usage.input_tokens` | int | Input tokens | +| `gen_ai.usage.output_tokens` | int | Output tokens | +| `gen_ai.usage.total_tokens` | int | Total tokens | +| `dify.message.status` | string | `succeeded`, `failed` | +| `dify.message.error` | string | Error message (if failed) | +| `dify.message.duration` | float | Duration (seconds) | +| `dify.message.time_to_first_token` | float | TTFT (seconds) | +| `dify.message.inputs` | string/JSON | Inputs (content-gated) | +| `dify.message.outputs` | string/JSON | Outputs (content-gated) | -Emitted for each message execution (LLM interaction). +#### `dify.tool.execution` -**Attributes:** -- `dify.event.name`: `"dify.message.run"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID (32-char hex) -- `span_id`: OTEL span ID (16-char hex) -- `tenant_id`: Tenant identifier -- `user_id`: User identifier (when available) -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.conversation.id`: Conversation identifier (when available) -- `dify.workflow.run_id`: Workflow run identifier (when part of workflow) -- `dify.invoke_from`: Source of invocation (`"service-api"`, `"web-app"`, `"debugger"`, `"explore"`) -- `gen_ai.provider.name`: LLM provider name -- `gen_ai.request.model`: LLM model name -- `gen_ai.usage.input_tokens`: Input tokens consumed -- `gen_ai.usage.output_tokens`: Output tokens generated -- `gen_ai.usage.total_tokens`: Total tokens consumed -- `dify.message.status`: Execution status (`"succeeded"`, `"failed"`) -- `dify.message.error`: Error message (when status is `"failed"`) -- `dify.message.duration`: Execution duration in seconds -- `dify.message.time_to_first_token`: Time to first token in seconds -- `dify.message.inputs`: Message inputs (content-gated) -- `dify.message.outputs`: Message outputs (content-gated) +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.tool.execution"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.tool.name` | string | Tool name | +| `dify.tool.duration` | float | Duration (seconds) | +| `dify.tool.status` | string | `succeeded`, `failed` | +| `dify.tool.error` | string | Error message (if failed) | +| `dify.tool.inputs` | string/JSON | Inputs (content-gated) | +| `dify.tool.outputs` | string/JSON | Outputs (content-gated) | +| `dify.tool.parameters` | string/JSON | Parameters (content-gated) | +| `dify.tool.config` | string/JSON | Configuration (content-gated) | -**Example:** -```json -{ - "trace_id": "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6", - "span_id": "a1b2c3d4e5f6g7h8", - "attributes": { - "dify.event.name": "dify.message.run", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "user_id": "660e8400-e29b-41d4-a716-446655440001", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.conversation.id": "990e8400-e29b-41d4-a716-446655440004", - "dify.invoke_from": "web-app", - "gen_ai.provider.name": "openai", - "gen_ai.request.model": "gpt-4", - "gen_ai.usage.input_tokens": 120, - "gen_ai.usage.output_tokens": 85, - "gen_ai.usage.total_tokens": 205, - "dify.message.status": "succeeded", - "dify.message.duration": 2.45, - "dify.message.time_to_first_token": 0.32, - "dify.message.inputs": "{\"query\": \"What is the weather?\"}", - "dify.message.outputs": "{\"answer\": \"The weather is sunny.\"}" - } -} -``` +#### `dify.moderation.check` -#### 8.2.2 `dify.tool.execution` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.moderation.check"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.moderation.type` | string | `input`, `output` | +| `dify.moderation.action` | string | `pass`, `block`, `flag` | +| `dify.moderation.flagged` | boolean | Whether flagged | +| `dify.moderation.categories` | JSON array | Flagged categories | +| `dify.moderation.query` | string | Content (content-gated) | -**Signal Type:** `metric_only` +#### `dify.suggested_question.generation` -Emitted for each tool invocation. +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.suggested_question.generation"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.suggested_question.count` | int | Number of questions | +| `dify.suggested_question.duration` | float | Duration (seconds) | +| `dify.suggested_question.status` | string | `succeeded`, `failed` | +| `dify.suggested_question.error` | string | Error message (if failed) | +| `dify.suggested_question.questions` | JSON array | Questions (content-gated) | -**Attributes:** -- `dify.event.name`: `"dify.tool.execution"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.tool.name`: Tool name -- `dify.tool.duration`: Execution duration in seconds -- `dify.tool.status`: Execution status (`"succeeded"`, `"failed"`) -- `dify.tool.error`: Error message (when failed) -- `dify.tool.inputs`: Tool inputs (content-gated) -- `dify.tool.outputs`: Tool outputs (content-gated) -- `dify.tool.parameters`: Tool parameters (content-gated) -- `dify.tool.config`: Tool configuration (content-gated) +#### `dify.dataset.retrieval` -**Example:** -```json -{ - "trace_id": "b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7", - "span_id": "b2c3d4e5f6g7h8i9", - "attributes": { - "dify.event.name": "dify.tool.execution", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.tool.name": "weather_api", - "dify.tool.duration": 0.85, - "dify.tool.status": "succeeded", - "dify.tool.inputs": "{\"location\": \"San Francisco\"}", - "dify.tool.outputs": "{\"temperature\": 72, \"condition\": \"sunny\"}", - "dify.tool.parameters": "{\"api_key\": \"***\"}", - "dify.tool.config": "{\"timeout\": 30}" - } -} -``` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.dataset.retrieval"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.dataset.id` | string | Dataset identifier | +| `dify.dataset.name` | string | Dataset name | +| `dify.retrieval.query` | string | Search query (content-gated) | +| `dify.retrieval.document_count` | int | Documents retrieved | +| `dify.retrieval.duration` | float | Duration (seconds) | +| `dify.retrieval.status` | string | `succeeded`, `failed` | +| `dify.retrieval.error` | string | Error message (if failed) | +| `dify.dataset.documents` | JSON array | Documents (content-gated) | -#### 8.2.3 `dify.moderation.check` +#### `dify.generate_name.execution` -**Signal Type:** `metric_only` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.generate_name.execution"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.conversation.id` | string | Conversation identifier | +| `dify.generate_name.duration` | float | Duration (seconds) | +| `dify.generate_name.status` | string | `succeeded`, `failed` | +| `dify.generate_name.error` | string | Error message (if failed) | +| `dify.generate_name.inputs` | string/JSON | Inputs (content-gated) | +| `dify.generate_name.outputs` | string | Generated name (content-gated) | -Emitted for content moderation checks. +#### `dify.prompt_generation.execution` -**Attributes:** -- `dify.event.name`: `"dify.moderation.check"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.moderation.type`: Moderation type (`"input"`, `"output"`) -- `dify.moderation.action`: Action taken (`"pass"`, `"block"`, `"flag"`) -- `dify.moderation.flagged`: Whether content was flagged (boolean) -- `dify.moderation.categories`: Flagged categories (JSON array) -- `dify.moderation.query`: Content being moderated (content-gated) +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.prompt_generation.execution"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.prompt_generation.operation_type` | string | Operation type (see appendix) | +| `gen_ai.provider.name` | string | LLM provider | +| `gen_ai.request.model` | string | LLM model | +| `gen_ai.usage.input_tokens` | int | Input tokens | +| `gen_ai.usage.output_tokens` | int | Output tokens | +| `gen_ai.usage.total_tokens` | int | Total tokens | +| `dify.prompt_generation.duration` | float | Duration (seconds) | +| `dify.prompt_generation.status` | string | `succeeded`, `failed` | +| `dify.prompt_generation.error` | string | Error message (if failed) | +| `dify.prompt_generation.instruction` | string | Instruction (content-gated) | +| `dify.prompt_generation.output` | string/JSON | Output (content-gated) | -**Example:** -```json -{ - "trace_id": "c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8", - "span_id": "c3d4e5f6g7h8i9j0", - "attributes": { - "dify.event.name": "dify.moderation.check", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.moderation.type": "input", - "dify.moderation.action": "pass", - "dify.moderation.flagged": false, - "dify.moderation.categories": "[]", - "dify.moderation.query": "What is the weather?" - } -} -``` +#### `dify.app.created` -#### 8.2.4 `dify.suggested_question.generation` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.app.created"` | +| `dify.event.signal` | string | `"metric_only"` | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.app.mode` | string | `chat`, `completion`, `agent-chat`, `workflow` | +| `dify.app.created_at` | string | Timestamp (ISO 8601) | -**Signal Type:** `metric_only` +#### `dify.app.updated` -Emitted when suggested questions are generated. +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.app.updated"` | +| `dify.event.signal` | string | `"metric_only"` | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.app.updated_at` | string | Timestamp (ISO 8601) | -**Attributes:** -- `dify.event.name`: `"dify.suggested_question.generation"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.suggested_question.count`: Number of questions generated -- `dify.suggested_question.duration`: Generation duration in seconds -- `dify.suggested_question.status`: Generation status (`"succeeded"`, `"failed"`) -- `dify.suggested_question.error`: Error message (when failed) -- `dify.suggested_question.questions`: Generated questions (content-gated, JSON array) +#### `dify.app.deleted` -**Example:** -```json -{ - "trace_id": "d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9", - "span_id": "d4e5f6g7h8i9j0k1", - "attributes": { - "dify.event.name": "dify.suggested_question.generation", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.suggested_question.count": 3, - "dify.suggested_question.duration": 1.2, - "dify.suggested_question.status": "succeeded", - "dify.suggested_question.questions": "[\"What about tomorrow?\", \"How about next week?\", \"Is it raining?\"]" - } -} -``` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.app.deleted"` | +| `dify.event.signal` | string | `"metric_only"` | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.app.deleted_at` | string | Timestamp (ISO 8601) | -#### 8.2.5 `dify.dataset.retrieval` +#### `dify.feedback.created` -**Signal Type:** `metric_only` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.feedback.created"` | +| `dify.event.signal` | string | `"metric_only"` | +| `trace_id` | string | OTEL trace ID | +| `span_id` | string | OTEL span ID | +| `tenant_id` | string | Tenant identifier | +| `dify.app_id` | string | Application identifier | +| `dify.message.id` | string | Message identifier | +| `dify.feedback.rating` | string | `like`, `dislike`, `null` | +| `dify.feedback.content` | string | Feedback text (content-gated) | +| `dify.feedback.created_at` | string | Timestamp (ISO 8601) | -Emitted for knowledge base retrieval operations. +#### `dify.telemetry.rehydration_failed` -**Attributes:** -- `dify.event.name`: `"dify.dataset.retrieval"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.dataset.id`: Dataset identifier -- `dify.dataset.name`: Dataset name -- `dify.retrieval.query`: Search query (content-gated) -- `dify.retrieval.document_count`: Number of documents retrieved -- `dify.retrieval.duration`: Retrieval duration in seconds -- `dify.retrieval.status`: Retrieval status (`"succeeded"`, `"failed"`) -- `dify.retrieval.error`: Error message (when failed) -- `dify.dataset.documents`: Retrieved documents (content-gated, JSON array) +Diagnostic event for telemetry system health monitoring. -**Example:** -```json -{ - "trace_id": "e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0", - "span_id": "e5f6g7h8i9j0k1l2", - "attributes": { - "dify.event.name": "dify.dataset.retrieval", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.dataset.id": "aa0e8400-e29b-41d4-a716-446655440005", - "dify.dataset.name": "Product Documentation", - "dify.retrieval.query": "installation guide", - "dify.retrieval.document_count": 5, - "dify.retrieval.duration": 0.45, - "dify.retrieval.status": "succeeded", - "dify.dataset.documents": "[{\"id\": \"doc1\", \"score\": 0.95}, {\"id\": \"doc2\", \"score\": 0.87}]" - } -} -``` +| Attribute | Type | Description | +|-----------|------|-------------| +| `dify.event.name` | string | `"dify.telemetry.rehydration_failed"` | +| `dify.event.signal` | string | `"metric_only"` | +| `tenant_id` | string | Tenant identifier | +| `dify.telemetry.error` | string | Error message | +| `dify.telemetry.payload_type` | string | Payload type (see appendix) | +| `dify.telemetry.correlation_id` | string | Correlation ID | -#### 8.2.6 `dify.generate_name.execution` +## Content-Gated Attributes -**Signal Type:** `metric_only` - -Emitted when conversation names are auto-generated. - -**Attributes:** -- `dify.event.name`: `"dify.generate_name.execution"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.conversation.id`: Conversation identifier -- `dify.generate_name.duration`: Generation duration in seconds -- `dify.generate_name.status`: Generation status (`"succeeded"`, `"failed"`) -- `dify.generate_name.error`: Error message (when failed) -- `dify.generate_name.inputs`: Generation inputs (content-gated) -- `dify.generate_name.outputs`: Generated name (content-gated) - -**Example:** -```json -{ - "trace_id": "f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1", - "span_id": "f6g7h8i9j0k1l2m3", - "attributes": { - "dify.event.name": "dify.generate_name.execution", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.conversation.id": "990e8400-e29b-41d4-a716-446655440004", - "dify.generate_name.duration": 0.75, - "dify.generate_name.status": "succeeded", - "dify.generate_name.inputs": "{\"first_message\": \"What is the weather?\"}", - "dify.generate_name.outputs": "Weather Inquiry" - } -} -``` - -#### 8.2.7 `dify.prompt_generation.execution` - -**Signal Type:** `metric_only` - -Emitted for prompt engineering operations. - -**Attributes:** -- `dify.event.name`: `"dify.prompt_generation.execution"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.prompt_generation.operation_type`: Operation type (`"rule_generate"`, `"code_generate"`, `"structured_output"`, `"instruction_modify"`) -- `gen_ai.provider.name`: LLM provider name -- `gen_ai.request.model`: LLM model name -- `gen_ai.usage.input_tokens`: Input tokens consumed -- `gen_ai.usage.output_tokens`: Output tokens generated -- `gen_ai.usage.total_tokens`: Total tokens consumed -- `dify.prompt_generation.duration`: Generation duration in seconds -- `dify.prompt_generation.status`: Generation status (`"succeeded"`, `"failed"`) -- `dify.prompt_generation.error`: Error message (when failed) -- `dify.prompt_generation.instruction`: Prompt instruction (content-gated) -- `dify.prompt_generation.output`: Generated output (content-gated) - -**Example:** -```json -{ - "trace_id": "g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2", - "span_id": "g7h8i9j0k1l2m3n4", - "attributes": { - "dify.event.name": "dify.prompt_generation.execution", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.prompt_generation.operation_type": "rule_generate", - "gen_ai.provider.name": "openai", - "gen_ai.request.model": "gpt-4", - "gen_ai.usage.input_tokens": 50, - "gen_ai.usage.output_tokens": 30, - "gen_ai.usage.total_tokens": 80, - "dify.prompt_generation.duration": 1.1, - "dify.prompt_generation.status": "succeeded", - "dify.prompt_generation.instruction": "Generate validation rules", - "dify.prompt_generation.output": "{\"rules\": [\"check_length\", \"validate_format\"]}" - } -} -``` - -#### 8.2.8 `dify.app.created` - -**Signal Type:** `metric_only` - -Emitted when an application is created. - -**Attributes:** -- `dify.event.name`: `"dify.app.created"` -- `dify.event.signal`: `"metric_only"` -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.app.mode`: Application mode (`"chat"`, `"completion"`, `"agent-chat"`, `"workflow"`) -- `dify.app.created_at`: Creation timestamp (ISO 8601) - -**Example:** -```json -{ - "attributes": { - "dify.event.name": "dify.app.created", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.app.mode": "workflow", - "dify.app.created_at": "2026-02-10T19:30:00Z" - } -} -``` - -#### 8.2.9 `dify.app.updated` - -**Signal Type:** `metric_only` - -Emitted when an application is updated. - -**Attributes:** -- `dify.event.name`: `"dify.app.updated"` -- `dify.event.signal`: `"metric_only"` -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.app.updated_at`: Update timestamp (ISO 8601) - -**Example:** -```json -{ - "attributes": { - "dify.event.name": "dify.app.updated", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.app.updated_at": "2026-02-10T20:15:00Z" - } -} -``` - -#### 8.2.10 `dify.app.deleted` - -**Signal Type:** `metric_only` - -Emitted when an application is deleted. - -**Attributes:** -- `dify.event.name`: `"dify.app.deleted"` -- `dify.event.signal`: `"metric_only"` -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.app.deleted_at`: Deletion timestamp (ISO 8601) - -**Example:** -```json -{ - "attributes": { - "dify.event.name": "dify.app.deleted", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.app.deleted_at": "2026-02-10T21:00:00Z" - } -} -``` - -#### 8.2.11 `dify.feedback.created` - -**Signal Type:** `metric_only` - -Emitted when user feedback is submitted. - -**Attributes:** -- `dify.event.name`: `"dify.feedback.created"` -- `dify.event.signal`: `"metric_only"` -- `trace_id`: OTEL trace ID -- `span_id`: OTEL span ID -- `tenant_id`: Tenant identifier -- `dify.app_id`: Application identifier -- `dify.message.id`: Message identifier -- `dify.feedback.rating`: Rating (`"like"`, `"dislike"`, `null`) -- `dify.feedback.content`: Feedback text (content-gated, omitted when gating enabled) -- `dify.feedback.created_at`: Feedback timestamp (ISO 8601) - -**Example:** -```json -{ - "trace_id": "h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3", - "span_id": "h8i9j0k1l2m3n4o5", - "attributes": { - "dify.event.name": "dify.feedback.created", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.app_id": "770e8400-e29b-41d4-a716-446655440002", - "dify.message.id": "880e8400-e29b-41d4-a716-446655440003", - "dify.feedback.rating": "like", - "dify.feedback.content": "Very helpful response!", - "dify.feedback.created_at": "2026-02-10T19:45:00Z" - } -} -``` - -#### 8.2.12 `dify.telemetry.rehydration_failed` - -**Signal Type:** `metric_only` (Diagnostic) - -Emitted when telemetry payload rehydration fails. This is a diagnostic event for monitoring telemetry system health. - -**Attributes:** -- `dify.event.name`: `"dify.telemetry.rehydration_failed"` -- `dify.event.signal`: `"metric_only"` -- `tenant_id`: Tenant identifier -- `dify.telemetry.error`: Error message -- `dify.telemetry.payload_type`: Type of payload that failed (`"workflow"`, `"node"`, `"message"`, etc.) -- `dify.telemetry.correlation_id`: Correlation ID of the failed payload - -**Example:** -```json -{ - "attributes": { - "dify.event.name": "dify.telemetry.rehydration_failed", - "dify.event.signal": "metric_only", - "tenant_id": "550e8400-e29b-41d4-a716-446655440000", - "dify.telemetry.error": "Workflow run not found in database", - "dify.telemetry.payload_type": "workflow", - "dify.telemetry.correlation_id": "bb0e8400-e29b-41d4-a716-446655440006" - } -} -``` - -## 9. Content Attributes - -When `ENTERPRISE_INCLUDE_CONTENT` is set to `false`, the following attributes are replaced with a reference string (e.g., `ref:workflow_run_id=...`) to prevent sensitive data leakage to the OTEL collector. +When `ENTERPRISE_INCLUDE_CONTENT=false`, these attributes are replaced with reference strings (`ref:{id_type}={uuid}`). | Attribute | Signal | |-----------|--------| @@ -809,103 +421,28 @@ When `ENTERPRISE_INCLUDE_CONTENT` is set to `false`, the following attributes ar | `dify.prompt_generation.output` | `dify.prompt_generation.execution` | | `dify.feedback.content` | `dify.feedback.created` | -### Content Gating Behavior +## Appendix -When `ENTERPRISE_INCLUDE_CONTENT=true` (default), content attributes contain the actual data: +### Operation Types -```json -{ - "dify.workflow.inputs": "{\"query\": \"What is the weather?\", \"location\": \"San Francisco\"}", - "dify.workflow.outputs": "{\"answer\": \"The weather in San Francisco is sunny, 72°F.\"}", - "dify.workflow.query": "What is the weather?" -} -``` +- `workflow`, `node_execution`, `message`, `rule_generate`, `code_generate`, `structured_output`, `instruction_modify` -When `ENTERPRISE_INCLUDE_CONTENT=false`, content attributes are replaced with reference strings: +### Node Types -```json -{ - "dify.workflow.inputs": "ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000", - "dify.workflow.outputs": "ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000", - "dify.workflow.query": "ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000" -} -``` - -### Reference String Format - -Reference strings follow the pattern `ref:{id_type}={uuid}`, where: - -- `{id_type}` identifies the entity type -- `{uuid}` is the entity's unique identifier - -**Examples:** - -``` -ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000 -ref:node_execution_id=660e8400-e29b-41d4-a716-446655440001 -ref:message_id=770e8400-e29b-41d4-a716-446655440002 -ref:conversation_id=880e8400-e29b-41d4-a716-446655440003 -ref:trace_id=990e8400-e29b-41d4-a716-446655440004 -``` - -**Usage:** To retrieve the actual content, query your Dify database using the provided UUID. For example, `ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000` means you can fetch the workflow run record from the `workflow_runs` table using that ID. - -**Special Case:** The `dify.feedback.content` attribute is **omitted entirely** (not replaced with a reference string) when content gating is enabled, as feedback text is not stored with a retrievable ID. - -## 10. Appendix - -### Operation Types (`operation_type`) -- `workflow` -- `node_execution` -- `message` -- `rule_generate` -- `code_generate` -- `structured_output` -- `instruction_modify` - -### Node Types (`node_type`) -- `start`, `end`, `answer`, `llm`, `knowledge-retrieval`, `knowledge-index`, `if-else`, `code`, `template-transform`, `question-classifier`, `http-request`, `tool`, `datasource`, `variable-aggregator`, `loop`, `iteration`, `parameter-extractor`, `assigner`, `document-extractor`, `list-operator`, `agent`, `trigger-webhook`, `trigger-schedule`, `trigger-plugin`, `human-input`. +- `start`, `end`, `answer`, `llm`, `knowledge-retrieval`, `knowledge-index`, `if-else`, `code`, `template-transform`, `question-classifier`, `http-request`, `tool`, `datasource`, `variable-aggregator`, `loop`, `iteration`, `parameter-extractor`, `assigner`, `document-extractor`, `list-operator`, `agent`, `trigger-webhook`, `trigger-schedule`, `trigger-plugin`, `human-input` ### Workflow Statuses -- `running`, `succeeded`, `failed`, `stopped`, `partial-succeeded`, `paused`. + +- `running`, `succeeded`, `failed`, `stopped`, `partial-succeeded`, `paused` + +### Payload Types + +- `workflow`, `node`, `message`, `tool`, `moderation`, `suggested_question`, `dataset_retrieval`, `generate_name`, `prompt_generation`, `app`, `feedback` ### Null Value Behavior -The handling of null/missing values differs between spans (traces) and structured logs: +**Spans:** Attributes with `null` values are omitted. -**In Spans (Traces):** -Attributes with `None`/`null` values are **omitted** from the exported span data. This follows OpenTelemetry conventions for efficient trace storage. +**Logs:** Attributes with `null` values appear as `null` in JSON. -Example - a workflow span without optional parent attributes: -```python -# These attributes are NOT included in the span: -# - dify.parent.trace_id (null) -# - dify.parent.workflow.run_id (null) -# - dify.parent.node.execution_id (null) -# - dify.parent.app.id (null) -``` - -**In Structured Logs:** -Attributes with `null` values appear as `null` in the JSON payload. - -Example: -```json -{ - "attributes": { - "dify.workflow.error": null, - "dify.conversation.id": null - } -} -``` - -**Conditional Attributes:** -Many attributes are only present under specific conditions: - -- **LLM-related attributes** (`gen_ai.provider.name`, `gen_ai.usage.total_tokens`, etc.) only appear for LLM nodes -- **Plugin attributes** (`dify.node.plugin_name`, `dify.node.plugin_id`) only appear for tool/knowledge-retrieval nodes -- **Dataset attributes** (`dify.dataset.id`, `dify.dataset.name`) only appear for knowledge-retrieval nodes -- **Error attributes** (`dify.workflow.error`, `dify.node.error`) only appear when status is `"failed"` -- **Parent attributes** (`dify.parent.*`) only appear for nested/sub-workflow executions - -**Content-Gated Attributes:** -When `ENTERPRISE_INCLUDE_CONTENT=false`, content attributes are replaced with reference strings (not set to `null`). See Section 9 for details. +**Content-Gated:** Replaced with reference strings, not set to `null`. diff --git a/api/enterprise/telemetry/README.md b/api/enterprise/telemetry/README.md new file mode 100644 index 0000000000..2c8ca988e1 --- /dev/null +++ b/api/enterprise/telemetry/README.md @@ -0,0 +1,116 @@ +# Dify Enterprise Telemetry + +This document provides an overview of the Dify Enterprise OpenTelemetry (OTEL) exporter and how to configure it for integration with observability stacks like Prometheus, Grafana, Jaeger, or Honeycomb. + +## Overview + +Dify Enterprise uses a "slim span + rich companion log" architecture to provide high-fidelity observability without overwhelming trace storage. + +- **Traces (Spans)**: Capture the structure, identity, and timing of high-level operations (Workflows and Nodes). +- **Structured Logs**: Provide deep context (inputs, outputs, metadata) for every event, correlated to spans via `trace_id` and `span_id`. +- **Metrics**: Provide 100% accurate counters and histograms for usage, performance, and error tracking. + +### Signal Architecture + +```mermaid +graph TD + A[Workflow Run] -->|Span| B(dify.workflow.run) + A -->|Log| C(dify.workflow.run detail) + B ---|trace_id| C + + D[Node Execution] -->|Span| E(dify.node.execution) + D -->|Log| F(dify.node.execution detail) + E ---|span_id| F + + G[Message/Tool/etc] -->|Log| H(dify.* event) + G -->|Metric| I(dify.* counter/histogram) +``` + +## Configuration + +The Enterprise OTEL exporter is configured via environment variables. + +| Variable | Description | Default | +|----------|-------------|---------| +| `ENTERPRISE_ENABLED` | Master switch for all enterprise features. | `false` | +| `ENTERPRISE_TELEMETRY_ENABLED` | Master switch for enterprise telemetry. | `false` | +| `ENTERPRISE_OTLP_ENDPOINT` | OTLP collector endpoint (e.g., `http://otel-collector:4318`). | - | +| `ENTERPRISE_OTLP_HEADERS` | Custom headers for OTLP requests (e.g., `x-scope-orgid=tenant1`). | - | +| `ENTERPRISE_OTLP_PROTOCOL` | OTLP transport protocol (`http` or `grpc`). | `http` | +| `ENTERPRISE_OTLP_API_KEY` | Bearer token for authentication. | - | +| `ENTERPRISE_INCLUDE_CONTENT` | Whether to include sensitive content (inputs/outputs) in logs. | `true` | +| `ENTERPRISE_SERVICE_NAME` | Service name reported to OTEL. | `dify` | +| `ENTERPRISE_OTEL_SAMPLING_RATE` | Sampling rate for traces (0.0 to 1.0). Metrics are always 100%. | `1.0` | + +## Correlation Model + +Dify uses deterministic ID generation to ensure signals are correlated across different services and asynchronous tasks. + +### ID Generation Rules +- `trace_id`: Derived from the correlation ID (workflow_run_id or node_execution_id for drafts) using `int(UUID(correlation_id))` +- `span_id`: Derived from the source ID using `SHA256(source_id)[:8]` + +### Scenario A: Simple Workflow +A single workflow run with multiple nodes. All spans and logs share the same `trace_id` (derived from `workflow_run_id`). + +``` +trace_id = UUID(workflow_run_id) +├── [root span] dify.workflow.run (span_id = hash(workflow_run_id)) +│ ├── [child] dify.node.execution - "Start" (span_id = hash(node_exec_id_1)) +│ ├── [child] dify.node.execution - "LLM" (span_id = hash(node_exec_id_2)) +│ └── [child] dify.node.execution - "End" (span_id = hash(node_exec_id_3)) +``` + +### Scenario B: Nested Sub-Workflow +A workflow calling another workflow via a Tool or Sub-workflow node. The child workflow's spans are linked to the parent via `parent_span_id`. Both workflows share the same trace_id. + +``` +trace_id = UUID(outer_workflow_run_id) ← shared across both workflows +├── [root] dify.workflow.run (outer) (span_id = hash(outer_workflow_run_id)) +│ ├── dify.node.execution - "Start Node" +│ ├── dify.node.execution - "Tool Node" (triggers sub-workflow) +│ │ └── [child] dify.workflow.run (inner) (span_id = hash(inner_workflow_run_id)) +│ │ ├── dify.node.execution - "Inner Start" +│ │ └── dify.node.execution - "Inner End" +│ └── dify.node.execution - "End Node" +``` + +**Key attributes for nested workflows:** +- Inner workflow's `dify.parent.trace_id` = outer `workflow_run_id` +- Inner workflow's `dify.parent.node.execution_id` = tool node's `execution_id` +- Inner workflow's `dify.parent.workflow.run_id` = outer `workflow_run_id` +- Inner workflow's `dify.parent.app.id` = outer `app_id` + +### Scenario C: Draft Node Execution +A single node run in isolation (debugger/preview mode). It creates its own trace where the node span is the root. + +``` +trace_id = UUID(node_execution_id) ← own trace, NOT part of any workflow +└── dify.node.execution.draft (span_id = hash(node_execution_id)) +``` + +**Key difference:** Draft executions use `node_execution_id` as the correlation_id, so they are NOT children of any workflow trace. + +## Content Gating + +When `ENTERPRISE_INCLUDE_CONTENT` is set to `false`, sensitive content attributes (inputs, outputs, queries) are replaced with reference strings (e.g., `ref:workflow_run_id=...`) to prevent data leakage to the OTEL collector. + +**Reference String Format:** + +``` +ref:{id_type}={uuid} +``` + +**Examples:** + +``` +ref:workflow_run_id=550e8400-e29b-41d4-a716-446655440000 +ref:node_execution_id=660e8400-e29b-41d4-a716-446655440001 +ref:message_id=770e8400-e29b-41d4-a716-446655440002 +``` + +To retrieve actual content when gating is enabled, query the Dify database using the provided UUID. + +## Reference + +For a complete list of telemetry signals, attributes, and data structures, see [DATA_DICTIONARY.md](./DATA_DICTIONARY.md).