---
title: "AgentCore Observability — Transaction Search, OTEL spans, and the GenAI dashboard you didn't have to build"
date: 2026-05-31
service: "Amazon Bedrock AgentCore"
component: "Observability"
tags: [agentcore, observability, opentelemetry, otel, adot, cloudwatch, transaction-search, spans, traces, gen-ai-observability, sessions, baggage, w3c-traceparent]
source: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html
verified_on: 2026-05-31
url: https://vanemmerik.ai/aws-ai/2026-05-31.html
---

# AWS Bedrock & AgentCore · Tip of the Day · 2026-05-31

## AgentCore Observability — Transaction Search, OTEL spans, and the GenAI dashboard you didn't have to build

Yesterday we plugged every Lambda, REST API, and external MCP
server into the agent via Gateway. Today's question is the one
that turns up the morning after that ships: *where do the traces
go?* AgentCore Observability is the **OpenTelemetry-native
monitoring layer** that captures session metrics, structured
spans, and application logs for every AgentCore resource — and
surfaces them in the CloudWatch **generative AI observability**
page without you wiring up a collector.

    # one-time per account/Region
    aws xray update-trace-segment-destination --destination CloudWatchLogs
    # then, for any Runtime-hosted agent
    agentcore deploy   # ADOT auto-instrumentation is on by default

≈ 10 min read · Bedrock AgentCore · Observability

---

## 01 · The shape of the system

From the
[observability overview](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html):
*"Amazon Bedrock AgentCore provides a number of built-in metrics
to monitor the performance of resources for the AgentCore
runtime, memory, gateway, built-in tools, and identity resource
types."* That sentence is doing a lot of work. The shape is:

- **Service-provided telemetry** is emitted by AgentCore itself
  for every resource — Runtime, Memory, Gateway, Built-in Tools,
  Identity, Payments, Policy.
- **Code-emitted telemetry** is whatever you (or your framework)
  instrument via the **AWS Distro for OpenTelemetry (ADOT)** SDK
  — GenAI semantic-convention spans for LLM calls, tool calls,
  and agent turns.
- Everything lands in **Amazon CloudWatch**: spans in the
  `aws/spans` log group, structured EMF metrics under the
  `bedrock-agentcore` namespace, application logs in per-resource
  log groups, and a tailored dashboard at the
  **generative AI observability** page in the CloudWatch console.

> **The shift.** You don't build a collector. You don't write the
> dashboard. AgentCore Runtime auto-instruments your container the
> moment you `agentcore deploy`, and the GenAI Observability page
> in CloudWatch already knows what an agent session, a model
> invocation, and a tool call look like — because the spans use
> the OpenTelemetry GenAI semantic conventions out of the box.

---

## 02 · The one-time setup: Transaction Search

Before any AgentCore span will become searchable, you have to
flip **CloudWatch Transaction Search** on in the account and
Region. From
[Get started with AgentCore Observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-get-started.html):
*"After you enable Transaction Search, it can take ten minutes
for spans to become available for search and analysis."* It's
three CLI calls:

    # 1. Allow X-Ray to write spans into the aws/spans log group.
    aws logs put-resource-policy \
      --policy-name MyResourcePolicy \
      --policy-document file://transaction-search-policy.json

    # 2. Route trace segments from X-Ray to CloudWatch Logs.
    aws xray update-trace-segment-destination --destination CloudWatchLogs

    # 3. (Optional) raise the indexed-span sampling percentage.
    aws xray update-indexing-rule \
      --name "Default" \
      --rule '{"Probabilistic": {"DesiredSamplingPercentage": 5}}'

A few things to know that the console-driven setup hides:

- The default indexing rule keeps **1% of spans at no cost** —
  raise the percentage only when you understand the per-span
  ingest cost.
- Spans land in the well-known log group **`aws/spans`**. The
  trace-search console reads from there directly.
- Console users get to the same setting under
  **Application Signals (APM) → Transaction search → Enable
  Transaction Search**.

---

## 03 · Runtime-hosted agents: instrumentation is automatic

For agents you deploy via the AgentCore CLI, you write zero OTEL
code. The runtime ships ADOT, runs `opentelemetry-instrument`
ahead of your entrypoint, and emits OTEL traces, EMF metrics, and
structured logs for you. From the get-started doc:

> *"When you deploy an agent using the AgentCore CLI, the runtime
> automatically instruments your agent with OpenTelemetry — no
> additional OTEL libraries or configuration are needed."*

Two CloudWatch log groups are created for each Runtime agent:

| Log group | What lands there |
| --- | --- |
| `/aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>/[runtime-logs]` | Standard stdout/stderr — `print`, `logging.info`, framework chatter. |
| `/aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>/otel-rt-logs` | OTEL structured logs — execution details, error tracking, correlation IDs that link to span IDs. |

The Runtime span is documented in
[AgentCore generated runtime observability data](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-runtime-metrics.html).
It's a single operation:

- **Span name:** `InvokeAgentRuntime`
- **Attributes:** `aws.operation.name`, `aws.resource.arn`,
  `aws.request_id`, `aws.agent.id`, `aws.endpoint.name`,
  `aws.account.id`, `session.id`, `latency_ms`, `error_type`,
  `aws.resource.type`, `aws.xray.origin`, `aws.region`.
- **`error_type`** is one of `throttle`, `system`, or `user`, and
  is only present on failed invocations.

That gives you a usable trace tree on day one even before you add
your own framework spans.

---

## 04 · Non-runtime agents: the OTEL env vars

For agents hosted elsewhere — ECS, EKS, Lambda, anything outside
AgentCore Runtime — you get the same dashboards by adding the
ADOT distro to your image and exporting six environment
variables. From the
[configure doc](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html):

    AGENT_OBSERVABILITY_ENABLED=true
    OTEL_PYTHON_DISTRO=aws_distro
    OTEL_PYTHON_CONFIGURATOR=aws_configurator
    OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
    OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/<agent-id>,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore
    OTEL_RESOURCE_ATTRIBUTES=service.name=<agent-name>,aws.log.group.names=/aws/bedrock-agentcore/runtimes/<agent-id>,cloud.resource_id=<AgentEndpointArn:AgentEndpointName>

Three things that catch people:

- **`aws-opentelemetry-distro>=0.10.0`** is the minimum version
  pinned in the doc — older releases predate the GenAI
  conventions AgentCore expects.
- For frameworks other than Strands, LangChain, or CrewAI, you
  also need a GenAI instrumentation library —
  **OpenInference**, **Openllmetry**, **OpenLit**, or
  **Traceloop** are the four the docs explicitly support.
- Session correlation is opt-in. The recommended snippet:

      from opentelemetry import baggage, context
      ctx = baggage.set_baggage("session.id", session_id)
      context.attach(ctx)

  Without this, the dashboard groups spans by trace ID but not by
  session ID — and the per-session views go empty.

---

## 05 · The six headers that change the trace

Whether you invoke through `boto3` or curl, AgentCore Runtime
honours six HTTP headers that flow straight into the OTEL pipeline
and the resulting spans. From the
[enhanced runtime observability table](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html#observability-configure-invoke):

| Header | Purpose |
| --- | --- |
| `X-Amzn-Trace-Id` | X-Ray trace ID. Sets root/parent IDs and the sampling decision (`Sampled=1` = 100%). OTEL auto-generates one if absent. |
| `traceparent` | W3C trace context. Required for distributed traces with non-AWS systems. |
| `tracestate` | W3C vendor-specific trace state alongside `traceparent`. |
| `baggage` | W3C baggage — `userId=alice,serverRegion=us-east-1`. Propagated across spans for filtering. |
| `X-Amzn-Bedrock-AgentCore-Runtime-Session-Id` | AgentCore session correlation. Surfaces sessions in the GenAI Observability dashboard. |
| `mcp-session-id` | MCP session ID. Lets you trace one MCP session across multiple `tools/call` invocations. |

The practical implication: if you have a frontend already
instrumented with W3C `traceparent`, you can forward it on the
`invoke_agent_runtime` call and the resulting agent spans plug
straight into your existing distributed trace.

---

## 06 · Service-provided telemetry by resource type

Each AgentCore resource type ships a different default. From
[Amazon Bedrock AgentCore generated observability data](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-service-provided.html):

| Resource | Service-provided data | Available in GenAI Observability | Available in CloudWatch (Logs/metrics) |
| --- | --- | --- | --- |
| **Agent (Runtime)** | Metrics | Yes | Yes |
| **Memory** | Metrics, Spans\*, Logs\* | No | Yes |
| **Payments** | Metrics, Spans, Logs | No | Yes |
| **Gateway** | Metrics | No | Yes |
| **Built-in Tools** | Metrics | No | Yes |
| **Policy** | Metrics, Spans\*\* | Yes | Yes |

\* Memory spans and logs are off by default — you turn them on
per memory resource via the `Tracing` and `Log delivery` panes.
\*\* Policy observability surfaces under the Gateway tab on the
GenAI Observability page.

The Runtime is the only resource whose default telemetry shows up
in the GenAI Observability dashboard with no extra configuration.
For Memory, Gateway, Identity, and Built-in Tools you have to
attach a log destination (CloudWatch Logs, S3, or Firehose) and,
where applicable, toggle the **Tracing** widget — once you do,
spans flow into the same `aws/spans` log group as everything else.

---

## 07 · Limits and gotchas worth knowing

Distilled from the get-started, configure, and runtime-metrics
pages:

- **10 minutes after enabling Transaction Search before spans
  appear.** Plan the runbook around it.
- **Default span sampling is 1%.** Production debugging usually
  needs 5–10%; pricing matters above that.
- **Resource-usage telemetry can lag by up to 60 minutes** and
  precision differs across metrics — don't alert on
  `CPUUsed-vCPUHours` for real-time scaling decisions.
- **`USAGE_LOGS` are 1-second granularity** and contain
  `agent.runtime.vcpu.hours.used` / `agent.runtime.memory.gb_hours.used`
  per session. They're vended; you opt them in per agent.
- **Memory spans require enablement.** A memory resource with the
  Tracing toggle off will surface zero spans even with
  Transaction Search on.
- **WebSocket agents emit extra metrics** —
  `ActiveStreamingConnections`, `InboundStreamingBytesProcessed`,
  `OutboundStreamingBytesProcessed`. Sum over 1 minute, not
  average.
- **Custom dashboards live under namespace `bedrock-agentcore`.**
  ADOT writes EMF metrics there — that's the namespace to point
  CloudWatch alarms at.

---

## 08 · Try it in five minutes

A complete loop on a Runtime-hosted Strands agent:

    # 1. Enable Transaction Search once per account/Region.
    aws xray update-trace-segment-destination --destination CloudWatchLogs

    # 2. Scaffold and ship a tiny agent.
    npm install -g @aws/agentcore
    agentcore create --name ObservabilityDemo
    cd ObservabilityDemo
    agentcore deploy

    # 3. Invoke it with a session ID and a W3C traceparent so the
    #    span tree groups correctly.
    python - <<'PY'
    import boto3, json, uuid

    client = boto3.client("bedrock-agentcore")
    response = client.invoke_agent_runtime(
        agentRuntimeArn="arn:aws:bedrock-agentcore:us-east-1:111122223333:runtime/ObservabilityDemo-xxxxxx",
        runtimeSessionId=str(uuid.uuid4()),  # ≥ 33 chars; UUID4 is 36
        payload=json.dumps({"prompt": "What is 2 + 2?"}),
        qualifier="DEFAULT",
    )
    print(json.loads(response["response"].read()))
    PY

    # 4. Open the GenAI Observability page in CloudWatch.
    open "https://console.aws.amazon.com/cloudwatch/home#gen-ai-observability"

Within a minute you should see one **session**, one trace whose
root is `InvokeAgentRuntime`, child spans for each LLM call (with
GenAI semantic-convention attributes like `gen_ai.system`,
`gen_ai.request.model`, `gen_ai.usage.prompt_tokens`), and a
metrics row under the `bedrock-agentcore` namespace.

Tomorrow we'll look at **AgentCore Code Interpreter** — the
sandboxed code-execution tool and how its session model differs
from Runtime's.

---

**Verified against the official AWS docs on 2026-05-31.**
Sources:
<https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-get-started.html>,
<https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html>,
<https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-service-provided.html>,
<https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-runtime-metrics.html>,
<https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-view.html>,
<https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search.html>.

If the docs change, this lesson is a snapshot of that day — check
the sources for current behaviour.

---

> **This page — research, writing, verification, and deployment — was built by
> Claude Cowork.** No human touched the prose, the layout, or the upload
> pipeline. The lesson was generated this morning, cross-checked against the
> official AWS docs by an independent verification pass, and published
> to Cloudflare R2 on a schedule.
>
> A daily experiment by Monty van Emmerik · <https://vanemmerik.ai/>

— AWS Bedrock & AgentCore · Tip of the Day · No. 006 · vanemmerik.ai