AgentCore Observability, no collector required.
Yesterday we plugged every Lambda, REST API, and external MCP
server into the agent via Gateway. Today's question is the one
that turns up the morning after that ships: where do the
traces go? AgentCore Observability is the
OpenTelemetry-native monitoring layer that
captures session metrics, structured spans, and application logs
for every AgentCore resource — and surfaces them in the
CloudWatch generative AI observability page without
you wiring up a collector.
aws xray update-trace-segment-destination --destination CloudWatchLogs && agentcore deploy
01The shape of the system
From the observability overview: "Amazon Bedrock AgentCore provides a number of built-in metrics to monitor the performance of resources for the AgentCore runtime, memory, gateway, built-in tools, and identity resource types." That sentence is doing a lot of work. The shape is:
- Service-provided telemetry is emitted by AgentCore itself for every resource — Runtime, Memory, Gateway, Built-in Tools, Identity, Payments, Policy.
- Code-emitted telemetry is whatever you (or your framework) instrument via the AWS Distro for OpenTelemetry (ADOT) SDK — GenAI semantic-convention spans for LLM calls, tool calls, and agent turns.
- Everything lands in Amazon CloudWatch: spans in the
aws/spanslog group, structured EMF metrics under thebedrock-agentcorenamespace, application logs in per-resource log groups, and a tailored dashboard at the generative AI observability page in the CloudWatch console.
You don't build a collector. You don't write the dashboard. AgentCore Runtime auto-instruments your container the moment you agentcore deploy, and the GenAI Observability page in CloudWatch already knows what an agent session, a model invocation, and a tool call look like — because the spans use the OpenTelemetry GenAI semantic conventions out of the box.
02The one-time setup: Transaction Search
Before any AgentCore span will become searchable, you have to flip CloudWatch Transaction Search on in the account and Region. From Get started with AgentCore Observability: "After you enable Transaction Search, it can take ten minutes for spans to become available for search and analysis." It's three CLI calls:
# 1. Allow X-Ray to write spans into the aws/spans log group.
$ aws logs put-resource-policy \
--policy-name MyResourcePolicy \
--policy-document file://transaction-search-policy.json
$ # 2. Route trace segments from X-Ray to CloudWatch Logs.
$ aws xray update-trace-segment-destination --destination CloudWatchLogs
$ # 3. (Optional) raise the indexed-span sampling percentage.
$ aws xray update-indexing-rule \
--name "Default" \
--rule '{"Probabilistic": {"DesiredSamplingPercentage": 5}}'
A few things the console-driven setup hides:
- The default indexing rule keeps 1% of spans at no cost — raise the percentage only when you understand the per-span ingest cost.
- Spans land in the well-known log group
aws/spans. The trace-search console reads from there directly. - Console users get to the same setting under Application Signals (APM) → Transaction search → Enable Transaction Search.
03Runtime-hosted agents: instrumentation is automatic
For agents you deploy via the AgentCore CLI, you write zero
OTEL code. The runtime ships ADOT, runs
opentelemetry-instrument ahead of your entrypoint,
and emits OTEL traces, EMF metrics, and structured logs for
you. From the get-started doc:
"When you deploy an agent using the AgentCore CLI, the runtime automatically instruments your agent with OpenTelemetry — no additional OTEL libraries or configuration are needed."
Two CloudWatch log groups are created for each Runtime agent:
| Log group | What lands there |
|---|---|
/aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>/[runtime-logs] | Standard stdout/stderr — print, logging.info, framework chatter. |
/aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>/otel-rt-logs | OTEL structured logs — execution details, error tracking, correlation IDs that link to span IDs. |
The Runtime span is documented in AgentCore generated runtime observability data. It's a single operation:
- Span name:
InvokeAgentRuntime - Attributes:
aws.operation.name,aws.resource.arn,aws.request_id,aws.agent.id,aws.endpoint.name,aws.account.id,session.id,latency_ms,error_type,aws.resource.type,aws.xray.origin,aws.region. error_typeis one ofthrottle,system, oruser, and is only present on failed invocations.
That gives you a usable trace tree on day one even before you add your own framework spans.
04Non-runtime agents: the OTEL env vars
For agents hosted elsewhere — ECS, EKS, Lambda, anything outside AgentCore Runtime — you get the same dashboards by adding the ADOT distro to your image and exporting six environment variables. From the configure doc:
Six env vars + ADOT distro = same dashboard
Three things that catch people:
aws-opentelemetry-distro>=0.10.0is the minimum version pinned in the doc — older releases predate the GenAI conventions AgentCore expects.- For frameworks other than Strands, LangChain, or CrewAI, you also need a GenAI instrumentation library — OpenInference, Openllmetry, OpenLit, or Traceloop are the four the docs explicitly support.
- Session correlation is opt-in. The recommended snippet:
ctx = baggage.set_baggage("session.id", session_id); context.attach(ctx). Without this, the dashboard groups spans by trace ID but not by session ID — and the per-session views go empty.
05The six headers that change the trace
Whether you invoke through boto3 or curl, AgentCore
Runtime honours six HTTP headers that flow straight into the
OTEL pipeline and the resulting spans. From the
enhanced runtime observability table:
| Header | Purpose |
|---|---|
X-Amzn-Trace-Id | X-Ray trace ID. Sets root/parent IDs and the sampling decision (Sampled=1 = 100%). OTEL auto-generates one if absent. |
traceparent | W3C trace context. Required for distributed traces with non-AWS systems. |
tracestate | W3C vendor-specific trace state alongside traceparent. |
baggage | W3C baggage — userId=alice,serverRegion=us-east-1. Propagated across spans for filtering. |
X-Amzn-Bedrock-AgentCore-Runtime-Session-Id | AgentCore session correlation. Surfaces sessions in the GenAI Observability dashboard. |
mcp-session-id | MCP session ID. Lets you trace one MCP session across multiple tools/call invocations. |
The practical implication: if you have a frontend already
instrumented with W3C traceparent, you can forward
it on the invoke_agent_runtime call and the
resulting agent spans plug straight into your existing
distributed trace.
06Service-provided telemetry by resource type
Each AgentCore resource type ships a different default. From Amazon Bedrock AgentCore generated observability data:
| Resource | Service-provided data | GenAI Observability | CloudWatch |
|---|---|---|---|
| Agent (Runtime) | Metrics | Yes | Yes |
| Memory | Metrics, Spans*, Logs* | No | Yes |
| Payments | Metrics, Spans, Logs | No | Yes |
| Gateway | Metrics | No | Yes |
| Built-in Tools | Metrics | No | Yes |
| Policy | Metrics, Spans** | Yes | Yes |
* Memory spans and logs are off by default — you turn them
on per memory resource via the Tracing and
Log delivery panes.
** Policy observability surfaces under the Gateway tab on
the GenAI Observability page.
The Runtime is the only resource whose default telemetry shows
up in the GenAI Observability dashboard with no extra
configuration. For Memory, Gateway, Identity, and Built-in
Tools you have to attach a log destination (CloudWatch Logs,
S3, or Firehose) and, where applicable, toggle the
Tracing widget — once you do, spans flow into
the same aws/spans log group as everything else.
07Limits and gotchas worth knowing
Distilled from the get-started, configure, and runtime-metrics pages:
- 10 minutes after enabling Transaction Search before spans appear. Plan the runbook around it.
- Default span sampling is 1%. Production debugging usually needs 5–10%; pricing matters above that.
- Resource-usage telemetry can lag by up to 60 minutes and precision differs across metrics — don't alert on
CPUUsed-vCPUHoursfor real-time scaling decisions. USAGE_LOGSare 1-second granularity and containagent.runtime.vcpu.hours.used/agent.runtime.memory.gb_hours.usedper session. They're vended; you opt them in per agent.- Memory spans require enablement. A memory resource with the Tracing toggle off will surface zero spans even with Transaction Search on.
- WebSocket agents emit extra metrics —
ActiveStreamingConnections,InboundStreamingBytesProcessed,OutboundStreamingBytesProcessed. Sum over 1 minute, not average. - Custom dashboards live under namespace
bedrock-agentcore. ADOT writes EMF metrics there — that's the namespace to point CloudWatch alarms at.
08Try it in five minutes
A complete loop on a Runtime-hosted Strands agent:
# 1. Enable Transaction Search once per account/Region.
$ aws xray update-trace-segment-destination --destination CloudWatchLogs
$ # 2. Scaffold and ship a tiny agent.
$ npm install -g @aws/agentcore
$ agentcore create --name ObservabilityDemo
$ cd ObservabilityDemo && agentcore deploy
$ # 3. Invoke with a session ID so the trace tree groups by session.
$ python - <<'PY'
import boto3, json, uuid
client = boto3.client("bedrock-agentcore")
response = client.invoke_agent_runtime(
agentRuntimeArn="arn:aws:bedrock-agentcore:us-east-1:111122223333:runtime/ObservabilityDemo-xxxxxx",
runtimeSessionId=str(uuid.uuid4()), # ≥ 33 chars; UUID4 is 36
payload=json.dumps({"prompt": "What is 2 + 2?"}),
qualifier="DEFAULT",
)
print(json.loads(response["response"].read()))
PY
$ # 4. Open the GenAI Observability page in CloudWatch.
$ open "https://console.aws.amazon.com/cloudwatch/home#gen-ai-observability"
Within a minute you should see one session,
one trace whose root is InvokeAgentRuntime, child
spans for each LLM call (with GenAI semantic-convention
attributes like gen_ai.system,
gen_ai.request.model,
gen_ai.usage.prompt_tokens), and a metrics row
under the bedrock-agentcore namespace.
Tomorrow we'll look at AgentCore Code Interpreter — the sandboxed code-execution tool and how its session model differs from Runtime's.
Sources: Get started with AgentCore Observability, Add observability to your Amazon Bedrock AgentCore resources, Amazon Bedrock AgentCore generated observability data, AgentCore generated runtime observability data, View observability data for your Amazon Bedrock AgentCore agents, CloudWatch Transaction Search.
If the docs change, this tip is a snapshot of that day — check the sources for current behaviour.
This page — research, writing, verification, and deployment — was built by Claude Cowork. No human touched the prose, the layout, or the upload pipeline. The tip was generated this morning, cross-checked against the official AWS docs by an independent verification pass, and published to Cloudflare R2 on a schedule.