Send AI Agent Audit Logs to Splunk (HEC)

Part of SIEM audit forwarding. Decision Terrain sends audit events to Splunk's HTTP Event Collector (HEC).

Prepare Splunk

In Splunk, create an event index for the approved audit data, enable HEC, and create a token restricted to that index. Separately grant the people who will review the data permission to search the index. Set retention, capacity, rate and license limits, and approve the destination for every Organization you forward.

Use HEC event ingestion with a trusted HTTPS certificate. Leave indexer acknowledgement off: delivery does not poll HEC acknowledgement IDs, so you verify with real searches instead. See Splunk's HEC setup instructions.

Optional indexed confirmation

To let Decision Terrain confirm that submitted events reached the index, prepare a separate Splunk REST API authentication token. The destination form calls this the Read-only search token. Create it for a dedicated account whose role permits searches (search capability) and read access to the approved audit index, without administrative or data-write permissions. The token inherits the account's permissions; “read-only” describes those permissions, rather than a separate HEC token type. Enable token authentication on the Splunk instance and set an appropriate token expiry and rotation process. See Splunk's token authentication setup.

Supply the Search API base URL, for example https://splunk.example:8089, and the token value together. Use the search head's HTTPS management API address, reachable from the managed audit bridge. Omit /services/collector and the search API path: Decision Terrain appends /services/search/v2/jobs/export and authenticates with Authorization: Bearer <token>. Enter only the token value in the form. The HEC ingestion token cannot authenticate these searches. See Splunk's search API reference.

For Splunk Cloud, confirm that REST API access is enabled and that the bridge's network address is allowed to reach the deployment's API endpoint. Your deployment may require a Splunk Support request; free trials do not provide REST API access. See Splunk Cloud REST API access requirements.

Without the search endpoint and search token, Decision Terrain reports submission only. You can still verify indexing manually in Splunk Search.

Add the destination

In Governance → Policies → External security → Audit forwarding, choose Add destination and select Splunk. Enter:

• the HEC base endpoint, such as https://splunk.example:8088/services/collector (Decision Terrain builds the event requests),
• the index and the HEC token,
• the CA to trust, if your certificate is not publicly trusted,
• optionally, the Search API base URL and Read-only search token described above.

Save, validate, send a synthetic test, and activate collection. See SIEM audit forwarding.

Synthetic tests, activation and other destination controls require independent human security-policy approval. A pending request has not taken effect; after a different authorized administrator approves it, the original requester must repeat the exact action.

Verify indexed events

Create known product events and note their family, Organization, event ID, outcome and original occurrence time. Events use source="decisionterrain-audit" and sourcetype="decisionterrain:audit:otel". Search a window that covers the original occurrence time, and inspect raw rows before deduplicating:

index=<audit_index> source="decisionterrain-audit" earliest=-30m
"audit.organization.id"="<organization-uuid>"
| table _time event.name audit.schema.version audit.event.id audit.organization.id audit.event.occurred_at audit.event.captured_at audit.action audit.result audit.actor.user_id audit.actor.service_identity_id audit.agent.id audit.run.id audit.security.agent_id audit.security.work.known audit.security.work.unknown

_time follows the original event time. Security and agent events carry different attributes, so family-specific columns are blank for the other family. Do not use export time in place of the original occurrence time. All events have severity INFO, so use the outcome and risk fields, not severity, to decide what is an incident.

A security investigation that deduplicates event identities before counting:

index=<audit_index> source="decisionterrain-audit" earliest=-15m
"event.name"="organization.security" "audit.schema.version"=1
| dedup event.name audit.organization.id audit.event.id
| where 'audit.result'="denied" OR 'audit.result'="failed"
| stats count BY audit.organization.id audit.category audit.action audit.result

This search does not create an alert or a response.

Starter detections

splunk-detections.conf defines eleven saved searches for the conditions listed in SIEM audit forwarding. Every search is disabled and has no alert action. Copy the stanzas into a private savedsearches.conf in the app that owns your audit index, replace <audit_index>, review the schedules and thresholds, test the searches against your data, and add your own alert actions. Then enable only the searches you want by setting disabled = 0 in their stanzas. Adding alert actions alone does not enable a search.

To detect a forwarding path that went silent, compare received heartbeats with an explicit list of expected Organizations. Heartbeats require an active destination that includes the security audit family; the default interval is one hour, but your operator can change it. This absence detection is not included in the starter file.

In Splunk, create a lookup file named dt_expected_siem_organizations.csv, available to the account and app that run the alert. Include one row per Organization expected to forward security events to this index, with these columns (replace the example UUID):

audit.organization.id,expected
<organization-uuid>,1

Maintain this list when Organizations start or intentionally stop forwarding. Do not build it only from recent events: an Organization that never reported or has been silent longer than the search window would be missing.

index=<audit_index> source="decisionterrain-audit" "event.name"="organization.security" "audit.action"="siem_heartbeat" earliest=-24h latest=now
| stats max(_time) AS last BY audit.organization.id
| inputlookup append=true strict=true dt_expected_siem_organizations.csv
| stats max(last) AS last max(expected) AS expected BY audit.organization.id
| where expected=1 AND (isnull(last) OR last < relative_time(now(), "-2h"))
| table audit.organization.id last

Schedule this search and alert when it returns any rows. Listed Organizations with no heartbeat in the search window, including those that never reported, remain in the results. A blank last means no heartbeat was found in that window. The two-hour threshold assumes hourly heartbeats; adjust it for your configured interval and indexing delay. Treat a missing or unreadable lookup as an alert-search failure, not evidence that forwarding is healthy. Verify the alert with a known missing heartbeat before relying on it.

Troubleshoot

Observation	Check
Authentication or index rejection	The HEC token is enabled and allowed to write to that index, and the endpoint is correct
TLS rejection	Hostname and certificate trust. Keep verification on and supply the CA
HEC accepted but no search result	The index, search permissions, the search time window, and indexing latency
Automatic indexed confirmation fails	The separate API token's expiry and account permissions, token authentication enabled, and bridge access to the Search API base URL
Only the event body is available	The destination expects structured fields; do not enable raw export
Duplicate rows	Retries after a lost acknowledgement. Deduplicate by family, Organization and event ID

HEC success alone is not proof of indexing, and this route is not exactly-once or a permanent archive.