Finding unusual source IPs in CloudTrail Event History

The question I wanted answered

Most days, the management API calls in an AWS account have fairly predictable origins: a CI runner, an office or VPN connection, an EC2 instance, or a person working from a familiar network. If credentials turn up somewhere else, I want to know which principal used them, where the calls came from, and what they tried to do.

I already have a host that can run a scheduled Python job and receive webhook notifications. For this particular question, I didn't need to start by delivering CloudTrail logs into CloudWatch Logs, configuring metric filters, and maintaining alarms. CloudTrail's Event History is enabled by default and exposes recent management events through the LookupEvents API . The script polls that history and applies the policy in Python.

This is a useful fit for a modest account and an existing scheduler. It avoids the additional log delivery and alarm setup for this one check. The host, network access, and time spent operating the job still have costs. If you need faster reaction, managed retries, or many event-driven rules, EventBridge is a sensible option. I wouldn't claim this script is cheaper for every account without measuring that account's event volume and existing infrastructure.

What counts as unusual?

The policy starts with two kinds of expected source:

• Your own public CIDRs, such as a VPN exit address, NAT gateways from this or other related AWS accounts, or a fixed CI runner egress address.
• Published AWS IP ranges in the AWS source Regions you choose to allow.

The script downloads AWS's published IP ranges , looks up the source IP in each CloudTrail event, and compares matching AWS ranges with the allowed source Regions. It can suppress exact role, service, and API combinations that you have reviewed as routine. Findings are grouped by actor with source IPs, API calls, timestamps, and event IDs.

An AWS IP range is a broad signal. Anyone able to use an AWS-hosted machine in an allowed Region could originate traffic there. A source outside the expected ranges is a reason to inspect an event; a source inside them is not proof that its actor was authorised. sourceIPAddress is also occasionally a service identifier; the current script skips those events.

Why monitor if IAM can restrict source IPs?

For a credential used exclusively from fixed public networks, I would consider an IAM policy with <code>aws:SourceIp</code> . An explicit deny can stop a request before it succeeds. It needs testing against the actual call paths: calls through VPC endpoints and service-to-service requests have different network context, and people working from changing connections may need a different policy.

The monitor is useful where a strict source policy is impractical, and for visibility into unexpected attempts or configuration drift. An alert gives you the principal, API names and source to investigate. I would still enforce source restrictions for credentials with genuinely fixed network origins.

Running the check

Download cloudtrail-event-watch.py and install boto3 in a Python environment. Give its AWS identity cloudtrail:LookupEvents access in the accounts you intend to inspect. Configure every Region you need to scan; Event History is regional, and the script scans one account at a time using the selected AWS profile.

For example, a scheduled host with a persistent working directory could use:

export AWS_PROFILE=monitoring
export AWS_REGIONS_TO_SCAN=eu-west-2,us-east-1
export ALLOWED_AWS_SOURCE_REGIONS=eu-west-2
export EXPECTED_CIDRS=203.0.113.10/32,2001:db8::/48
export STATE_FILE=/var/lib/cloudtrail-watch/monitoring.json
export LOOKBACK_MINUTES=90
export WEBHOOK_URL=https://alerts.example.net/cloudtrail
export WEBHOOK_TOKEN=your-secret-token
python3 cloudtrail-event-watch.py

The example addresses and URL are placeholders. Keep the webhook token in your scheduler's secret store. Avoid putting it in a repository. The endpoint receives a JSON rollup with alert_id, counts, actor information, source IPs, event names, and event IDs. It can translate that into a Google Chat or Slack message, an email, or an incident in your own system. A Google Chat or Slack incoming webhook cannot consume this arbitrary rollup directly: each has its own message format, so use a receiver that formats the alert if those are your destinations.

Schedule the script more frequently than LOOKBACK_MINUTES. Overlapping windows and a state file avoid sending the same event on each run. Keep the state file on persistent storage and run a single instance per state file. The script locks concurrent runs and writes its state atomically. It saves alert event IDs after successful webhook delivery; a failed delivery will be retried on later runs while the event remains in the lookback window. The receiving endpoint should also deduplicate on alert_id, since an HTTP timeout might follow a successful delivery.

By default, a successful delivery exits 0; set ALERT_EXIT_CODE=2 if your scheduler should also flag a run containing findings. A delivery failure exits 1. Monitor failures and missed runs separately: a broken scheduled job can go silent.

What this check can and cannot see

Event History contains the last 90 days of management events in each Region. It does not contain ordinary data events such as S3 object reads, and it does not aggregate accounts for you. A missing Region or account leaves a gap. API activity may also take time to appear, so an hourly job is a periodic investigation aid with a detection delay.

The local state keeps a bounded list of 10,000 event IDs. If your lookback window contains more than that many events, old IDs can fall out and trigger repeat alerts. Reduce the window, run more often, or use a durable state store if that describes your account. The script also needs outbound access to CloudTrail, AWS's IP range file, and your webhook. Check that the job is still running and that those dependencies are reachable.

I treat an alert as a prompt to review the CloudTrail record, the principal's expected workload, and related events. It is one small layer of operational visibility, useful precisely because the question is narrow: did recent management activity come from a source I didn't expect?