IAM users / groups in AWS

Metric	Target
RPO	n/a (no backups possible)
RTO	4 hours

Start by figuring out what exactly was deleted.

This can be done using AWS CloudTrail and AWS Athena. Our crate account has an Athena table called cloudtrail_logs_crate_cloudtrail created, which can be used to query events in all of AWS. i.e., to find a list of deleted IAM resources, we can construct an Athena query like:

SELECT * FROM "default"."cloudtrail_logs_crate_cloudtrail"
WHERE eventsource = 'iam.amazonaws.com'
AND eventname LIKE 'DeleteUser'
ORDER by eventtime DESC
LIMIT 10;

Note

Athena queries can take a pretty long time (5 min+) as they scan through data in s3.

This will give you a list of all the recently deleted users (adjust accordingly for groups etc.). Look at requestparameters for the details of what was deleted.

Recovering cloud backup users / groups

Most users and groups in IAM are for cloud projects and used to store backups in s3.

Refer to this support article for details.

Recovering users / groups managed in Terraform

These are most, but not all, of the other users and groups.

$ cd salt/terraform/cratedb-cloud
$ terraform plan
... check the difference
$ terraform apply

Note

This requires the VPN being active and access to Vault.