We can make sure Athena only reads as much data as it needs for a particular query by partitioning our data. We do this by storing the data files in a Hive folder structure that represents the patitions we’ll use in our queries. s3://mybucket/data/year=2021/month=06/day=27/file1.json s3://mybucket/data/year=2021/month=06/day=27/file2.json s3://mybucket/data/year=2021/month=06/day=28/file1.json We can then create...

AWS finally released ECS Anywhere last week, which allows you to use ECS to schedule tasks on on-premise hosts. The whole setup is very straightforward, and it’s quite reasonably priced at $0.01025 per hour for each managed ECS Anywhere on-premises instance - about $1.72 per week per host. We need...

Even with session manager for accessing instances, sometimes it’s handy to just open up a port to your current IP address - to allow access to a load balancer for example. One quick way to do this is with an external data source. data "external" "current_ip" { program = ["bash",...

Pending Update Metrics APT conveniently has some hooks available to run custom scripts before, during and after patching. We can take advantage of these to publish a metrics file that can be picked up by node_exporter to monitor the status of pending updates across our servers. First we need a...

Wrote a little script recently to send volume size metrics to prometheus. I’m already using cadvisor which provides a container_fs_usage_bytes metric with labels for container_label_com_docker_compose_project and container_label_com_docker_compose_service, but I wanted a bit more detail on where data was being used. The script is run every minute by cron and writes...

The last CE release of Observium (19.8.10000) removed several alert transports that are now only available in their paid edition. Since the external program transport still exists we can create a reasonable replacement for this with a simple shell script and the environment variables that Observium makes available when calling...

After adding VLAN interfaces to my server, I discovered that using the interfaces independently (eg curl --interface enp1s0.10 example.com) wouldn’t work. Because the default route on the system is via enp1s0, the router drops the packet since the gateway for enp1s0 has no route back to the source of the...