Mysql connect/read/write timeouts in rails

Ran into a problem in production recently where mysql/OS ran out of memory, didn't die, but didn't actually do anything, so connections to it were still open. As a result, the rails mysql adapter would just hang indefinitely in the middle of a request to mysql, with no exception being thrown, thereby not triggering our db failover and causing our entire site to come to a screeching halt!

In order to prove my hypothesis correct, not to mention to have a way to test that I'd fixed the problem, I first had to be able to duplicate this problem, i.e. I had to simulate mysql hanging without actually killing it. Fortunately *nix has a good way of doing this via kill:

To stop a process: kill -s STOP mysqld_pid
To resume it: kill -s CONT mysqld_pid

This duplicated the problem perfectly - any ActiveRecord::Base.execute would hang indefinitely.

Digging around some, I discovered that the ruby mysql gem does have a way to set timeouts, but this config is not exposed in database.yml, so I dutifully forked rails (yay github!), patched it, and solved my problem.

Here's the Rails issue from which you can get the patch

Added EBS support to rubber

I recently committed a change to rubber that adds some basic support for Amazon EC2's persistent volumes (Elastic Block Store, EBS). The way it works is that you tell rubber the host volume configurations, and those volumes will get created/formatted/mounted automatically on those hosts. If the host gets destroyed and recreated, the volume will get remounted without being formatted.

For example, in rubber.yml you'd have something like:

hosts:
db01:
  availability_zone: us-east-1a
  ec2_volumes:
    - size: 100 # size of vol in GBs
    zone: us-east-1a # zone to create volume in, needs to match host's zone
    device: /dev/sdh # OS device to attach volume to
    mount: /mnt/mysql_data # The directory to mount this volume to
    filesystem: ext3 # the filesystem to create on volume

When you first create/bootstrap the db01 hostname, the volume will automatically get created, formatted and mounted to the given path on the instance. If you then destroy db01, you're given the option to cleanup the volume, but if you don't, recreating db01 will automatically re-attach and remount the volume (without formating it).

This addresses the use case (mine) where you assign persistent storage to say a DB server, and want to be able to quickly rebuild that instance if it dies, without having to figure out which ec2 volume IDs map to which instance IDs. Just make sure you answer NO to destroying the instance if you do a rubber:destroy and want to keep the volume around for when you recreate the instance.

I also changed Elastic IPs to work in a similar fashion - you specify you want an instance to have an elastic ip, and rubber allocates and assigns the static ip to that instance, remembering the IP/instance mapping so that if you destroy/recreate the instance, it will get back the same IP.

Similar to the volumes support, when you destroy an instance you get prompted to cleanup the IP, but if you answer NO it keeps the mapping around to reassign it to the instance at recreation time.
e.g.

hosts:
 web01:
   use_static_ip: true