Mysql connect/read/write timeouts in rails

Ran into a problem in production recently where mysql/OS ran out of memory, didn't die, but didn't actually do anything, so connections to it were still open. As a result, the rails mysql adapter would just hang indefinitely in the middle of a request to mysql, with no exception being thrown, thereby not triggering our db failover and causing our entire site to come to a screeching halt!

In order to prove my hypothesis correct, not to mention to have a way to test that I'd fixed the problem, I first had to be able to duplicate this problem, i.e. I had to simulate mysql hanging without actually killing it. Fortunately *nix has a good way of doing this via kill:

To stop a process: kill -s STOP mysqld_pid
To resume it: kill -s CONT mysqld_pid

This duplicated the problem perfectly - any ActiveRecord::Base.execute would hang indefinitely.

Digging around some, I discovered that the ruby mysql gem does have a way to set timeouts, but this config is not exposed in database.yml, so I dutifully forked rails (yay github!), patched it, and solved my problem.

Here's the Rails issue from which you can get the patch

Added EBS support to rubber

I recently committed a change to rubber that adds some basic support for Amazon EC2's persistent volumes (Elastic Block Store, EBS). The way it works is that you tell rubber the host volume configurations, and those volumes will get created/formatted/mounted automatically on those hosts. If the host gets destroyed and recreated, the volume will get remounted without being formatted.

For example, in rubber.yml you'd have something like:

hosts:
db01:
  availability_zone: us-east-1a
  ec2_volumes:
    - size: 100 # size of vol in GBs
    zone: us-east-1a # zone to create volume in, needs to match host's zone
    device: /dev/sdh # OS device to attach volume to
    mount: /mnt/mysql_data # The directory to mount this volume to
    filesystem: ext3 # the filesystem to create on volume

When you first create/bootstrap the db01 hostname, the volume will automatically get created, formatted and mounted to the given path on the instance. If you then destroy db01, you're given the option to cleanup the volume, but if you don't, recreating db01 will automatically re-attach and remount the volume (without formating it).

This addresses the use case (mine) where you assign persistent storage to say a DB server, and want to be able to quickly rebuild that instance if it dies, without having to figure out which ec2 volume IDs map to which instance IDs. Just make sure you answer NO to destroying the instance if you do a rubber:destroy and want to keep the volume around for when you recreate the instance.

I also changed Elastic IPs to work in a similar fashion - you specify you want an instance to have an elastic ip, and rubber allocates and assigns the static ip to that instance, remembering the IP/instance mapping so that if you destroy/recreate the instance, it will get back the same IP.

Similar to the volumes support, when you destroy an instance you get prompted to cleanup the IP, but if you answer NO it keeps the mapping around to reassign it to the instance at recreation time.
e.g.

hosts:
 web01:
   use_static_ip: true

Time Capsule restores from a secure shared disk

My macbook pro died this week. Turns out my logic board went caput, so they had to put in a new one. In the interim, I had a spare I could use, but first I had to do a full system restore from my backup on our Time Capsule.

We have our time capsule setup for secure shared disks with accounts so that each user has their own password protected private share so that no other user can snoop on their backup.

Unfortunately, the Full System Restore available when booting the Leopard DVD failed to see my backup even though I used my private time capsule account credentials when connecting to it. All I could see were other backups that were available on the public share, leading me to believe that my credentials were working to mount the image, but the restore app wasn't smart enough to mount my private mount point instead of (or in addition to) the public one. I tried calling apple support, and after escalating the issue, all they could suggest was to copy my backup from the private share to the public one. Needless to say, it takes a really long time to a couple of hundred GB even when connected directly to the time capsule using an ethernet cable.

Fortunately, when booting from the Leopard DVD, one also has access to a Terminal. By manually mounting the private share from Terminal, the full system restore UI was then able to see my private share and the restore was able to take place.

To Summarize, Boot from the Leopard DVD, select Utilities->Terminal, then use the mount command to mount your private time capsule share. In my case, my username and share are named the same - mconway - I think this is just the way time capsule works, so the pattern should be the same for you. Also, I had to add the .local to the end of the time capsule's name for the mount to work.

# Create directory to mount to
mkdir -p /Volumes/mconway
# Do the mount
mount -t afp 'afp://mconway:mypassword@my-time-capsule-name.local/mconway' /Volumes/mconway

Once mounted, you can verify with a "ls /Volumes/mconway" and you should see the sparseimage for your backup. Quit Terminal, go to Utilities->Full System Restore, and your backup should be visible, though you might have to lead the UI through the hoops by connecting to your time capsule and typing in your credentials again.

Hopefully this will save someone else the hours of grief I was faced with.

Converting to blogger from wordpress

The HTPC I was running wordpress on got replaced with an appletv+boxee (boxee rocks btw), and rather than waste electricity just to run my blog, I figured I'd convert it to a hosted solution. I don't want to have to do this again any time soon, so I wanted to use a hosted service thats pretty reliable and will be around for a while. I didn't feel like paying wordpress for the privilege of mapping a custom domain, so blogger it was. Problem was, I couldn't get my wordpress export imported into blogger. I tried wxr2blogger, but while the script ran fine, blogger wouldn't actually import the results. I also found a ruby script which seemed to work, but wouldn't pull my comments across, so I ended up tweaking it so that it would work with comments as well. Here is my improved wordpress2bloggger.rb script.

Rubber presentation at Boston Ruby Group

I gave a presentation about using rubber to deploy rails apps to ec2 to the Boston Ruby Group on Tuesday July 8th, 2008.  I'm not quite sure how good my speaking skills are, but it seemed to go well.  See the slides on slideshare.

RailsConf 2008

In Portland, Oregon all this weekend for RailsConf 2008, any like-minded readers interested in chatting over a beer?I gave a lightning talk on rubber today, if anyone wangts a more detailed demo, seek me out (twitter: mattconway). 

Replacing AppleScript with Ruby

I had a brief run-in with AppleScript trying to overcome an OS X terminal bug.  and its not something I enjoyed.  I would much rather be able to do scripts like this in a language I'm more comfortable in like ruby.  I've known for a while that one of the enhancements in OS X Leopard is that it has native support for scripting the UI from within other scripting languages like python and ruby, however, I didn't realize that I could drop those scripts in the user scripts folder (~/Library/Scripts) and have them available from the scripts toolbar icon. Well, one can!  The trick to dong this is that the script has to be executable (chmod +x ~/Library/Scripts/Application/Terminal/Layout.rb), and needs to have a shebang to the desired  interpreter (#!/usr/bin/ruby as the first line.) Finally I can get rid of the only AppleScript code I was forced to write.  Here is a (better!) ruby equivalent of the AppleScript from the link above.

#!/usr/bin/ruby

require 'osx/cocoa'
include OSX
OSX.require_framework 'ScriptingBridge'

yMin = 22
xMin = 187
xSize = 504
ySize = 365
xCount = 2

term = SBApplication.applicationWithBundleIdentifier_("com.apple.Terminal")
sorted_windows = term.windows.sort_by {|w| w.name.to_s.match(/([0-9]+)$/)[1] }
sorted_windows.each_with_index do |window, i|
  newx = xMin + xSize * (i % xCount)
  newy = yMin + ySize * (i / xCount).to_i
  # Setting rect doesn't work, so set position instead even though it is
  # deprecated
  # rect = NSRect.new(newx, newy, xSize, ySize)
  # window.bounds = rect
  pos = NSPoint.new(newx, newy)
  window.position = pos
  # size doesn't work reliably
  # size = NSPoint.new(xSize, ySize)
  # window.size = size
end