Swapfile for my brain

Update to Scripting with Ruby on OS X

noreply@blogger.com (Matt Conway) — Thu, 20 Oct 2011 11:52:00 +0000

After updating to lion, I needed to tweak the script I had written here, so here is the updated version with some more comments to help people modify and write their own Ruby/OSX scripts


#!/usr/bin/ruby
# -*- coding: utf-8 -*-

# Put this in Library/Scripts/Applications/Terminal
#
# To get the script to show up in menubar,
# run "AppleScript Editor" -> Preferences -> "Show script menu in menu bar"
#
# To figure out what properties are available, "AppleScript Editor" -> File -> "Open Dictionary"

require 'osx/cocoa'
include OSX
OSX.require_framework 'ScriptingBridge'

def alert(title, message='', is_error=false)
  puts "Displaying #{is_error ? 'error ' : ''}dialog - #{title}: #{message}"
  if is_error
    NSRunCriticalAlertPanel(title, message, 'OK', nil, nil)
  else
    NSRunAlertPanel(title, message, 'OK', nil, nil)
  end
end

rowSize = 24
colSize = 80
yMin = 22
xMin = 187
xCount = 2

# To get bundle ID, View Package Contents on App -> Contents/Info.plist -> Bundle ID
term = SBApplication.applicationWithBundleIdentifier_("com.apple.Terminal")

# set the # of rows and columns (global setting, so set to one tab, sets for all)
# term.windows.first.tabs.first.currentSettings.numberOfRows = rowSize
# term.windows.first.tabs.first.currentSettings.numberOfColumns = colSize
term.defaultSettings.numberOfRows = rowSize
term.defaultSettings.numberOfColumns = colSize

# sort on title to order according to ⌘N window selection shortcuts, index is some other order
sorted_windows = term.windows.sort_by {|w| w.name.to_s.match(/([0-9]+)$/)[1] }
sorted_windows.each_with_index do |window, i|
  # puts "#{window.index}: #{window.name}"

  # sizes should be the same for all windows after setting numberOfRows/Cols
  xSize = window.bounds.width
  ySize = window.bounds.height

  newx = xMin + xSize * (i % xCount)
  newy = yMin + ySize * (i / xCount).to_i

  rect = NSRect.new(newx, newy, xSize, ySize)
  window.bounds = rect

  # If setting rect doesn't work, set position instead even though it is
  # deprecated
  # pos = NSPoint.new(newx, newy)
  # window.position = pos
  #
  # size also deprecated
  # size = NSPoint.new(xSize, ySize)
  # window.size = size
end

JConsole via a SOCKS ssh tunnel

noreply@blogger.com (Matt Conway) — Wed, 25 Aug 2010 01:33:00 +0000

I've been learning how to use cassandra lately, and one of the most useful tools when working with it is to be able to query status via all the JMX hooks it exposes. At first I was using jmxterm to query all the available settings, and while this works great, it was tough as a cassandra newb to figure out which beans I needed to query.

What I wanted to do was use JConsole so that I could browse the available beans without having to know a priori what they were. Unfortunately, our production system is all remote (ec2), so this makes it tricky to use JConsole.

At first I tried to use a basic ssh port mapping, but this didn't work due to RMI's need to create its own ports in the JConsole runtime to which the monitored process tries to connect back to - only the single port is tunnelled by ssh. There are a number of documented solutions to doing this, but they all seemed very intrusive and hard to setup correctly (e.g. running a hacked jmx agent within your monitored process).

I then tried using JConsole running remotely but displaying on my local X server, but the performance wasn't that great, and its a pain to have to install all the X runtime on my production system just for running JConsole.

Eventually I smacked my forehead after remembering that ssh can also act as a SOCKs proxy, which works great for any application that supports it. Fortunately, JConsole does (in jdk 1.6 at least).

Here is the bash alias I use to setup the socks connection and connect to my cassandra instance via jconsole. I have multiple remote remote cassandra hosts, so this lets me run JConsole independently for each.


function jc {
        host=$1
        proxy_port=${2:-8123}
        jconsole_host=wrongway
        ssh -f -D$proxy_port $host 'while true; do sleep 1; done'        ssh_pid=`ps ax | grep "[s]sh -f -D$proxy_port" | awk '{print $1}'`
        jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=$proxy_port ser
vice:jmx:rmi:///jndi/rmi://${jconsole_host}:8181/jmxrmi
        kill $ssh_pid
}

Once this was in my shell environment, I simply run "jc remote_host", and I get a jconsole connected to my cassandra instance on that host.

Note that you need to set "jconsole_host" in the alias for this to work for you. I'm not sure why, but Its sensitive to the value in here, and only worked for me for the first entry I had in /etc/hosts (not localhost) that maps my local hostname to 127.0.0.1

Automating gists with QuickSilver

noreply@blogger.com (Matt Conway) — Wed, 10 Feb 2010 19:39:00 +0000

I like to use gist for sharing code snippets. I like to use QuickSilver for automating common tasks. To combine the two is fairly simple:

Install the gist gem

gem install gist

Enable the Terminal Module in QuickSilver Plug-ins
Create a new HotKey Trigger
Hit "." then type the following command into the Select Item field.  You only need the bash part if you installed the gist gem using rvm/ruby instead of the system rubybash -l -c "pbpaste | gist | pbcopy"
Tab over to Action field, and select "Run Command in Shell"
Save the trigger, then set whatever hotkey you want - I use Cmd-Shift-G - you might need to restart quicksilver for this to take effect

Now you can select some text, copy to clipboard, then hit Cmd-Shift-G to create a gist from the clipboard contents, ending up with a url to that gist now in the clipboard.

Mysql connect/read/write timeouts in rails

noreply@blogger.com (Matt Conway) — Wed, 22 Apr 2009 21:38:00 +0000

Ran into a problem in production recently where mysql/OS ran out of memory, didn't die, but didn't actually do anything, so connections to it were still open. As a result, the rails mysql adapter would just hang indefinitely in the middle of a request to mysql, with no exception being thrown, thereby not triggering our db failover and causing our entire site to come to a screeching halt!

In order to prove my hypothesis correct, not to mention to have a way to test that I'd fixed the problem, I first had to be able to duplicate this problem, i.e. I had to simulate mysql hanging without actually killing it. Fortunately *nix has a good way of doing this via kill:

To stop a process: kill -s STOP mysqld_pid
To resume it: kill -s CONT mysqld_pid

This duplicated the problem perfectly - any ActiveRecord::Base.execute would hang indefinitely.

Digging around some, I discovered that the ruby mysql gem does have a way to set timeouts, but this config is not exposed in database.yml, so I dutifully forked rails (yay github!), patched it, and solved my problem.

Here's the Rails issue from which you can get the patch

Added EBS support to rubber

noreply@blogger.com (Matt Conway) — Sun, 05 Apr 2009 02:07:00 +0000

I recently committed a change to rubber that adds some basic support for Amazon EC2's persistent volumes (Elastic Block Store, EBS). The way it works is that you tell rubber the host volume configurations, and those volumes will get created/formatted/mounted automatically on those hosts. If the host gets destroyed and recreated, the volume will get remounted without being formatted.

For example, in rubber.yml you'd have something like:


hosts:
db01:
  availability_zone: us-east-1a
  ec2_volumes:
    - size: 100 # size of vol in GBs
    zone: us-east-1a # zone to create volume in, needs to match host's zone
    device: /dev/sdh # OS device to attach volume to
    mount: /mnt/mysql_data # The directory to mount this volume to
    filesystem: ext3 # the filesystem to create on volume

When you first create/bootstrap the db01 hostname, the volume will automatically get created, formatted and mounted to the given path on the instance. If you then destroy db01, you're given the option to cleanup the volume, but if you don't, recreating db01 will automatically re-attach and remount the volume (without formating it).

This addresses the use case (mine) where you assign persistent storage to say a DB server, and want to be able to quickly rebuild that instance if it dies, without having to figure out which ec2 volume IDs map to which instance IDs. Just make sure you answer NO to destroying the instance if you do a rubber:destroy and want to keep the volume around for when you recreate the instance.

I also changed Elastic IPs to work in a similar fashion - you specify you want an instance to have an elastic ip, and rubber allocates and assigns the static ip to that instance, remembering the IP/instance mapping so that if you destroy/recreate the instance, it will get back the same IP.

Similar to the volumes support, when you destroy an instance you get prompted to cleanup the IP, but if you answer NO it keeps the mapping around to reassign it to the instance at recreation time.
e.g.

hosts:
 web01:
   use_static_ip: true

Time Capsule restores from a secure shared disk

noreply@blogger.com (Matt Conway) — Tue, 10 Feb 2009 22:28:00 +0000

My macbook pro died this week. Turns out my logic board went caput, so they had to put in a new one. In the interim, I had a spare I could use, but first I had to do a full system restore from my backup on our Time Capsule.

We have our time capsule setup for secure shared disks with accounts so that each user has their own password protected private share so that no other user can snoop on their backup.

Unfortunately, the Full System Restore available when booting the Leopard DVD failed to see my backup even though I used my private time capsule account credentials when connecting to it. All I could see were other backups that were available on the public share, leading me to believe that my credentials were working to mount the image, but the restore app wasn't smart enough to mount my private mount point instead of (or in addition to) the public one. I tried calling apple support, and after escalating the issue, all they could suggest was to copy my backup from the private share to the public one. Needless to say, it takes a really long time to a couple of hundred GB even when connected directly to the time capsule using an ethernet cable.

Fortunately, when booting from the Leopard DVD, one also has access to a Terminal. By manually mounting the private share from Terminal, the full system restore UI was then able to see my private share and the restore was able to take place.

To Summarize, Boot from the Leopard DVD, select Utilities->Terminal, then use the mount command to mount your private time capsule share. In my case, my username and share are named the same - mconway - I think this is just the way time capsule works, so the pattern should be the same for you. Also, I had to add the .local to the end of the time capsule's name for the mount to work.


# Create directory to mount to
mkdir -p /Volumes/mconway
# Do the mount
mount -t afp 'afp://mconway:mypassword@my-time-capsule-name.local/mconway' /Volumes/mconway

Once mounted, you can verify with a "ls /Volumes/mconway" and you should see the sparseimage for your backup. Quit Terminal, go to Utilities->Full System Restore, and your backup should be visible, though you might have to lead the UI through the hoops by connecting to your time capsule and typing in your credentials again.

Hopefully this will save someone else the hours of grief I was faced with.

Converting to blogger from wordpress

noreply@blogger.com (Matt Conway) — Tue, 30 Dec 2008 22:35:00 +0000

The HTPC I was running wordpress on got replaced with an appletv+boxee (boxee rocks btw), and rather than waste electricity just to run my blog, I figured I'd convert it to a hosted solution. I don't want to have to do this again any time soon, so I wanted to use a hosted service thats pretty reliable and will be around for a while. I didn't feel like paying wordpress for the privilege of mapping a custom domain, so blogger it was. Problem was, I couldn't get my wordpress export imported into blogger. I tried wxr2blogger, but while the script ran fine, blogger wouldn't actually import the results. I also found a ruby script which seemed to work, but wouldn't pull my comments across, so I ended up tweaking it so that it would work with comments as well. Here is my improved wordpress2bloggger.rb script.

Rubber presentation at Boston Ruby Group

noreply@blogger.com (Matt Conway) — Wed, 09 Jul 2008 21:17:00 +0000

I gave a presentation about using rubber to deploy rails apps to ec2 to the Boston Ruby Group on Tuesday July 8th, 2008. I'm not quite sure how good my speaking skills are, but it seemed to go well. See the slides on slideshare.

RailsConf 2008

noreply@blogger.com (Matt Conway) — Sat, 31 May 2008 16:17:00 +0000

In Portland, Oregon all this weekend for RailsConf 2008, any like-minded readers interested in chatting over a beer?I gave a lightning talk on rubber today, if anyone wangts a more detailed demo, seek me out (twitter: mattconway).

Replacing AppleScript with Ruby

noreply@blogger.com (Matt Conway) — Thu, 29 May 2008 15:24:00 +0000

I had a brief run-in with AppleScript trying to overcome an OS X terminal bug. and its not something I enjoyed. I would much rather be able to do scripts like this in a language I'm more comfortable in like ruby. I've known for a while that one of the enhancements in OS X Leopard is that it has native support for scripting the UI from within other scripting languages like python and ruby, however, I didn't realize that I could drop those scripts in the user scripts folder (~/Library/Scripts) and have them available from the scripts toolbar icon. Well, one can! The trick to dong this is that the script has to be executable (chmod +x ~/Library/Scripts/Application/Terminal/Layout.rb), and needs to have a shebang to the desired interpreter (#!/usr/bin/ruby as the first line.) Finally I can get rid of the only AppleScript code I was forced to write. Here is a (better!) ruby equivalent of the AppleScript from the link above.


#!/usr/bin/ruby

require 'osx/cocoa'
include OSX
OSX.require_framework 'ScriptingBridge'

yMin = 22
xMin = 187
xSize = 504
ySize = 365
xCount = 2

term = SBApplication.applicationWithBundleIdentifier_("com.apple.Terminal")
sorted_windows = term.windows.sort_by {|w| w.name.to_s.match(/([0-9]+)$/)[1] }
sorted_windows.each_with_index do |window, i|
  newx = xMin + xSize * (i % xCount)
  newy = yMin + ySize * (i / xCount).to_i
  # Setting rect doesn't work, so set position instead even though it is
  # deprecated
  # rect = NSRect.new(newx, newy, xSize, ySize)
  # window.bounds = rect
  pos = NSPoint.new(newx, newy)
  window.position = pos
  # size doesn't work reliably
  # size = NSPoint.new(xSize, ySize)
  # window.size = size
end

Capistrano and ssh connection limit

noreply@blogger.com (Matt Conway) — Tue, 11 Mar 2008 11:44:00 +0000

We use capistrano to deploy our rails applications from a subversion repository that is accessed using svn+ssh. Recently we ran into a problem where Capistrano was unable to connect to subversion from some of the deployment hosts with an error like:

*** [err :: host.domain.com] ssh_exchange_identification: Connection closed by remote host

Since capistrano tries to fire off N svn checkouts simultaneously, it was running into a connection limit for ssh on the the subversion server. Turns out there is a MaxStartups setting for sshd_config which effectively determines how many clients can be trying to connect simultaneously. It defaults to 10, and our failures started happening when we went above 10. Changing the value to 50 on our svn/ssh server cleared up the problem.

Client for nettica dns service

noreply@blogger.com (Matt Conway) — Mon, 28 Jan 2008 16:12:00 +0000

One of the drawbacks of EC2 is that there are no static IPs to be had, so short of using an external (to ec2) solution, the only way to have redundant load balancers is to use dns round robin. If a load balancer instance goes down, we should ideally remove it from the round robin so that clients don't continue to use the dead IP.

Nettica is one of the few dns providers that has an API that allows changes to round robin records. They have a reference implementation in Java, but since I'm a ruby fanboy these days, I figured it would be nice to have a ruby version instead, so I added one to rubyforge: nettica ruby client. I did it as a gem, so assuming I released it correctly, you should be able to do a "sudo gem install nettica" and then run the simple test script "nettica -u my_user -p my_secret -c list_zones". This script still needs to be expanded to exercise the full api.

One thing that blocked me for the longest time was that the nettcia API required the password to be sent in Base64 encoding, which I was doing, but for some reason the ruby Base64.encode64() method was adding a newline to the end of the encoded string. Once I stripped that out, it all seems to work fine.

Deploying Rails Apps to EC2 with Capistrano

noreply@blogger.com (Matt Conway) — Mon, 28 Jan 2008 15:30:00 +0000

I've created yet another system for deploying rails apps to ec2 using capistrano. None of the others were doing it for me, primarily because I needed a way to deploy/configure/scale-up to multiple instances as needed.
What makes this system different?

Role and/or host based configuration of instances

All configuration for your instances lives in your project's source tree and gets applied on deploy

Configuration system handles dynamically adding instances

Configuration templates for sharing deployment scenarios

Start with a single instance and scale up as needed without editing config

Use whatever AMIs you want (so long as they are debian based :)

You can find the README file in subversion on rubyforge (project rubber). You can install the framework from subversion as a rails plugin in the usual way - the README walks you through a demo using the plugin on a dummy rails app, and there is no need to check anything into subversion to do the demo.

This is still very much a work in progress, so don't use it in production, and expect things to drastically change requiring you to reinstall and reconfigure.

Any feedback appreciated, but check the TODO first :)

Using Capistrano without source control

noreply@blogger.com (Matt Conway) — Sat, 26 Jan 2008 22:27:00 +0000

I'm in the process of writing some capistrano recipes to make deploying to Amazon's Elastic Compute Cloud (ec2) easier. Since I'm making this project available for general consumption, I wanted a way to make it easy for someone to try it out without having to check something into source control.

Fortunately, the capistrano code is nicely organized, and that made it easy for me to add a "noscm" provider that will work with the "copy" deploy strategy to simply copy the projects directory tree to a remote server rather than trying to check it out from the remote server. You can get the code here: http://rubber.rubyforge.org/svn/trunk/lib/capistrano/noscm.rb and then add the following lines to your deploy script to get it to work:


require 'capistrano/noscm'
set :scm, :noscm
set :deploy_via, :copy
set :copy_strategy, :export

This works for my limited use case, YMMV - things like "cap deploy:pending:diff" definately won't work as all I've implemented from the capistrano scm interface is the "export" ability.

Mounting Xen DomU LVM disk in Dom0

noreply@blogger.com (Matt Conway) — Sat, 26 Jan 2008 21:57:00 +0000

I setup a xen based system for our production environment at work. I'm using LVM throughout for all the benefits it provides (resizing, snapshot backups, etc). Each domU gets its own logical volume on dom0's volume group. This logical volume actually looks like an entire physical disk to the domU, and the domU then further partitions it into a boot partition and a nested LVM partition (swap + root fs). Aren't the layers of unix file abstractions a thing of beauty to behold? :)

In retrospect I probably should have mapped LVM logical volumes direct from dom0 as partitions in domU rather than having domU add the extra LVM layer on top - though reading around shows that this _may_ not be supported in the future, so...Recently I completely horked our mysql db slave replica and needed to re-replicate the master to the db. Since our DB is ~ 60GB, I didn't want to shut down (read lock) our master DB for the length of time it would take to copy it over to the slave. The solution to this would be to use an LVM snapshot. This way I could read lock the master db, create the LVM snapshot (very quick - a few seconds at most), unlock the master, then copy the data from the snapshot to the slave at my leisure, letting standard db replication do the catch up later.

For those not in the know about DB repication, one has to make a copy of the master and record the position in its transaction log at that point in time - this way the slave can know where in the transaction log to start repicating from once it comes up with the copy of the data up to that point.

Ok, so, now knowing what I needed to do, I just needed to do it - not as easy as I thought it would be due to the extra layer of LVM.

Fortunately, googling around found me my solution, which I'm recording here for the next time I need to do it :)

Basically, the trick is two fold, frst we have to use kpartx to map the partitions from our virtual disk into partition devices in /dev. Then we have to tell LVM to rescan these devices to find our embedded LVM partition, and activate the logical volumes on there so that we can then mount them as regular filesystems.

lvcreate --size 1G --snapshot --name ss_dbmaster /dev/vol1/dbmaster

kpartx -av /dev/vol1/ss_dbmaster

vgscan

lvscan inactive '/dev/VolGroup00/LogVol00'

[61.84 GB] inherit inactive '/dev/VolGroup00/LogVol01' [2.00 GB] inherit

lvchange -ay /dev/VolGroup00/LogVol00

lvscan ACTIVE '/dev/VolGroup00/LogVol00' [61.84 GB] inherit

mount -t ext3 /dev/VolGroup00/LogVol00 /tmp/dbmaster/

Lets hope I don't have to do that again anytime soon! I just think its cool how unix can have these layers of abstractions between files and partitions, with files in partitions and partitions in files :)

Wireless Shenanigans

noreply@blogger.com (Matt Conway) — Fri, 21 Sep 2007 19:35:00 +0000

To date, my wireless setup at home consists of 2 linksys WRT54Gs modded with the OpenWRT firmware + x-wrt (webif2) for a better web ui. One of these routers serves as my gateway/firewall and the other connects my upper floors to the gateway via WDS (basically, its only there so my Series-1 Tivo can get online as the only mod I have for it is a wired ethernet card :) I also have a server (mythtv) which connects to my gateway via an ethernet cable cleverly hidden under a rug between the living room and study. Something my wife isn't too happy with! :)

Since my macbook pro has wireless N, I wanted to make use of it transferring files to/from my home server. Since my home server is a mythtv box, the files I transfer are usually large video ones, and it would be nice to have 802.11n networking for the speed. Also, my photo printer (HP photosmart 8250) sits above my myth box in the living room, and I haven't had much luck getting it to work from OSX -> myth box (CUPS) - it prints ok, but I can't get OS X to select the 4x6 paper feed when printing photos.

Enter the famed $100 iphone rebate, and I have a recipe for buying the Airport Extreme 802.11N wireless router which has a usb port for printer/storage sharing. My hope is to use it for a WDS link between my myth box and gateway thereby eliminating the cabling, allow a better experience using my printer, and allow me to have a high speed connection to my myth box from my macbook.

Setting it up was fairly painless once I chose the manual setup option - I disabled all the stuff that my gateway already provides (DHCP, etc). I'm happy to report that the printer sharing works really well - basically no different than if the printer is connected via usb! I'm also able to get good speed between my macbook and myth box. My only unsolved issue so far has to do with connecting the airport to my linksys using WDS.

The only way I can get WDS to work between the airport <-> linksys is if I disable encryption on the link. Normally, I use WPA+AES for my wireless and WDS connections, and that works fine for both my wireless connections from my macbook as well as for WDS between the routers. However, no matter which encryption settings I tried (all combinations of WPA1/2, AES, TKIP - didn't try WEP), I was unable to get the airport to connect via WDS to the linksys.

Things left to try:

My Openwrt version is about 6 months old, so maybe an upgrade would help.

WEP - airport wouldn't accept the key I generated from the linksys size, not sure whats up there, but I'm not keen on using something as insecure as WEP anyway.

Until I get encrypted WDS working between airport/linksys, I'm sticking with the ethernet cable.
If anyone has gotten this to work, I'd appreciate some pointers so that I can keep my wife happy (not those kind of pointers! ;)

As a side note, I did notice something interesting with the working (but unencrypted) WDS connection between the airport/linksys. If I configured the airport to have the same SSID as the linksys, my macbook only achieved 802.11g speeds when connected to that SSID. However, if I gave it its own SSID different from the linksys' own, and connected my macbook using that SSID, I was able to achieve N speeds. The WDS link was up and functioning in both directions in either case. Anyone trying to use WDS in a mixed N/G/B setup take note if you care about speed!

Update:

As another side note, my primitive benchmarks (scping a large file from macbook to myth box) yielded the following:

Speed (MB/s)	Configuration
1.5	Connecting to airport with Shared G SSID, WDS, no encryption
3.1	Connecting to airport with Split N/G SSID, WDS, no encryption
5.1	Connecting to airport with no WDS, Radio in n/b/g compatibility mode
9.1	Connecting to airport with no WDS, Radio in only (5GHz) mode

Looks like my wife is going to have to live with that cable for now :)

Create your own ringtone

noreply@blogger.com (Matt Conway) — Wed, 12 Sep 2007 15:03:00 +0000

We just added a cool feature called Make My Ringtone, where you can turn any MP3 file into a custom ringtone for your phone. It’s free, easy to use, and works on all major U.S. carriers including Verizon.

To check it out,
1. Go to www.mobicious.com/ringtones/new
2. Select an MP3 file from your desktop
3. Choose and preview any 20 second clip
4. Send your custom ringtone to your mobile phone

Camping for an iPhone

noreply@blogger.com (Matt Conway) — Fri, 29 Jun 2007 12:05:00 +0000

Yes, I actually am - I think this is the first tme I have ever camped for anything. Not sure if one can call sitting in line for 3 hours "camping", but is as close as I'll ever come :)

For those nearby - the AT&T store on Needham St in Needham MA, only has abou 40 people in line.

Web crawling with Ruby

noreply@blogger.com (Matt Conway) — Fri, 15 Jun 2007 15:13:00 +0000

Learned a little bit about scRUBYt! today (damn thats hard to type!).

It looks like a pretty cool way to do web crawlers. To use it, you define a "learning" crawler using the scRUBYt! DSL, which combined with the actual site at a specific point in time, creates the real crawler to do the dirty work - essentially, example text gets converted to xpath expressions. The cool thing about this is that when the site changes, one has to do minimal changes to create a new working crawler, not to mention that creating the crawler in the first place is a lot easier.

The examples on the site are a little out of date, but here is a simple one which shows the basics. The learning crawler learns how to extract URLs on a google search for 'ruby', but then the production version ends up being able to crawl the urls for any search term. It also automatically crawls to the next page by looking for the 'Next' link.

The learning crawler that you write:


require 'rubygems'
require 'scrubyt'


google_data = Scrubyt::Extractor.define do
  fetch 'http://www.google.com/ncr'
  fill_textfield 'q', 'ruby'
  submit

  link "Ruby Programming Language/@href"
  next_page "Next", :limit => 2
end

puts google_data.to_xml

Becomes for production use:



require 'rubygems'
require 'scrubyt'

google_data = Scrubyt::Extractor.define do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit

  link "/html/body/div/div/div/a"

  next_page "Next",:limit => 2
end

google_data.to_xml.write($stdout, 1)

And outputs:



  http://www.ruby-lang.org/
  http://www.ruby-lang.org/en/20020101.html
  http://en.wikipedia.org/wiki/Ruby_programming_language
  http://en.wikipedia.org/wiki/Ruby
  http://www.rubyonrails.org/
  http://www.rubycentral.com/
  http://www.rubycentral.com/book/
  http://www.youtube.com/watch?v=JMDcOViViNY
  http://www.w3.org/TR/ruby/
  http://www.zenspider.com/Languages/Ruby/QuickRef.html
  http://poignantguide.net/
  http://www.rubynz.com/
  http://www.ruby-doc.org/
  http://tryruby.hobix.com/
  http://www.rubycentral.org/
  http://www.gemstone.org/gem-by-gem/english/ruby.html
  http://whytheluckystiff.net/ruby/pickaxe/
  http://intertwingly.net/blog/
  http://lotusmedia.org/
  http://www.oreillynet.com/ruby/

One of the authors showed me a good example which I don't think is linked to from anywhere yet, so here it is in case you are interested.

We'll see if this lives up to its promise in real world usage, but looking good so far. One major drawback is that it breaks on heavy AJAX sites e.g. clicking a paging link which actually loads the next page in place using an ajax call. However, one can usually find a non-ajax workaround to get to the same data. The authors are aware of this and planning on integrating somethign like FireWatir to allow it to handle ajax/dhtml requests in a seamless fashion.

Custom loggers in rails

noreply@blogger.com (Matt Conway) — Fri, 15 Jun 2007 14:22:00 +0000

I ran into some problems trying to setup a custom logger in rails which I thought I would share to ease someone else's pain. Everything I read said that to setup a custom logger instance, I should set config.logger in my environment.rb, e.g.


environment.rb:

config.logger = MyLogger.new(config.log_path)

However, I just could not get this to have any effect! It turns out that this was because I was starting up my server using script/server using mongrel. Using "script/server webrick", or "mongrel_rails start" worked just fine.

I think this is caused by code in the mongrel.rb file that script/server uses which loads just the logging environment (but not my environment) in order to figure out which log file to tail


railties/lib/server/mongrel.rb:

require 'initializer'
Rails::Initializer.run(:initialize_logger)

Loading the environment here causes the RAILS_DEFAULT_LOGGER constant to get set, and thus when my config finally gets loaded when mongrel itself starts up, the config.logger I set gets ignored as the rails init code does nothing if RAILS_DEFAULT_LOGGER is set:


railties/lib/initializer.rb:

def initialize_logger
  # if the environment has explicitly defined a logger, use it
  return if defined?(RAILS_DEFAULT_LOGGER)
  ...
end

The patch on this bug report seems related, but didn't fix my problem.

I'm too much a rails newb to figure out the right way to patch this, hopefully someone with more of a clue will see this blog entry and submit a patch :)

Life at a Startup (Mobicious)

noreply@blogger.com (Matt Conway) — Mon, 11 Jun 2007 05:42:00 +0000

So I've been working at Mobicious for a week now, and the task list just keeps getting longer :) Not really surprised by this, as it is a startup. Is actually kinda cool to have this much to do, especially when most of it is interesting. I guess that makes me a workaholic. :)

Of course, being mostly a Java dev at the last job, I'm having to do a lot of reading and catchup on Rails. We are using Ruby on Rails at mobicious and I can see no overwhelming reason to change this. Scalability will be a concern eventually, but from all I've read, Rails can scale, just not quite as well as Java yet. This is not surprising given their relative levels of maturity. If that hasn't changed by the time we need it to scale hugely (here's hoping), we'll re-evaluate then. One option will be to switch to java for running rails, but to do so we'll have to come up with alternatives to binary rails plugins like RMagick. For now though, I'm sticking with Rails as it does give us an advantage in development time - though not quite there for me personally yet :)

So far I'm really liking the startup environment, and the people are all really great, the work is interesting, and I'm having fun!

Farewell TripAdvisor

noreply@blogger.com (Matt Conway) — Wed, 30 May 2007 11:37:00 +0000

Today is my last day working for TripAdvisor, where I've spent the last 3+ years working as a senior software engineer with a very talented group of people. I'm going to miss working with everyone here, but the lure of the startup is just too great :)

Next stop, is Mobicious, a startup in the mobile space near Waltham, MA (office location still undetermined). I'll be employee #4, so fun times ahead. Stay tuned as my adventures unfold.

Switched to wordpress

noreply@blogger.com (Matt Conway) — Mon, 21 May 2007 19:59:00 +0000

Finally got tired of messing with pebble, and decided to give wordpress a try. Much better! Only problem is that my permalinks have all changed. Hopefully I'll figure out a way to map the old permalinks into the new layout - one of the wordpress custom formats gets close, but instead of the leaf being the pebble article id, its the article title - which I do prefer. I guess I'll have to add in custom rewrite rules for all the old ones. Anyone else have a better idea?

genlibs.sh

noreply@blogger.com (Matt Conway) — Mon, 21 May 2007 19:30:00 +0000

For generating eclipse library files from thirdparty directory structure

About

noreply@blogger.com (Matt Conway) — Mon, 21 May 2007 18:52:00 +0000

Matthew Conway is a software engineer currently working in the Boston area for a mobile community/directory website.