Ruby
Speeding up Ruby on Rails’ ActiveRecord INSERT rate
0A project that I’m working on (OK, it’s TSDBExplorer) generates a metric shedload of database rows. For a record that says “This train runs between 01-01-2011 and 31-05-2011 on Mondays – Fridays”, the code generates a timetable for each day. It takes an age to import, and I hope it’s going to be fantastically quick at querying data.
There’s a big downside with ActiveRecord out-of-the-box – it takes a long time to INSERT a record in to a MySQL database. I left some INSERTs going at 9.50am, and they’d just about finished when I got back from the gym five hours later. 1.2 million rows in five hours is shockingly poor.
activerecord-import appears to solve the problem in the least impact way. To group up your INSERTs, you create new instances of a model object – say, Association. You push these in to an array, and then use the new ‘import’ method to do a mass INSERT.
I am quite happy at 62 minutes to insert 1.2 million rows, including processing, considering it’s an activity that only needs to be done twice, maybe three times a year.
Converting OSGB36 (Eastings/Northings) to WGS84 (Longitude/Latitude) in Ruby
8The excellent people at the Greater London Assembly have released a list of bus stops and bus routes in London. The coordinates of each bus stop are in eastings and northings, and I wanted to convert these to longitude and latitude for my Ruby on Rails application.
Using the proj4rb gem and some projection definitions from spatialreference.org – with some help from Harry Wood’s blog, I came up with the following code:
#!/usr/bin/ruby
require 'rubygems'
gem 'proj4rb'
require 'proj4'
easting = 529978
northing = 186491
srcPoint = Proj4::Point.new(easting, northing)
srcProj = Proj4::Projection.new('+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +datum=OSGB36 +units=m +no_defs')
dstProj = Proj4::Projection.new('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs')
dstPoint = srcProj.transform(dstProj, srcPoint)
print "lat=#{dstPoint.lat * (180 / Math::PI)}\n"
print "lon=#{dstPoint.lon * (180 / Math::PI)}\n"
To convert WGS84 to OSGB36:
#!/usr/bin/ruby
require 'rubygems'
gem 'proj4rb'
require 'proj4'
latitude = 51.5623279577278
longitude = -0.126277004538848
srcPoint = Proj4::Point.new(longitude * (Math::PI / 180), latitude * (Math::PI / 180))
srcProj = Proj4::Projection.new('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs')
dstProj = Proj4::Projection.new('+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +datum=OSGB36 +units=m +no_defs')
dstPoint = srcProj.transform(dstProj, srcPoint)
print "easting=#{dstPoint.x}\n"
print "northing=#{dstPoint.y}\n"
Distributed Ruby
0A colleague of mine, Brian Candler, showed me the sheer power of Distributed Ruby (DRb) this week.
Wow. I am impressed – marshaling objects between separate processes, automatically executing actions on the right process – I like it.