Speeding up Ruby on Rails' ActiveRecord INSERT rate

A project that I’m working on (OK, it’s TSDBExplorer) generates a metric shedload of database rows. For a record that says “This train runs between 01-01-2011 and 31-05-2011 on Mondays – Fridays”, the code generates a timetable for each day. It takes an age to import, and I hope it’s going to be fantastically quick at querying data.
There’s a big downside with ActiveRecord out-of-the-box – it takes a long time to INSERT a record in to a MySQL database. I left some INSERTs going at 9.50am, and they’d just about finished when I got back from the gym five hours later. 1.2 million rows in five hours is shockingly poor.
activerecord-import appears to solve the problem in the least impact way. To group up your INSERTs, you create new instances of a model object – say, Association. You push these in to an array, and then use the new ‘import’ method to do a mass INSERT.
I am quite happy at 62 minutes to insert 1.2 million rows, including processing, considering it’s an activity that only needs to be done twice, maybe three times a year.

Doubled Sided Printing

Having run out of blank A4 paper and needing to print something, I decided “Hey, I’ve got 24 sheets, I’ll print this 40 page document double-sided!”
Why is it so difficult for me to get my head around how to do this? There are many ways I could screw up – pages back to front (printing odd and even pages on the same side), pages upside down (printing odd pages in one direction, and even upside down on the back), printing the odd pages in order, but the even pages in reverse order (the first sheet having page 1 on the front, and page 39 on the back), offsetting the pages by one sheet…
I only misprinted four pages. I call this a success 🙂

Virgin Media in stealth Ofcom marketing tactic

The BBC are reporting that “Ofcom wants to ban misleading broadband speed ads”.
All well and good, but marketing a service as “up to 24Mbps” makes many people believe they will about 24Mbps. In reality, there is negative correlation between the length of a phone line and the speed you’ll get from ADSL. Is there a widespread user perception problem, or is it just down to marketing? (Remember the “Up to 50% off!” adverts you see on the high street – you may not realise that everything may be 10% off apart from one item which is discounted by 50%…)
What bothers me is that Ofcom claim Virgin Media – who provide a much different service using a wholly different technology – provide speeds much closer to their “up to” figures. This is wholly wrong – Ofcom are not comparing apples with apples. Virgin Media’s service is dissimilar to ADSL, with more equipment closer to users’ locations, and fewer limits on the amount of power they can shove down a piece of co-axial cable.
The BBC article goes further to make it look like your ISP has a lot to do with your “broadband speed”. The reality is that all ISPs using BT Wholesale’s DSL infrastructure for a phone line will get about the same speeds across that line to the exchange, but depending on the level of oversubscription and contention in their network, their customers may not reach that speed.
Is this clever stealth marketing from Virgin Media? Have Ofcom forgotten that Virgin are advertising their service as “fibre-optic broadband”, when it’s really a fibre-optic backbone and copper cabling to your house? Fibre is not broadband – it’s a very narrow, specific range of frequencies. Have BT, who are advertising their “fibre to the cabinet” (FTTC) service as “Infinity”, forgotten that there is no number greater than infinity, and they’ve just shot themselves in the foot marketing-wise?

Seven days of 3

A little over a week ago, feeling horribly overcharged and dissatisfied with Orange, I terminated my contract and moved over to 3.
The process was incredibly smooth. I ordered a 3 SIM, which arrived the next day along with a free PAYG SIM. Orange sent me my PAC. I called 3, gave them my PAC – but had the usual overjoyed offshore callcentre guy thank me far too much for moving – and two days later, my number moved.
I have had one problem with data in the last week, and I was in a low-signal area and had to restart my phone. I could get a 3G signal when in the middle of nowhere in Kent yesterday, and browsing when in London is really very quick and responsive.
I made a good choice, and it’s £30/month cheaper than Orange.
I am on a train at the moment, updating my blog using my 3 PAYG data connection, which is equally fast and a very usable tool for when I’m out and about.
3, you’ve made me happy.

Goodbye Orange, Hello 3

I can’t remember exactly how long I’ve been with Orange for, but it’s been something in the order of a decade. That’s a long time.
I remember the days of Wildfire, withdrawn because they didn’t develop it and people stopped using it. I remember HSCSD, where I could get a blazingly fast (for the day) 28.8kbps connection out of my mobile. That’s been superseded by GPRS, 3G and UMTS. I remember, and loved, Everyphone, where I could divert all my incoming calls elsewhere if my battery ran out – but that’s just for business users now.
I remember being with a mobile phone network that was different, innovative and fun.
Now they’ve gone downhill in my view. They are slow to offer new phones, and they put special Orange firmware on them which means months of delays getting updates. Their GPRS service was good, but their 3G service is abysmal and I can’t even connect half the time (and I can tell when that is now – Android 2.3 shows the H or 3G icon in green if the connection will give you Internet access, and in grey if it isn’t) despite having good signal coverage. They proxy and filter the life out of any Internet connection you do get, to the point at which proxying my web traffic over an SSH tunnel through their network is *faster*. They tease me with Orange Wednesdays, but I pay a lot for the privilege of half-price cinema tickets. To top it off, they took away my loyalty bonus when I dared upgrade my phone in-store because “the loyalty bonus only applies when you upgrade online” – why!?
Orange, you’ve had your day, and unless you suddenly manage to fix your 3G service for me and reduce your prices drastically, I’m using my “PAC code” and going to a 1-month rolling contract on 3 for a mere £10/month.

Android, Facebook and the undialable phone numbers

For some length of time, I’ve had problems with my Nexus One and the Facebook for Android application. Phone numbers from Facebook are imported in to the Android Contacts application with a country code but without a leading +, for example – 447700952155 rather than +447700952155. This makes them rather difficult to dial, as your mobile carrier won’t recognise them.
If you have this problem, try http://www.253below.com/prefixer (or search for Prefixer on the Android Market – find the 253 Below application). Set up a rule to match 44[0-9]+ and tell it to add a prefix of + to the number. Hey presto, you can now dial successfully.

Customer Service

I wouldn’t mind if I could just ask a question and get a reply….
UPDATE – I strolled in to a 3 store, asked “Do you sell the MiFi?”, and was told “Yes”. I bought one and was back on my way within five minutes – the way it should be!
I want to get a 3 MiFi so I can get on the Internet for a semi-sane price (Orange charge a small fortune, and their 3G service rarely works on my phone, if I can actually get 3G that is) whilst I’m not at home. I choose the product on 3’s website and click ‘Support Chat’ to see if I can buy it in-store. Y’know, where you walk in to a store and buy something, and have the gratification of an instant purchase, as well as the product when you need it.
Here’s our conversation…
[Peter] Can I buy the MiFi PAYG + 3Gb product in-store?
[Srinivas] Hi Peter.
[Srinivas] Welcome to Three.
[Srinivas] I’ll be glad to help you with the purchase of the deal query and thank you for choosing Three.
(this is not answering my question)
[Srinivas] I would not be able to comment on the store deals.
[Srinivas] I can help you purchase this deal online with a free gift for today’s online purchase.
(purchasing things on-line is beyond me, I find it reeeeally difficult)
[Peter] I need it today
[Srinivas] I’m sorry if y ou order now , you would get the delivery the day after tomorrow.
(this it not ‘today’)
[Peter] OK, I will find my nearest 3 store and ask in there
[Srinivas] So would you like to go ahead with the order online?
(……..)
[Peter] NO! I need it today!
[Peter] If I order online, I will have a free gift that I don’t want, and will have no means of Internet access for the next four days when I am away
[Srinivas] You can check at the store .
(Great, I can check in-store to see if they sell something in-store. Who’d have thought?)
[Peter] I was hoping I could check with you now to save me a wasted trip in to town.
[Peter] Thank you for your help – I will put some clothes on and find my nearest 3 store and ask in there instead.
(I am fully-clothed actually, I just wanted to vent some frustration)
[Peter] Have a good day, and thank you for your time
[Srinivas] http://www.three.co.uk/Help_Support/Store_locator/ Click on this link to find the nearest store.
(JEEZ, YOU THINK I’D JUST TAKE A RANDOM BUS AROUND LONDON AND STOP WHEN I SEE A 3 STORE?)
[Srinivas] Anything else I can help you with?
[Peter] No, that is all – thank you
…so I’ve wasted 10 minutes chatting to him just to find myself in a situation no better than before.
Grr.

Jobs

I am now ‘between jobs’, as some unemployed people say. Or am I? I have some short contracts lined up – a few days here and there working on specific tasks. Is that really unemployed or not?
To celebrate the occasion, I bought a MacBook Pro. I am mightily impressed, particularly at the sheer speed of the SSD and ease of use, but I have been a Mac user for the past four and a half years and only used Windows under duress or by requirement.
One thing puzzles me – after hardly touching Microsoft Word for the best part of four years, why is it still largely the same as it was in the mid-90s? Are all the new features generally invisible? Are they so tightly integra… no, actually, I probably don’t need most of the features that are in there.

A problem isn't always what it seems

I had a call today from somebody who was trying to use an Avaya IP Phone from their office at home. The phone has built-in IPSec VPN capability, and their phone switch is some distance away on – surprisingly – a DSL line. Call quality is not an issue, which is testament to Zen Internet‘s network quality.
In the office, the phone works fine. At home, it fails to establish a VPN, displaying an “Invalid PSK” error. Looking on the VPN router at the site with the softswitch, I see nothing untoward, so I set about asking the caller to tell me his default gateway address – which is the same address as the network range that the softswitch is no. No prizes for guessing why it doesn’t work.
Here I was, expecting a full-on IPSec debugging session, and it turns out to be IP addresses.

Open Transport Data

The Guardian published an article on National Rail Enquiries’ refusal to be sensible about licensing its data. Malcolm Barclay has mused on NRE’s inflexibility, claiming “They are stuck in the command & control mentality of the industrial age and have zero understanding of what open data is or it’s benefits”.
This whole debacle is reminiscent of Eric S Raymond’s “The Cathedral and The Bazaar” for me – where
TfL have had no problems making their data available free of change, and they’re working really hard to bring the Trackernet service back to life. Emer Coleman, Director of Digital Projects at the GLA, posted “But you can be assured that it definitely will be back and hopefully the solution will be so robust there will be no chance of it falling over like it did the last time. That is TfL’s main concern that once it foes back its there for good and in a robust way”. Hats off to them – TfL were caught by surprise with the popularity of the Trackernet API, and they’re tackling it head-on.
If TfL can be this innovative and forward-thinking, why can’t NRE?
NRE’s jealous data-guarding is not just limited to real-time train running information. If you want daily-updates of fares, timetable and routeing guide (the official definition of the routes you can use with ‘Any Permitted’ route tickets – a complicated beast that few people properly understand), you will need to part with £27,430 according to ATOC’s RSP Data Feeds document. That’s a ludicrous price that serves only to lock data on our country’s rail system away from prying eyes. Heck, it’s £600 if you want a CD with test/trial information.
This data is probably of most value to companies who are deeply involved in selling tickets, such as TheTrainLine and those who