For those who know me (hi people on facebook) you might know that over the past few weeks I have been slowly making a gallery in Rails for my personal photos. This started off the back end of being annoyed at Flickr’s free account only allowing you to create 3 sets (or albums) before you had to pay for more. This annoyed me as I do like the Flickr layout, but I’m not really prepared to pay for it just yet.

So instead, I decided to make my own Flickr clone, and this has been aptly named Snapr!

I started off by looking through their design, seeing how they spaced and laid things out, but one thing in particular that interested me was how pictures seemingly look better on Flickr. Now of course, its not just because of its design that compliments images and that’s enough to make them appear to be better. But as stupid as it sounds, pictures on Flickr do look better, it took some research and testing to finally work out how they do it, but here it is.

After digging through the code in the Mac Flickr Uploader, which is quite handily open source, it was just a case of attempting to follow the code through to work out what they did. They use GraphicsMagick for image manipulation and this is the code that makes images look great…

// Find the sharpen sigma as the website does
double sigma;
if (base <= 800) { sigma = 1.9; }
else if (base <= 1600) { sigma = 2.85; }
else { sigma = 3.8; }

// Create the actual thumbnail
img.scale(dim.str());
img.sharpen(1, sigma);
img.compressType(Magick::NoCompression);
img.write(*thumb_s);

This is the last part of the generation of thumbnails, and as you can see, it scales down the image, then sharpens it by this magical sigma amount. To replicate this I had to actually play around (for quite a while) with that sigma figure so that I could make a like for like replication of their images. Obviously they don’t compress the thumbs, as if you do, they look horrible.

Now initially, this was easily to replicate in Rails with RMagick, however when I switched over to JRuby there isn’t a RMagick gem, which is more than a slight annoyance. After some help from a friend I had my own implementation of pure Java for image manipulation, and this worked fine. However I had been using the attachment_fu plugin to make uploading and resizing of images a breeze. (Read: if you aren’t using attachment_fu, use it!). So the next step, how to get this new Java code to work with attachment_fu. 2 evenings worth of playing and testing later (and my complete lack of any real Java knowledge) I admitted defeat. So, at this point I have a fully working gallery, minus image uploading and processing, which as far as galleries go, meant it pretty much useless. What was I to do next?

Some Googling later and ImageVoodoo was my knight in shining armor! ImageVoodoo is a JRuby gem which has like for like API methods for ImageScience (an image processor supported by attachment_fu). Everything looked like the gallery could finally begin working again. However the next stumbling block was going to appear, ImageVoodoo doesn’t support image sharpening. I was wondering at this point just how much more annoying this could get; this project was only meant to be something quick to get me back into writing Ruby again, but was quickly turning into a complete headache. After some more eventual playing I again had to admit defeat, not so much through not wanting this to work or lack of knowledge, but the want of a working gallery was greater at this point.

Where am I at now then? Well, the gallery is operational, but image sharpening & image compression are out at the moment. So thumbnails look like a 4 year old has been let loose with Photoshop and somehow managed to find the resize controls but on doing so, decided to lower the quality to about 10. Meaning that my lovely photos don’t look as good as they could at this moment. However with attachment_fu storing the original image all is not lost just yet, as when I finally figure it out I can simply regenerate the thumbnails.. (well, I hope I can, haven’t actually looked into how to do that yet).

It baffles me however that RMagick, which is quite frankly a huge part of Ruby image manipulation, doesn’t have a JRuby gem. Admittedly there is RMagick4J, but it’s lacking the same problems as ImageVoodoo, no sharpening support or image compression controls. If only RMagick worked on JRuby without having to think about it…

This is sometihng that baffled me for a good few months until I finally figured out what the hell was going on. If you develop in rails chances are you’ll either be using Mongrel or WEBrick to run it locally, but sometimes when I went to start either of them I would get a “This port is already in use”, which wasn’t exactly fantastic.

I would look around to try and find out what was using it, but “netstat -a” doesn’t help much, it just says its taken by a local device on the machine, although it wasn’t just port 3000, it’s the entire range of 3000 to 3030 were “in use”.

Some days it would work, some days it would fail, from what I could tell it was completely random whether I would be able to start up the server. After senses of humours were lost to the pixies I finally worked it out, and it comes down to a little green icon with two arrows…

It’s ActiveSync, and you’ll have this program running if you have a PocketPC phone and if the phone is connected, say goodbye to those port numbers. It steals 3000-3030 straight away when you plug it in. Some solutions I have found are to unplug my phone, start Mongrel, then plug the phone back in. Although again, this working is random and highly frustrating. Currently I have resorted to unplugging my phone when I am developing in rails and plug it back in when I’m done.

I actually figured this out sometime last year, but saw someone come into #rubyonrails having the same problem and I thought I should probably share it.

This was more of a test to see how easy it would be to scrape some data using Ruby as I usually use PHP to do this for me. However when using PHP it always seems to be a mess of explodes and regexp’s to get what I want. So I wanted to see how other languages do it.

So first things first I found out which packages/libraries were around for Ruby, and admittedly there’s quite a few, however upon first look what appear to be the better ones don’t work out of the box. Which is a big shame, as I spent a good few hours trying to get scRUBYt! working on my Windows XP development machine, but in the end I had to admit defeat. It just wasn’t going to work properly even though I had followed every guide you could find to get it working in a Windows environment. I was a bit disheartened as this appeared to be the best one out there as its first example was exactly what I was looking for.

Alas, I moved on, had a quick look at Hpricot, but it didn’t seem to do what I wanted easily. After trying to find some examples of how they all worked I found srcAPI, which was very similar to scRUBYt however no matter how much I tried, I could only find one example, which was its own eBay one. It had a nice installation, just a gem, which worked first time (which one usually expects), from there I tried their example. I had read on their website that it uses Tidy to do the HTML cleaning, but you could use it without that if you told it to use the built in one, however it warns you that this should only really be used for testing, and and that was what I was doing I told it exactly that.

Scraper::Base.parser :html_parser

To cut to the chase, nearly 3hours, much googling and frustration later and the simplest of process’s it was refusing to scrape part of a table I already had. Telling it to use the built in one instead of Tidy was my downfall, the second I switched it back to Tidy (and a quick re-ordering of the loading code to stop it trying to load the linux version first) and it was working perfectly. This was someones comment about it all…

Phases of scrAPI usage:

1. Elation - Wow, this is so easy and powerful. I’m gonna scrape the world!!!!!

2. Despair - What the hell is the syntax for the selectors, I’m so confused, and there are no docs

3. Elation - scrAPI has great test coverage, you can learn everything you need to know about the selectors from the tests.

Couldn’t agree more, although instead of getting help from the test cases I got help within a scrAPI cheat sheet I found. Without that I would have probably given up and tried the next one in line.

So I’m now testing and doing more complex scraping with srcAPI, but for anyone who is a beginner to Ruby you may find this task quite challenging and probably beyond your scope of knowledge to begin with. The lack of documentation and examples for this made it increasingly hard, and most was done with guess work on how it was interpreting the HTML source code.

Bottom line, if you can get scRUBYt! working, go with that, it’s very powerful and from the examples I have seen, very programmer friendly and will allow you to get to the data you want fast. But if you can’t get it on the go, then there’s many others available including many I haven’t mentioned, but Google knows all! (apart from examples of srcAPI :P)