- Make sure Macbook has 4Gigs of ram.
- Purchase VM Fusion or find a reasonable free alternative
- Create a Virtual Machine that copies as much of your actual production setup as possible. I happen to deploy on Ubuntu and CentOS. I do most of my development on the Ubuntu Server VM.
- I often have branches I have to support. These VM’s let me load up a system identical to that branch.
- You can probably get by with giving these VM’s 256 meg of ram and 10 gigs of space. Also, you don’t have to have multiple running at a time. So system resources most likely won’t be an issue to most of the readers.
- Take advantage of cloning and snapshot features of the VM. This really should be a part of every developers toolkit.
- You can mount the directories on the VM of you want to use OS X tools such as TextMate for editing. I used to do this, but eventually caved into to using Vi as I am SSHing into different servers all day.

Every one has a different situation. I wish I used this setup earlier in my career. With this you can still have the look and feel you love about OS X while not giving up an environment that resembles your production server.

Well it’s been a while since I’ve last posted, I’ve been busy,blah blah blah…And now to the good stuff.

In my new position as a Server Lead for a start up, I have come up with some interesting challenges. I decided that to break away from my love of the RDMS and go with something that scales horizontally…with out breaking. I’ve spent the last week reading up and watching videos on Hadoop. I think it would be interesting for others to see how my transition is going. I’ll try to keep this blog updated, not only for others, but as a reference to myself on useful links, tips, and explanations.

To start off I would like to explain exactly what I plan on accomplishing. Basically I have a RESTful web service that is sending in protobuf serialized data. Currently I use a django web service to with a few mappers to massage and validate the data a bit and throw it right into Postgres. I think will need to access this data for historical purposes, so something similar to web analytic reports, as well as few looks at the freshest data available.

In the Hadoop world I thought ‘HBase’ would be a good candidate for the fresh data access I need. Where as Hive seems to be more of the normal data warehousing/report building framework I would need.

Because I’m so new to Hadoop, I’m in days ahead, if not hours, I’ll look back at the horrible mistakes I’ve made and correct these entries accordingly.

Over the weekend I read the ‘Hadoop: The Definitive Guide’ by Tom White. I highly recommend it. It actually is a fairly easy read and can easily flip around to chapters, i was more interested in HBase than the details of MapReduce, so I read that chapter first.

Tom White is a consultant at Cloudera. Their site has the best reference material that I have seen for Hadoop. They also have a distribution you can download. It is a nice collection of various packages of Hadoop. But what is even better, they have a VM you can just throw up and get to testing Hadoop in minutes after the installation.

Today what I hope to get done is get the VM installed here on my work macbook, use osmosis to get some real data into the HDFS, and write some code to mimic some of the aggregation queries I’m doing in SQL right now. I’m not sure if I will use Hive today for this, but we shall see. I’m now going through this blog (http://blog.lars-francke.de/2010/07/22/processing-openstreetmap-data-with-hive/) which has already pointed me to a few useful tools for getting data into HDFS from postgres.

This is an exciting time for me. I love learning new things.

http://blog.wekeroad.com/2009/11/27/paging-records-sucks–use-jquery-to-scroll-just-in-time

I wanted the same effect but within a Div. I found out an easy way to get to the bottom of the Div with this guys post:

http://yelotofu.com/2008/10/jquery-how-to-tell-if-youre-scroll-to-bottom/

I combined the two to have some pretty cool stuff going on.

Just replace Rob Conery’s scroll event code with this:

$(function() {
$('#dataContainer').scroll(
function() {
var elem = $('#dataContainer');

if (elem[0].scrollHeight - elem.scrollTop() == elem.outerHeight())
{
loadMore();
}
});
});


You just need a div named #dataContainer. Everytime the user scrolls to the bottom it will call loadMore(). What I do in load more is JQuery prepend() to add rows in a table inside the div. Pretty cool all around. Sorry for the crappy wording and bad examples, it’s 3:53 A.M here, but wanted to share this.

ClosureAPI.com

$.GetJson(url,printResults,textStatus);

Well what if you need to send additional information to printResults() ? I’ve been coding C# to long. I vaguely remember events before delegates and I haven’t made a function pointer since freshmen year in college. I’m not actually sure if there is a way of sending extra parameters to an event like this, but my solution was just to use an anonymous function before hand.

So something like this:

$.getJSON(request, ajaxData,
function(data, textStatus) {
printResults(extraDataToSend, data, textStatus);
}
);

You probably are only getting this problem if you are doing some ajax stuff, so anyone else wont’ be affected. The inputs still get updated with correct values, just not at the time this event is fired.

Traditionally JavaScript programmer would hook most of their code in an onload event in the <body> tag of the HTML document. This would insure a way to hold off executing of the script until the DOM was finished loading. Unfortuntely this also waited for all the exeternal media on the page too load also. This resulted in not so smooth page loading, as you have to download all the images before the script and start executing.

JQuery solves this by executing the JavaScript after the DOM is finished but before the media has finished downloading, resulting in a smooth gradual evolvation of the page.