Some things I was aiming for:

1. Set them up with normal migrations.
2. Test them with the normal test suite/normal fixtures.
3. Make recovery/reset simple for when the table (inevitably) is somehow out of sync.

The “frequent counts” table

I’ll have multiple counts but not TOO many — enough that I don’t want to have a column per count but not enough that I mind using “LIKE” to lookup patterns, so my table has: id, code, current_count.

Code will be a unique key (important later) and be formatted like “style_ABC_size_456″.

So, when a new item is added it’ll be associated with a style and some sizes – each combination will either need to be setup (with a current_count = 1) or an existing combo will be found and +=1.

The FrequentCount class has the fairly straightforward finders that you’d expect + methods to reset each of the counts that it contains.  The reset methods follow the pattern “reset_frequent_count_COUNT_NAME” -> they clear the existing counts that they maintain before repopulating them.

I also threw in a reset_all method that looks for anything on the class following the “reset_frequent_count_COUNT_NAME” pattern and runs them.

The trigger-SQL

The SQL for creating the triggers will be needed by the migration as well as the test suite.  In fact, the test suite will need to run them somewhat often due to the way the standard tests ”prepare” the database.

I ended up throwing it in lib/trigger_sql.rb.  Methods there are named with the pattern “sql_for_TABLE_OPERATION_TRIGGER_NAME” ex: sql_for_items_insert_style_and_size

create trigger items_insert_style_and_size after insert on items
for each row
begin
insert into frequent_counts(code, current_count)
values (concat(‘style_’, new.style, ‘_size_’, new.size), 1)
on duplicate key update current_count = current_count + 1;
end;

# find methods that follow our pattern of “methods providing trigger sql” and execute the contents of each

TriggerSql.methods.select{ |m| m =~ /sql_for_.+/ }.each do |method_name|

ActiveRecord::Base.connection.execute(TriggerSql.send(method_name))

end

Night one went something like this:

I got an invite (did they do invites?) to checkout Heroku really early on.  At the time, I remember devoting a night to it – working with an online editor (or maybe I was ssh’d in with vi or something?) and basically just getting a little “Hello World” up and running.  Ever since bumping into Jim Fiorato’s app/case study the other day, I’ve been meaning to revisit a bit.

It’s entirely likely that the friend’s site will actually not need anything but static HTML, but hey: I can always tune the caching and free hosting can’t be argued with, right?

So, I opened an account out at herku.com and grabbed the heroku gem; installed git (officially taking me off the short list of people that still hadn’t given it a chance) and set to work on getting my app out there.

I hadn’t written anything, so I was just wanting to throw the “welcome to Rails” app out there to make sure everything was setup correctly.  The first bump was some fun with SSH keys.  I’m actually still not sure exactly what the issue was but there seemed to be some commands that were respecting the path to the key that I had setup but some others that seemed to be looking in the default location (~/.ssh).  I am actually thinking now that I probably could have got around it with a little more effort put into the config but I ended up just using the default key location — no probs after that.

At that point + some very simple/standard Rails app config, it was incredibly early to push the app out there, setup a few tables (via migrations) and get to work.

A little effort into a pretty basic layout and we’re underway.  It’s nothing spectacular to checkout at the moment – generic copy, placeholder colors/blocks and text… but someday it’ll be a star.

Google Docs is great but let’s face it, the best part about it is plain and simple: It groks .docs (and xls and whatever else).  If it didn’t… I imagine I might have a few docs out there — some random things people “shared” with me and what not, but add in the ability to handle .doc files and it becomes an essential tool for anyone lacking a copy of Office.

So, Microsoft’s announcement that Office will be available FREE online is a Goog Docs killer, right?  … nah.  I really think it would be but the thing about “Free online Office” is… it’s only free if you’re already paying… and while 400M people are — YOU, my friend, are not.

So, what will it take for this to be a success amongst the 2 groups of people I care about?  Let’s label them (1) Me and (2) My Uncle Rob.

Me.

I love having things online and everything that goes along with it, but… gotta be free.  Google’s online offerings are far from having perfect interfaces, but as any good 36-signals-following kid will tell you: free + simple = winner.  I just can’t justify paying $50/yr (to make up a number) to have Word online.  I know it’s $4 a month… but sorry: I’m out.

Uncle Rob

Uncle Rob doesn’t really get the whole online thing.  That’s not to say he doesn’t use/like technology.  or that he doesn’t like his Y! Mail and playing some online Euchre.  It literally means: he doesn’t know what it means to have documents hosted online.  If you give him Office online, he’s going to wonder why he can’t go to his harddrive to copy all his files to his fancy new flash drive.

But here’s the thing: he wants Office.  He doesn’t want something that can read and write doc files via “Save As” (or even worse “Export As”.  He just wants it to work – and look like the thing he uses at work.  THIS is why this whole plan may work.

So, what will it take to make me recommend it to him?  … gotta be free.  If he has to pay $50/year — no question about it: I’m telling him to have his kid buy him the $150 Student copy for his own machine.  It’ll be easier for him to understand and it doesn’t force him to send all his Uncle-Robby-Friends “share links” or use that pesky “Export As”.  If the prices are even close (and they really are), it’s the obvious choice for him.

Maybe they’ll figure these things out?  … or maybe they won’t AND the interface will suck.  It’ll be interesting to see how things look when we actually get closer to the release.

10 minutes later I received a text from someone not in my contacts list. Upon further inspection, it was a good friend of mine (who I frequently trade messages with) — what the heck? His number isn’t in my address book?

A quick check of my laptop revealed that he was still there but wasn’t properly being synched to my phone. It wasn’t just him – 25ish contacts were mysteriously not showing up properly. Editing individual entries and resynching got the job done but finding these 25 people was really getting annoying – I needed a truer solution.

The winner:
iSynch -> Preferences -> Reset Sync History

That, plus one more “synch” through iTunes – and I was back on track.  Hope that helps if you’re in that same boat!

Much much better.

Ah, what can be prettier than a screen full of baseball stat projections!  The print is a bit small but that’s by choice: make sure to play with the settings when you grab this thing.

My fantasy draft preparation always involves the creation of a spreadsheet.  This nugget of gold will start with ratings and stats from various sites and sources before getting my own personal notes, organization, and shuffling (aka “messing it up”).  and what kind of card-carrying software engineer would do such a compilation by hand…

My basic strategy for compiling this sucker this year:

1. Scrape some stats and rankings from a few popular sites and dump them locally.
2. Load the stats from local, merge em, and kick out a csv.
3. Mess with stuff in the spreadsheet.
4. Win millions.

Scraping

I’ve used hpricot in the past but wanted a bit of a refresher — Man, this thing makes this task nice and easy.  Most of the sites have a fairly sane markup scheme for the tables they store players data in, so it’s normally as simple as (this is Ruby btw):

doc = Hpricot(open(url))
players = doc.search(".playerDataRow")
players.each do |player|
  meta = player.search("td .playerMeta")
  stats = player.search("td .playerStats")
  < do stuff >
end

Local Storage

Last year, I wrote this as a single script (scrape -> csv). Things got hairy when I needed to tweak the merger and thus had to re-run the whole thing (annoying) or hack it to run only a portion… and the same portion on each site (annoying). So, this year I got a little smart and wanted to dump the scrape results locally.

I fully intended to evaluate a few options here but as it turns out, the first try was just dead simple and worked perfectly: yaml. My scrapers each end up with a couple Arrays: one for hitters and one for pitchers. Each entry is a Hash of player data. I considered skipping the Array here but didn’t want to have the logic for name collisions in the scraper.

How hard was it to write my +player_data+ Hash to yaml?

YAML.dump(player_data, File.open(File.dirname(__FILE__) + "/yaml/" + filename, "w"))

Merging the scrape sources

This is a separate script here now…

Resurrecting the player data proved as simple as storing it in the first place.

YAML.load(File.open(File.dirname(__FILE__) + "/yaml/" + filename))

First pass I just wanted to merge based on player names – ignoring the imperfections that surely come with that… I was pleasantly surprised at how well things actually came out. I wrote a little throwaway script (as if this whole thing isn’t throwaway) to tell me how much of a problem I actually have. Basically: how many players in the top 400 of any site don’t have a match from the other sites? The answer was basically: a lot of Latin guys and a few others.

So, one problem I had was character sets used in the Latin player names. This is an area where “you’re going to throw this away” came into play: I just grabbed the few codes that I was having trouble with and regex’d to replace them with their friendlier counterparts. This is now on my list of “things to figure out how to do right”.

That out of the way, I re-ran and found that I really only had problems with about 20 guys. People with names like “Mike Smith”, that were “Michael Smith” in the other set. Here again: cheated. 20 guys? I can handle typing 20 names instead of figuring this one out, so I just modified the yaml manually here (this immediately breaks down if I have to re-run the scraper, but… I didn’t).

After determining which of the dumped data I wanted to keep (and which site could trump the others when they both had something like HRs), I created my csv with FasterCSV and I was rockin’.

In the spreadsheet

At this point, I was realizing that I should have created some new fields in that script… maybe look for things like big differences between sources (why does Y! think this guy is #20 and ESPN thinks he’s #100?) or even just: what’s the average of all the sites scraped. I was enamored with my spreadsheet though and haven’t had to been able to work with formulas in spreadsheets since my consulting-at-AmFam days, so I thought I’d give it a spin… and was quickly reminded of how easy they make it… to bash your head against the wall:

=IF(AND(P13="n/a";F13="");M13;IF(P13="n/a";(M13+F13)/2;IF(F13="";(M13+P13)/2;(M13+P13+F13)/3)))

But while that’s the real story, the reason I’m writing is I finally dedicated 15 seconds to actually clicking on the “lab” icon to check out what kind of features they had to offer.  And there it was.  Queue the bright light and the “heavenly sound”….

Send & Archive

Every once in a while you find a feature in a product that makes you say “hey, I bet these guys actually use this product themselves”.

It takes a couple hour to get a good set of 100-150.  I need to sort through my collection, head out to my brother’s Flickr account, try some friends accounts, and just generally “gather”.

After having DropBox recommended by a few friends, I think we may be on to something that not only makes the process simpler – the resulting collection is much higher quality as well.

The premise is pretty simple: you get a magic folder that looks to be almost the same as any other folder on your machine, but DropBox is keeping it in sync with it’s remote storage location: Amazon S3.  So, it’s obviously handy for just backing things up, but it doesn’t end there.  It makes it simple to share.  I share a directory with my parents, and my sister-in-law and immediately we all can put in our good/new pics and immediately have them broadcast to everyone else.

The strongest point of DropBox though is the Python-based client.  It’s incredibly intuitive – just drop things in and it’ll take care of business.  Looks just like a “real folder” to anyone who is used to any flavor of Windows.  No worries about telling it when to synch, easy visual indicators on whether it’s actively working on anything.  It’s just really real nice.

Dead simple and a great solution to a common problem.  It’s free for the first 2GB, so take the tour here.

It’s really intuitive, well designed, and implemented in a very clean way.  Great work guys!

newtunes.com has launched and is open for business.  The beta label is still on so there may be some bumps and bruises to be had but it should definitely be checked out.

And while you’re new-tunesing, add Music Streaker to your RSS reader.  The blog provides much more than NewTunes updates (although you’ll get that too) and is run by my friend Jon Friesch.