“One day while raking leaves I thought about fields of data that are available online and how you can treat them like hammers. If all you've got is a hammer, then everything looks like a nail. I say that with a new kind of hammer, there can be new sets of nails to discover.
I then thought about my day job as a blogger and how I wasn't doing anything with blog comments! Blog comments were structured, publicly accessible, tied to people, and timely. So I thought of a way I could leverage blog comments to learn things early.
I took Robert Scoble's Most Influential in Tech list on Twitter and I scraped all of the home page URLs off of the bios. I grabbed those URLs and I took them to a service called BackType (since acquired by Twitter and shut down). BackType would take any URL and scour all of the comment fields in blogs around the web, returning new comments where that URL appeared in the URL field of the comment, and delivered by RSS. I took those RSS feeds and I plugged them into an RSS to IM real-time notification system.
This allowed me to break several news stories: an important engineer would post a comment on some obscure blog asking about a secret forthcoming project, and I would get a real-time notification of the comment. So then I'd go report on the otherwise secret forthcoming project. It was pretty awesome and I never told any of my research subjects how I found out about their news, except for once at 4 am in a pizza line at SXSW.”