Here’s another FeedFlare mashup that I put together for Feedburner-enabled blogs:
This mashup utilizes Yahoo’s Term Extraction API to extract ‘key phrases’ from blog post titles, then uses Google’s Cross-Language Search to locate related content in the desired foreign language (select from 15 available languages). Discovered ‘related content’ results are then displayed below each blog post as a clickable link, with any text displayed as automatically translated English. For example, if you are a Chinese student living in the USA, you can integrate automatic Chinese-language pre-searches into your English-language blog.
On a side note, congratulations to my friends over at Cerulean Studios, whose Trillian Instant Messenger product was a winner in the Communications category of the recent WebWare 100 Awards (ranked #2, beaten only by GMail!). Speaking of this, does anyone know why the WebWare folks took down their individual product rankings for each category? These category rankings were up a few days ago, but now they’re gone and all winners have been rearranged in alphabetic order. Considering that this is a listing of Awards winners, I don’t really see the utility of an alphabetic ordering.
One of the central themes surrounding the Implicit Web is the power of the electronic footprint.
Wherever we go, we leave footprints. In the real world, these are quickly washed away by erosion and other natural forces. The electronic world, however, is far, far different: footprints often never disappear. Every move we make online, every bit of information we post, every web link we click, can be recorded.
The Implicit Web is about leveraging this automatically-recorded data to achieve new and useful goals.
One area that’s particularly exciting to me is the utility provided by merging implicit data collection/analysis and automatic information retrieval.
Neat stuff! I love the idea of “re-searching” automatically, leveraging an Internet user’s original search query.
A few days ago I decided to mess around with this “re-search” idea and ended up with something that I’ve been calling “pre-search.”
Pre-search is the concept of preemptive search, or retrieving information before a user asks for it (or even knows to ask). This idea can be of particular use with blogs and other topical information sources.
I created two basic pre-search mashups for Feedburner-enabled blogs, using the Feedburner FeedFlare API:
Both of these are pretty straightforward, doing the following:
1. For every blog post, use the Yahoo Terms Extraction API to gather ‘key terms’ from the post title.
2. Use Google Blog Search or Lijit Network Search to find related content for the previously extracted ‘key terms.’
3. Formulate the top three results into a clickable link and show them below the blog post.
This results in the automatic display of related content for a given blog post, using a combination of content analysis (on the blog post title) and information retrieval (Lijit/Google Blog Search).
I’ve enabled both Google and Lijit Pre-Search on my blog. You can add them to your Feedburner-enabled blog by visiting here and here.
Implicit, automatic, passive, or whatever you call it, this form of content analysis is starting to be recognized as a powerful tool in both a business and consumer/prosumer context. Companies like Adaptive Blue are using automatic content analysis techniques to personalize and improve the consumer shopping experience, while others like TapeFailure are leveraging this technology to enable more powerful web analytics.
Content analysis takes clickstream processing one step further, providing a much deeper level of insight into user activity on the Web. By peering into web page content (in the case of Adaptive Blue) or user behavioral data (as with TapeFailure), all sorts of new & interesting capabilities can be provided to both end-users and businesses. One capability that I’ll focus on in this post is automatic republishing.
Automatic republishing is the process of taking some bit of consumed/created information (a web page, mouse click, etc.) and leveraging it in a different context.
Let me give an example:
I read Slashdot headlines. Yes, I know. Slashdot is old-hat, Digg is better. Yadda-yadda. That’s beside the point of this example.
Note that I said “I read Slashdot headlines.” This doesn’t include user comments. There’s simply too much junk. Even high-ranked posts are often not worthy of reading or truly relevant. But alas, there is some good stuff in there — if you have the time to search it out. I don’t.
So this is a great example of a situation where passive/implicit content analysis can be extremely useful. Over the course of building and testing my company’s AlchemyPoint mashup platform, I decided to play with this particular example to see what could be done.
What I was particularly interested in addressing related to the “Slashdot comments problem” was the ability to extract useful related web links from the available heap of user comments. Better yet, I wanted to be able to automatically bookmark these links for later review (or consumption in an RSS reader), generating appropriate category tags without any user help.
What I ended up with was a passive content analysis mashup that didn’t modify my web browsing experience in any way, but rather just operated in the background, detecting interactions with the Slashdot web site.
When it sees me reading a Slashdot article, it scans through the story’s user comments looking for those that meet my particular search criteria. In this case, it is configured to detect any user comment that has been rated 4+ and labeled “Informative” or “Interesting.”
Upon finding user comments that match the search criteria, the mashup then searches the comment text for any URL links to other web sites. It then passively loads these linked pages in the background, extracting both the web page title and any category tags that were found. If the original Slashdot article was given category tags, these also are collected.
The mashup then uses the del.icio.us API to post these discovered links to the web, “republishing them” for future consumption.
Using an RSS reader (Bloglines), I subscribe to the del.icio.us feed generated by this mashup. This results in a filtered view of interesting/related web links appearing in the newsreader shortly after I click on a Slashdot story via my web browser.
This is a fairly basic example of content analysis in a user context, but does prove to be interesting because the entire process (filtering user comments, harvesting links from comment text, crawling any discovered links, extracting title/tag information from linked pages, and posting to del.icio.us) happens automatically, with no user intervention.
I think we will see this this type of automatic prefiltering/republishing become increasingly prevalent as developers and Internet companies continue to embrace “implicit web” data-gathering techniques.
There’s been a lot of buzz recently surrounding the “Implicit Web” concept, something I’ve blogged about in the past.
ReadWriteWeb has a great write-up on the subject, with case studies focusing on several popular websites that incorporate implicit data collection techniques (Last.fm, etc.).
Even more interesting is a “live attention-stream” viewer created by Stan James of Lijit. This neat little webapp utilizes clickstream data gathered by the Cluztr social browsing plugin, allowing Internet users to “follow along” with another user’s web browsing session.
This and other recent work on leveraging implicit data-flows is pretty exciting stuff, and we’re really only starting to scratch the surface as to what’s possible.
I’ve been toying around with implicit data gathering techniques for the last six months or so, using my company’s AlchemyPoint platform to gain access to clickstreams and other information. Because the AlchemyPoint system operates as a transparent proxy-server, it makes it easy to build simple analysis/data-mining applications that “jack in” to web browsing, email, and instant messaging activity.
So what’s possible if you’re “jacked in”? Let’s start with something very basic: gathering statistics on the usage of various web sites.
Above is a snippit from something I’ve been calling a dash-up. So what’s a dash-up?
Think dashboard + mash-up.
Essentially, a dash-up is a presentation-level mashup that collects data from multiple sources and presents it in a useful graphical dashboard view (in this case, dynamically updating activity charts). The above screenshot is showing both a general web traffic history, and more detailed statistics on my music-listening activity on the popular Internet Radio web site, Pandora.com.
OK, statistics-gathering is kinda neat, but what about something more useful? One of my favorites is passive tagging or implicit blogging activity. Stay tuned for my next post, which will detail some of the ways passive/implicit data collection enhances (through filtering, tagging, etc.) the Internet sites I use on a daily basis.
Congrats to the Me.dium team who just closed a sizeable series B venture round, led by Commonwealth Capital Ventures!
I’ve been following Me.dium since seeing their first public demo at the Boulder NewTech Meetup. It will be exciting to see the strides in scaling and usability the company makes with this new capital infusion.
It’s great seeing articles like this:
Violent crime increased slightly across the United States in 2006, but Denver saw a slight decrease, the FBI reported today. Denver bucked the national trend on violent crimes with a 2.7 percent drop in 2006. Property crimes in Denver also dropped by 22 percent.
I’ve been a Denver resident for a few years now, but it felt like home on my first visit. Denver is a great city — for starting a company, a family, anything exciting. If you haven’t already, come check it out.