Dawn Patrol

Preamble

Tweet is a plugin for Yahoo! Search. Such plugins are called SearchMonkeys in honor of Greasemonkey for FireFox browser. Like Greasemonkey, SearchMonkey allows developers to enhance the experience, the search experience in this case. SearchMonkeys can enhance presentation with images and additional links or by combining Yahoo!'s Search index with other structured data.

Yahoo! Search users can add SearchMonkey applications to their profile on an opt-in basis. Add Tweet to yours if you like to get much improved search results for Twitter user profiles.

Current Twitter search results

While Twitter user profiles are being indexed by all major Search engines, their summary is extremely poor. Google and Yahoo's results are nearly identical. Yahoo!'s summary of my profile (below) doesn't even include my full name (Bart Teeuwisse), which is on the page.

SearchMonkey to the rescue¹

Luckily with SearchMonkey you can replace standard summaries with enhanced summaries. To improve Yahoo!'s Twitter user profile search results I wrote a SearchMonkey application called tweet that is triggered for all URLs matching *.twitter.com/*. Tweet calls Twitter's API to fetch user profile information not in the Yahoo! Search index. The result is a rich overview of a Twitter user, including last message (aka. tweet).

Tweet's challenges

Sounds simple doesn't it? Contact Twitter's API, get profile, present profile. The SearchMonkey's architecture splits this into 2 monkeys:

• A data monkey to contact Twitter's API to get the profile, and
• A presentation monkey (Tweet) to present said profile.

Well, not quite.

Speed, speed, speed

It turns out that execution speed of a SearchMonkey is key. To make the SearchMonkey Gallery a presentation monkey such as Tweet has to complete within a fraction of a second. Any call to fetch 3^rd party takes too long to satisfy this requirement. Certainly calling Twitter's API whose fluctuating response times are all over the map.

Secondly, Twitter's profile API call takes a user ID, which first has to be extracted from Yahoo!'s indexed data. An additional data SearchMonkey can do that and whose output is the input to Tweet's profile feching data monkey. However, this chaining of data monkeys makes Tweet only slower.

Thirdly -as I mentioned earlier- Twitter's API has wildly varying response times. And is by no means predictable enough to guarantee a prompt response. Furthermore Twitter is having scaling issues already. Adding a SearchMonkey that calls Twitter's API for up to 10 search results for each query could make things should Tweet gain many opt-in users.

Perhaps caching can help? The SearchMonkey platform does has some caching. Unfortunately SearchMonkey developers have no control over SearchMonkey's cache. Emperical data suggests that SearchMonkeys are cached for only a few minutes. Tweet could be cached much longer without sacrificing functionality.

SearchMonkey + AppEngine = Fast Data Monkey

To mitigate these challenges I decided to use a proxy of my own in between SearchMonkey & Twitter.

• A proxy that could scale, should Tweet become popular.
• A proxy where I could control my own cache.
• And a proxy that eliminates the need for an additional data Monkey to extract the Twitter user ID from the search result.
• And lastly a proxy that returns DataRSS to minimize (XSLT) transformations in SearchMonkey.

Why not Yahoo! Pipes?

I first turned to Yahoo! Pipes, but Pipes doesn't give me caching control and the only XML output format is RSS not DataRSS. So I turned to Google's App Engine instead, which satisfies all my requirements. It offers Memcache caching, is build to scale, allows me to extract the Twitter user ID, make the Twitter API call and transform its response to DataRSS.

Developing for Google App Engine

Even though this is my 1^st Python application worth mentioning, I didn't have too much trouble writing it. App Engine's documentation combined with Python's tutorials were sufficient to answer my questions. The biggest obstacle I encountered is the lack of good XML/XLT libraries for Python. There isn't a clear winner to begin with and App Engine's restriction to pure Python libraries eliminates all candidates, as I learned the hard way.

I really like the Googel App Engine SDK. No hassle configuring a web server or data base. No need to be online even. I developped about half the proxy while vanpooling to and from work!

How the proxy works

My proxy takes the URL of the search result as input from SearchMonkey. Given the trigger URL pattern these are all URLs to *.twitter.com. E.g. twitter.com, explore.twitter.com or m.twitter.com. The proxy first extracts the Twitter user ID, if any. In Twitter's URL schema, user IDs are the 1^st part of the URLs path. E.g. bartt in twitter.com/bartt or twitter.com/bartt/friends

It then checks the Memcache for a profile for this ID. If it has one it composes the DataRSS response and exits. If it doesn't it calls Twitter's API. Succesfull API calls are parsed and stored in Memcache for -currently- 2 hours, before composing a DataRSS response. Failed calls return an empty DatRSS response.

Fast enough?

My proxy speeds up cached profiles by a factor 3x to 10x. Most of the time, that is. Despite App Engine's claim to scale, it does have performance issues from time to time. App Engine had an outage for a day while I tested my proxy for example.

Odly enough, Twitter's API holds the record of the fastest response time, yet its average is many times App Engine's average response time (for cached profiles). App Engine's response time is very stable - about 200 milliseconds round trip from a west coast data center.

This doesn't make Tweet fast enough to be included into the SearchMonkey Gallery though. Not only is the proxy not fast enough, to that you'll have to add the XSLT process and 'render' times by SearchMonkey. Still, Tweet is now eminently more usable and shields Twitter from API overload.

Tweet & FriendNet combined

Combine Tweet with FriendNet, one of my other monkeys for an even richer search result. FrienNet displays profiles and contacts embedded in the page. It combines hCard profiles with XFN links embedded on the page to present a social graph.

In collapsed mode -the default- FriendNet shows the number of profiles, cards and contacts found on the page by Yahoo! Search.

Expanded, FriendNet shows details of Twitter friends.

Create your own SearchMonkeys

Got your own ideas for improving Yahoo! Search? Start monkeying around! You find everything you need at SearchMonkey on the Yahoo Developer Network.

Check out the SearchMonkey Gallery for more monkeys you can use. Or take my Better Amazon monkey for a spin.

1. You'll need to have tweet installed to get the same result.

FriendNet is a plugin for Yahoo! Search. Such plugins are called SearchMonkeys in honor of Greasemonkey for FireFox browser. Like Greasemonkey, SearchMonkey allows developers to enhance the experience, the search experience in this case. SearchMonkeys can enhance presentation with images and additional links or by combining Yahoo!'s Search index with other structured data.

Yahoo! Search users can add SearchMonkey applications to their profile on an opt-in basis. Add FriendNet to yours if you like to make your search more social aware.

FriendNet leverages microformats indexed by Yahoo! Search to make visible embedded user profiles and social connections. Numerous sites enrich the HTML markup of their pages with hCard and XFN microformats to make profile information and social connections machine readable.

Following are several examples of profiles and social connections presented by FriendNet.

Flickr

We all know and love photo sharing site Flickr. But did you know that people pages are marked up with microformats?

The example below shows Yahoo!'s standard summary for my Flickr profile followed by FriendNet's display of profiles and connections found on this page: 3 profiles, 10 contacts and 1 hCard. (For brevity not all contacts are shown.)

Above the grey line is FriendNet's synopsis. If you search for Flickr profiles ¹ you'll notice FriendNet's synopsis below results that contain hCard markup. Below the grey line are the expanded details for FriendNet which one gets by clicking the blue triangle in the synopsis.

The 1^st profile comes from the 1 embeded hCard. The 2 other profiles come from me XFN links. The hCard only included my full name and location. If Flickr adds my buddy icon it would replace the placeholder smiley.

The 10 contacts are all XFN links marked contact. FriendNet combines XFN links and hCards by the hCard's homepage URL for a richer presentation. As Flickr didn't have any hCards marked up for my contacts, FriendNet displays just the URLs. FriendNet's Twitter example further down the page shows how combining hCards and XFN links enriches the presentation.

FriendNet also turns Yahoo! Search into a social graph walker. Click on the magnifying glass next to a profile or connection to inspect the 'social graph' of that profile or connection.

A personal blog

FriendNet works with any kind of site that has embedded hCards. Take Dougal Campbell's personal blog for example. His homepage includes links to a number of contacts which he has tagged as 'colleague', 'acquaintance', 'met', etc. to indicate his relationship with them. See what FriendNet presents for Dougal's last connaction at Yahoo! Search.

Dougal doesn't have a XFN me link to dougal.gunters.org, which is why FriendNet placed Dougal's own hCard in the contact section instead of under his profiles.

The University of Notre Dame

The University of Notre Dame ¹ simply included 1 hCard with the universities contact information. Simple but effective.

Pownce

Pownce is another social messenging site. A bit like Twitter except that you can also send links, files and events. And a whole lot more confusing. As the example below shows, their use of hCards could be improved. The name and photo of my 1 contact on pownce weren't included in the hCard. If they add them FriendNet will dutifully show them. Add an XFN me link to pownce.com/bartt/ and my hCard will move from the contact list into the profile list where it belongs.

Twitter

Twitter is a service for friends, family and co-workers to send quick updates on what they are doing. Twitter has been growing like a weed.

Twitter's use of hCard and XFN is pretty good. Both my Twitter profile and my contacts have been marked up with hCard and XFN. The combination of hCard and XFN data yields an enriched presentation. Contacts are listed by their full name and buddy icon. Compare this to Flickr's example where contacts are listed as URLs. The one improvement Twitter could make is including my buddy icon and location in my hCard.

Note: I'm working on another SearchMonkey called Tweet for enhanced presentation of Twitter profiles. You see Tweet's synopsis in the above example. More about Tweet in a later post.

Yahoo! Finance

Doing a blanket search for hCards on Yahoo! Search ¹ I came upon some interesting use of hCards. Yahoo! Finance for example uses it to list their financial experts.

Create your own SearchMonkeys

Got your own ideas for improving Yahoo! Search? Start monkeying around! You find everything you need at SearchMonkey on the Yahoo Developer Network.

Check out the SearchMonkey Gallery for more monkeys you can use. Or take my Better Amazon monkey for a spin.

1. You'll need to have FriendNet installed.

The new Yahoo! Video -code named project Playground- launched yesterday in the US. Various International Tiger Team members worked on this project: Marco, Tak, Daniel and myself. I've architected the i18n and l10n of the site. Can't wait for the localized sites to launch in the near future.

Here are a couple of videos in the new embedded player, which is part of the new site. They showcase 16x9 wide screen support, the awesome and fun video team and some high quality user contributed content.

The Launch of the New Yahoo! Video

Indiana Jones and the Kingdom of the Crystal

PostSecret - A Valentine Video

Postcard to promote Paintings & Prints by Ginger Crawford Tolonen on exhibit at Artemis Healing Center in Los Gaots from Jan 25 - Apr 25 2008.

By the ever entertaining Joy of Tech.

Today I'm publishing 3 Ruby on Rails plugins:

• Crumblr, a Ruby on Rails plugin to protect against Cross-Site Request Forgery.
• ImageBundle, a Ruby on Rails plugin to automatically bundle local images into CSS sprites.
• StringExt, a Ruby on Rails plugin that adds class method rand to class String.

All plugins are listed on Agile Web Development's Ruby on Rails plugin directory: Crumblr, ImageBundle & StringExt.

If you haven't already, install rapt, a script/plugin replacement. Rapt taps directly into Agile Web Development's plugin directory. Now you can search the plugin directory instead of having to hunt for plugin sources.

Let me know what you think about these plugins.

Casper's hand outlined on paper. He must have been about 15 months at the time.

The fine people of the Walker Art Center recently asked me if they could use some of my photos on flickr in their catalog and on the center's web site. Specifically if they could use some of my Millenial Territory Orchestra photos for Steven Bernstein's Millennial Territory Orchestra performance on Dec 1, 2007 at the McGuire theater.

I'm honored that both the Millennial Territory Orchestra and Walker Art like my photos of MTO's performance at the Jazz Standard in NYC well enough to represent MTO. MTO's page in the catalog and the oversized postcard of the Walker Art '07-'08 program featuring the same photo are gorgeous. See the scans included in this blog post.

See MTO's performance at the McGuire theater for use of some additional images. Go see the Millennial Territory Orchestra should you be near Minneapolis on Dec 1st, 2007, they're fantastic.

To verify ownership of this blog on Technorati, Technorati's spiders look for a link to my Technorati Profile at the Code Mill.

So now there is one. Let the spiders feast!

Get the Flash Player to see this player.

I've started to experiment with SketchUp. SketchUp is an amazing 3D modelling program. Using simple geometric building blocks one can create sophisticated realistic 3D models. I'm going to use SketchUp to design our new workshop/barn.

A few years ago Tatyana took me to see this house in the shape of a barn that she had come across. I took some pictures of it because we both liked the shape. SketchUp allowed me to load the image and create a model of it. The result you see on the left. Play the video to see the model rotate. The photo of the barn has been draped over the model to texture it.

It turns out that the ratio of length and width of the model exactly match the ratio we could build a barn in on the site we have in mind.

Oversized postcard to promote Artemis Healing Center's Sampler Day, come Saterday Aug 18, 2007.

Oversized postcard sent out to Tatyanna Teeuwisse's clients to inform them of Tatyanna's new contact information.

Postcard to promote Pascale Wowak's Images For Life Exhibit at Artemis Healing Center from Aug 17 - Oct 25, 2007.

The exhibit opens tonight, can't wait to see how they'll transform the clinic compared to the previous exhibit.