The Definitive Search Engine Result Comparison – An Investigative Report

I’ve received enough praise and criticism of my perspective on Ask’s new campaign that I felt compelled to do investigate the quality of each engine in more detail. As you can imagine, these results are completely biased, being scored alone by my opinion; nonetheless, I felt the experience worth sharing.

Within five groups of terms, selecting from a mix of themes to ensure as broad a comparison as possible (brand names, store names, product categories, services, and things to do), I have, by way of a highly scientific, proprietary research methodology, selected 2 keywords.   One that I consider popular online and the other less so to give a balance to the theme under investigation.

The top 3 results from each engine and a general review of the first page (SERP) are given a score on a scale of 1-10.  This score, of course, is also the result of highly confidential analytics.

The evaluation includes Google, Yahoo!, Ask, MSN (Live Search), and my usual favorite, del.icio.us, but only considers natural listings and engine features and benefits.  Paid results are not influencing the outcome.

Click here to skip the details and go to my results

1. Brand Searches (Honda and Bubble Yum)
   • Honda
     • Google: (6) Prominently promotes Honda.com ignoring dealers, comparison options, reviews, or company information (i.e. history). In fact, almost the first page is really Honda.com.  Sublinks promote Autos, Motorcycles, ATVs, and Honda’s Jet (I’m sure that’s popular).  Second and third results repeat the sublinks with Honda’s Auto and Motorcycle sites prominently promoted.
     • Yahoo: (4) While I’m seeing more variety further down the page, the first three results are the same. Honda’s homepage (without the sublinks) while second, third (and forth) promote Autos.  Civic consumers are covered!
     • Ask: (5) A little selection with Honda.com, Autos, and Motorcycles.  No sublinks so not as much variety as Google but at least they don’t repeat the Autos content.  Forth is Toyota…. then Honda UK, Germany and their global website.  Then the well known Phoenix GWRRA.  Bonus points though for Ask’s binoculars which I haven’t given much consideration yet, the unique Ask feature is a nice way to quickly breeze through the SERP.
     • Live: (6) A leg up for variety throughout the page.  The first three are the same as the rest but following that are the variety of Honda product one gets from Google’s sublinks.  Unfortunate that they too seem to only promote Honda.com.
     • del.icio.us: (3) So much for my favorite.  I get the link to my favorite Honda commercial, Honda UK, another great commercial, and a cool but annoying site with popups and cleavage ads.
   • Bubble Yum
     • Google: (6) A good start stunted by over promotion of buying options (who buys gum online?).  The ever present Wikipedia link and an urban legends site make the top cut.  We get some history and
       lots of commerce.
     • Yahoo: (7) heh. Almost a mirror image of Google (perhaps I should ding them both for copying).  Wait… more variety down the page to keep me interested
     • Ask: (4) The history of bubble gum! Followed by a site that shouldn’t even break the top page and a bulk candy store.  Urban legends again (go snopes!) and our friends at Wiki.  Then the corporate links.
     • Live: (7) Some nice variety with the corporate page, online games featuring the gum, urban legends, and history.  Go Live.
     • del.icio.us: (4) Come on guys!  Bubble productivity software, online games, Bubble Tea, Chocolate Bubble Wrap
       (cool!). A link to the del.icio.us and slashdot homepages?!

Round 1 goes to Live with the greatest variety throughout the page.  Ask won some points for the binoculars while Google fared well for relevant links though suffered without variety.  Del.icio.us will make a come back with products, I’m sure!

2. Stores (eBay and Ikea)
   • eBay
     • Google: (8) Could only get better with some eBay schemes and 3rd party tools.  Links to the most popular categories and Wiki as well as their developer platform and philanthropic work.
     • Yahoo: (5) They have the variety concept figured out but need some work with links to Italy, Australia, UK, France, Singapore…  A few interesting resources mixed in.
     • Ask: (6) The security center, community site, official eBay time (that’s helpful), Italian and UK sites.  The refinement features on the right get them a bonus point but the first SERP needs work.
     • Live: (7) Promotion of the site, motors, corporate information, affiliate programs, and finances are nice but we’re missing the variety of products.
     • del.icio.us: (6) Ebay.com, UK, Germany, and Australia (how can Search be considered relevant when they serve results from different countries?).  Some links about eBay, making money, and shopping services help.
   • Ikea
     • Google: (7) Not so nice this time Google but at least you kept the variety going.  Links to Ikea include countries and corporate services.  A game, blogs, and Wiki add to the mix.
     • Yahoo: (6) Yahoo must be pushing their international footprint, Malaysia, Perth (Australia), Spain, and Hong Kong make the cut.  News from 2004 and YouTube links contribute to the mix.
     • Ask: (8) Popular links are great with a store locator, new products, and decorators.  The rest of the content is attractive with the Wiki, entertaining blogs, and Fan sites that certainly appeal to Ikea shoppers
     • Live: (6) The results seem stunted with what feels like fewer on this first page (not the case just my impression).  The homepage, corporate site, and franchising information are nice variety specific to the company but that’s it.
     • del.icio.us: (6) Ikea hacker! (cool), ikea.com, 2 Swedish sites and one from Asia.  Links to an office and kitchen planner are nice.

My international shopping needs are well taken care of with Yahoo!  Google takes this one with variety and relevance (go Base!) though Live and Ask are right behind them.  Come on D!

3. Products (digital cameras and tires)
   • digital cameras
     • Google: (5) A review site, a review site…. a review site… a shopping comparison site… hey! How Stuff Works, they’re always fun!
     • Yahoo: (9) This is how its done.  Links to brands, well known stores, comparison sites, review sites, and insights about digital cameras.
     • Ask: (7) Brownie points for ancillary benefits such as product reviews, news, and refinement links but the results themselves need help as they include information sites, a brand, second tier review sites, and small businesses.
     • Live: (6) Decent promotion of stores but we’re missing the brands.   I’m just not feeling it.  Sorry Microsoft.
     • del.icio.us: (7) Renewed my faith a little with the usual mix of strange links, this time adding value.  Some store and review sites as we need but also some innovative products, how to guides, and a professional blog
   • tires
     • Google: (7) First thought, who shops Tires online, right?  Discount Tire, Goodyear, TireRack, BFGoodrich, Dunlop, Yokohama, even the Wiki and How Stuff Works.  Probably the best results can get.
     • Yahoo: (9) Nope, Google wasn’t the best.  A couple tire stores, comparison sites (I went back up and dinged Google for not having these), different types of tires (I hadn’t thought of that, another ding to Google), and a nice Local feature at the top promoting stores in my vicinity.
     • Ask: (8) The results themselves remind me of Google but the refinement links on the right give them an edge
     • Live: (6) Local results are a nice touch but the results are deficient in the variety of tire brands and well known stores. Nothing informative except Wiki.
     • del.icio.us: (6) Its a score.  We’re on the board with some unique links including Tire Sandals, a YouTube video, a Tire size calculator, and links for Bike and ATV tires.  Of real value?  Only Tire Rack and Michelin.

Yahoo! does product search right (good thing too since I used to work in that group) but Ask is a great alternative.  Honorable mention to del.icio.us this time for having some interesting alternatives.

4. Services (web design and house cleaning)
   • web design
     • Google: (6) A nice list that includes descriptive information, usability insights, templates, business directories, a few business listings, and coding resources.  Missing how to guides.
     • Yahoo: (7) Strikingly similar to Google but a longer list including a few more businesses.
     • Ask: (8) These are the best results for a DIY (do it yourself).  Tools, guides, resources, directories, and examples of good and bad design.
     • Live: (5) The top 2 links are helpful but value falls off after that.  Includes a link to Google UK (??)
     • del.icio.us: (7) Now we’re at talking.  Design examples, advanced and basic code support, style guides, courses, tools and resources.  Clearly caters to the DIY designer though as we’re missing businesses and directories.  
   • house cleaning
     • Google: (6) A nice list of well known house cleaning businesses and advice websites but house cleaning, unless I’m doing it myself, is an "offline," local business need and these results don’t support that. 
     • Yahoo: (7) Same quality as Google but more of them so a bonus point for volume.
     • Ask: (5) Short list of similar results, some actually for specific states so those are worthless.
     • Live: (4) The most unique results so far but a greater proportion of localized results with Atlanta, Massachusetts, Boulder taking up space.
     • del.icio.us: (6) The best results but specific to those that want to clean themselves.  No support for businesses or local services.

Ask and Yahoo! strike me as the best resources for services.  Perhaps a result of their capability with product search. del.icio.us made a strong showing here though for those that want to learn or do these things themselves; hands down winner for DIY-ers.

5. Things to do (CES and Cats)
   • CES: Consumer Electronics Show
     • Google: (7) Good coverage of the show with sublinks to support different aspects of the conference.  A 404 error costs a point but they make up with it with CES news, information, and what appear to me to be other relevant "CES" results.
     • Yahoo: (8) Better coverage of the show and more prominent promotion of other "CES" choices.  Conference related links are a better selection of third party providers.
     • Ask: (4) The conference is here and otherwise I’m introduced to the North Carolina Coop and repeated promotion of Bill Gates.
     • Live: (5) Again the conference appears prominently followed by Microsoft self promotion of MSNBC and Microsoft.com.
     • del.icio.us: (5)  Specific to the Consumer Electronics Show with a number of links to 2006 content.
   • Cats: Musical
     • Google: (4) Felines.  The musical appears through a Wikipedia listing and Catsmusical.com but I’m not interested in house pets.  Ever heard of Citizens for an Alternative Tax System? Apparently, it is more popular than the Broadway musical.
     • Yahoo: (5) Tremendous variety of feline results (though not what I want, good score on that basis).  One for the musical coming in at 17th on the page.
     • Ask: (6) Wow. Didn’t expect that.  Some cat lovers at Ask.  Prominent promotion of cats with a photo, description, genome, taxonomy, etc.  All links are for feline enthusiasts save the last for the same Wiki link we’ve seen elsewhere.  
     • Live: (4) Our musical Wiki link makes it to the top otherwise we have images, videos, and a poor selection of cat sites.
     • del.icio.us: (10) Warning, Not where cat lovers should turn but I’ve had enough of the animals from the other engines that the highly popular cat humor found prominently here is a welcome change.

del.icio.us cat humor aside, it looks like Yahoo! wins; however, it is clear the general engines really only support significant, internationally marketed events while local shows and events are barely supported.  We’ll score del.icio.us a 4 on that last round as there really isn’t anything of value.

Natural Search Engine Result Quality

Sorted by theme:

• Brands – MSN Live, Google, Yahoo, Ask, del.icio.us
• Stores – Google, Ask, Live, del.icio.us, Yahoo
• Products – Yahoo, Ask, del.icio.us, Tie: Google & Live
• Services – Yahoo, Tie: Ask & del.icio.us, Google, Live
• Things to do – Yahoo, Google, Tie: Ask & del.icio.us, Live

Rank ordered: Yahoo, Google, Ask, Live, del.icio.us

Other insights?

1. Wikipedia shows up time and again (pay attention SEOs)
2. Local needs will drive demand for vertical search as the big engines simply don’t support local business needs.  I’ll continue to turn to Yahoo! Local as a directory and for things to do,  local
   search with a company like Zvents.
3. I imagine, to much surprise, that most flip-flopping between engines, when searchers can’t find what they want, likely occurs between Yahoo! and Google.  Save yourself time and headaches of digging through very similar results and
   alternate between Yahoo! or Google and one of the others.  Ask
   makes a strong choice.
4. Engines are not the same. Which one is right for your particular search? Which is right for you as a marketer?
5. All the fun links to share are at del.icio.us