Enterprise Search

Open Source for Enterprise Search: Webinar

I just learned from David Fishman of Lucid Imagination that KMWorld Magazine and Lucid are presenting a webinar Tuesday, June 23rd. In addition to KMWorld publisher Andy Moore, the webinar will feature Sue Feldman of IDC and Ranga Muvavarirwa and Tom Morton of Comcast Interactive Media.

The KMWorld mailer says:

Business and government are using enterprise search to lower barriers to information for search, retrieval, analysis and exploration. Now Open Source is lowering the barriers to search, accelerating time-to-value from rapidly growing, diversifying organizational document and data resources.

Register for the free webinar at http://www.kmworld.com/webinars/lucid/23jun2009/kmb/. You may need to register with KMWorld or log in if you've already attended any of their webinars.

We've blogged about Lucid and open source before,and the need for more enterprise packaging before open source really becomes an enterprise solution. We'll be excited to hear what the folks from Comcast have to say.

s/Miles

Keyword Search: Much Ado About Nothing

Recently, Mark Logic's CEO Dave Kellogg wrote Things Not To Do: Declare Your Category Dead, sage advice with a personal story harking back to his days at Ingres. Dave's story is in response to a Larry Hawes post in the Gilbane Group Blog about the recent conference they held in San Francisco. According to Larry, Microsoft's Jeff Fried claimed 'keyword search is dead'.

My first thought was 'of course it is'. There have been reports of the death of keyword search - even of enterprise search itself - for years. Those of us who work with search have long realized that the idea of finding what you want from millions of documents by entering a word or two is totally unrealistic. Modern search technologies provide the tools to engage users in a conversation, to lead them to the best result iteratively using facets, natural language search, entity extraction, and sentiment analysis.

Next, since I actually attended the presentations Jeff gave, I thought back to see if I could remember him making the claim, but sure didn't remember it. I checked with Carl Grimm, who was there with me, and he didn't recall hearing it either. So I decided to check with Jeff, who even went back and checked his presentation to see if he made the claim anywhere. As we talked, we both agreed about the value of other technologies beyond simple keyword search; but he's pretty sure he never made the claim that keyword search is dead.

We can say with some confidence, the reports of the reports of the death of keyword search are greatly exaggerated.

New SearchDev groups for Autonomy and FAST

Years ago, Verity had a very popular and successful user group meeting every year, but Autonomy has not seemed to be interested in continuing the event.  SearchDev grew out of that last Verity user group meeting, and has a very active membership looking for, and providing advice to, those developing search applications within their respective companies.

As the group has grown, we're seeing an interest in more specialization by search technology. For that reason, we're happy to announce creation of two new SearchDev groups: autonomy.searchdev.org and fast.searchdev.org. These, along with the original www.searchdev.org, will continue to provide technical assistance for search developers, and should allow us to drill down a bit in terms of specialization. Expect other groups in the near future for commercial and open source engines.

Since these communities will have a more narrow focus, the feeling is that the group can grow to include webinars and perhaps even regional user group meetings in the future.

People who are interested in participating can join the Yahoo-group-based forum and start working to make these new user group a success.

s/Miles

Enterprise search doesn't mean mortgaging the farm

Lynda Moulton, the Search Practice analyst at CMS firm Gilbane Group, really hit the mark on a recent blog post nominally about how advertising money can but editorial space. While it's true that many  publications (and analyst firms) are happy getting paid by both sides. (Note: I was called out on this by this by Theresa Regli of CMS Watch, so I no longer say 'all analysts' for anything!)

In my opinion, the real news in Lynda's post is this: "there are dozens of enterprise search solutions that will serve you extremely well, with much lower cost of ownership" than with the big industry players. In fact, open source is beginning to penetrate the corporate veil, and while Lucene and Solr are not right for everyone, it looks like they've just about implemented what Mark and I consider Verity's  "Topic 1.0" capabilities circa 1990. We went to a meet-up the other night that Mark has written about; and we were pretty impressed.

So before you decide you need to budget a half million dollars or more for search, consider what Walter Underwood, chief architect of Ultraseek and now search evangelist at Netflix once told me. Paraphrasing: "You can download Solr then spend a ton of money customizing it; or you can spend a ton of money licensing enterprise search software, then spend a ton of money installing and customizing it. Your call."

But get help!

s/Miles

Enterprise Search Engine Optimization: eSEO

Last week at the Gilbane Conference in San Francisco, I participated in a panel "Search Survival Guide: Delivering Great Results" moderated by Hadley Reynolds of IDC. In the presentation, I offered a new view on improving enterprise search engine relevancy that I call eSEO.

The term SEO is well understood by - and widely practiced in - the corporate world.  The concept of SEO, as summarized by one of the Gilbane talks, states that "Key to the value of any Web content is the ability for people to find it”. In the SEO world this is done by combining organic results and keyword placement - advertising - to improve placement, maintain ranking, and monitor search engine position - results- over time.

While we've been helping our customers improve their enterprise search results, it's hard to convince them that search results are not a problem they can solve once. I've decided to apply a new term to this process - Enterprise Search Engine Optimization, or eSEO. To paraphrase the role of SEO, eSEO is the process of combining organic results and best bets to deliver correct, relevant, timely content to enterprise search users - employees, customers, partners, investors, and others.

For both organic and best bets, the first step is to identify what we call the "top 100" queries. Start by creating a histogram that shows the top terms from your search engine. I hope you'll agree that if the top queries - whether 100, 50, or even 20 - deliver great results, you're on your way to having happy users. Talk to your content owners as you review the histogram, and ask them to identify the best result for each.

Once you have a list of queries and results, start the two step process: tune the search engine using its native query tuning capabilities. This will impact the shape of the histogram, and over time should start delivering better results. The bad news is tuning like this doesn't position all of your top terms, and it would be silly to try to micro-manage the results for each. That's why search engines have best bets.

When you feel pretty good about the curve through query tuning, it' time to start setting up best bets - the "ad words" of eSEO. Limit the number of bests bets to one or two at most - but remember that you can use other real-estate like the rightmost column of the screen to suggest additional content. Some guidelines for best bets:

• Use one or at most two best bets
• Don't repeat a document already at the top of the organic results
• Make sure your best bets respect security

Once you have tuned your search engine, and set up best bets for the most timely and actionable result, you're ready to roll it out. But then the ongoing part comes in: you need to review your search activity and best bets periodically. Usually, we'd suggest once a month for a while, then perhaps quarterly thereafter. You may find seasonal variations, and if you're not watching you'll miss a golden opportunity.

In Summary

1. eSEO is just as critical as SEO

• Lost time and revenue
• Legal exposure

2. Watch for trends over time: Search is not "fire and forget"

3. Make sure SEO doesn't impact your eSEO

• Use fielded data that web search engines ignore for your tuning (i.e., 'Abstract' rather than 'Description'.

This will get you started; but because your queries and your content changes over time, it's a never-ending story. Some companies - ours included - have tools that can help. But no matter what, hang in there!

s/Miles