For those of us that spend our days trying to tweak Mirlyn to make it better, one of the most important — and, in many ways, most opaque — questions is, “How good is our relevancy ranking?”
Research from the UMich Library’s Usability Group (pdf; 600k) points to the importance of relevancy ranking  for both known-item searches and discovery, but mapping search terms to the “best” results involves crawling deep inside the searcher’s head to know what she’s looking for.
So, what can we do?
Record interaction as a way of showing interest
One possibility is to look at those records that are somehow “touched” by a user in such a way that we can log it. If a user bothers to interact with an individual record, we’ll assume the record is interesting to her in the context of the current search.
There are three links associated with an individual record that a user can click on from the search results:
- (62% of all record interactions) The title
- (28%) An external link (HathiTrust, Google Books, or one of our vendors)
- (10%) The “see holdings” link for those items that have multiple holdings
Our first issue arises quickly: only about a quarter of Mirlyn sessions contain any of these actions. For a full 75% of sessions, we have no data about which records users are paying attention to. They get a call number — or determine they have a failed search — and move on.
Where on the page do users interact with items?
We don’t know how users that interact with items differ from those that don’t. But for those that do, more than half of all record interactions are with the first record.
Here are the numbers for the first five records:
- First record: 54%
- Second record: 12%
- Third record: 6%
- Fouth record: 3.7%
- Fifth record: 2.5%
More than 75% of all record interactions are with the first four items on the first page of results.
What does it all mean?
Frustratingly, we don’t know. Several possibilities are obvious:
- we’re doing a good job with relevancy ranking
- people do mostly known-item searches
- people don’t bother looking past the first few results
- excellent general search engines (e.g., Google) have trained people to believe that the first result is always worth a closer look.
The interactions between these (and unknown other) factors are likely complex.
In the meantime, though, to the extent these data can be extended to the general case (not at all obvious), we’re not doing too bad of a job.