[Summary: ALWAYS ALWAYS ALWAYS USE PARENTHESES TO GROUP BOOLEANS IN SOLR!!!]

What does Solr do, given the following query?

  a OR b AND c

I’ll give you three guesses, but you’ll get the first two wrong and won’t have any idea how to generate a third, so don’t spend too much time on it.

Boolean algebra and operator precedence

Anyone who’s had even a passing introduction to boolean alegebra knows that it specifies a strict order to how the operators are bound: NOT before AND before OR. So, one might expect the following grouping:

  a OR (b AND c)

That’s guess one. It’s not how Solr does it.

Left to right?

Some naive students, and at least one programming language (Smalltalk), do a simple left-to-right evaluation. So you might go with:

  (a OR b) AND c

Nope. Wrong again.

So what’s left???

Excellent question. I don’t know the code well enough to know what’s going on underneath, but here’s what we get under the lucene query parser.

    (b AND c)

That’s right. The first term is thrown away.(More correctly, the first term is deemed “optional”).

Do you let your users put AND/OR/NOT in their queries?

Hopefully, they don’t know any boolean algebra. If they do, hopefully they use parentheses, or you parse it out for them. And if not, well, they’re gonna be pretty damn confused.

It gets weirder

I populated a fresh solr (3.5) index with all possible subsets of the strings “curly”, “larry”, “moe”, and “shemp” (not Joe. Don’t talk to me about Joe). There are 15 of them, from the one-item ‘curly’ to all four at once.

I wrote a script to run a set of queries against the index under both lucene and edismax to see what I would get. In all cases the default lucene operator is ‘AND’ and the edismax mm parameter is set to 100% (equivalent to “all required”).

        Lucene                    EDismax

  --------------------------------------------------------



curly AND larry
    curly larry               curly larry

    curly larry moe           curly larry moe

    curly larry shemp         curly larry shemp

    curly larry moe shemp     curly larry moe shemp
curly AND larry OR moe
    curly                     curly larry

    curly larry               curly larry moe

    curly moe                 curly larry shemp

    curly shemp               curly larry moe shemp

    curly larry moe

    curly larry shemp

    curly moe shemp

    curly larry moe shemp
curly OR larry AND moe
    larry moe                 larry moe

    curly larry moe           curly larry moe

    larry moe shemp           larry moe shemp

    curly larry moe shemp     curly larry moe shemp
curly AND larry OR moe AND shemp
    curly moe shemp           curly larry moe shemp

    curly larry moe shemp
moe AND shemp OR curly AND larry
    curly larry moe           curly larry moe shemp

    curly larry moe shemp

Query 1 is as expected. Query 2 apparently reduces to just ‘curly’ under the lucene parser and ‘curly AND larry’ under edismax (and query 3 similarly reduces to the two AND’d words). Queries 4 and 5 are…well, you can look at the debugQuery output to see what it gets, but not why. And then tell me how to explain it to a user.

Where does this leave us?

The good news is that both lucene and edismax behave predictably when you use parentheses for grouping. So do that.

I’m generally not one to complain about open-source software, at least partially because I don’t have the chops to do anything about it most of the time, but I don’t understand how this could seem OK to anyone. There are a couple lucene Jira tickets (Lucene-167 and Lucene-1823) and a 2005 mailing list thread denouncing the current behavior, but it persists.

Until the Solr/Lucene powers that be decide to tackle this, the rest of us will either have to write pre-parsers to make sure users get something sensible, or cripple our applications to disallow unrestricted boolean queries.

We had another baby. :-)

Shai Brown Dueber was born last Monday, the 3rd, at a very moderate 7lbs 7.2oz (his brothers were 9lbs and 9.5lbs). Mother, baby, and older brothers are all doing well. Father is freakin’ tired.

[Yes, another post about ruby code; I'll get back to library stuff soon.]

Quite a while ago, I released a little gem called threach (for “threaded #each”). It allows you to easily process a block with multiple threads.

Nice, right?

The problem is that I could never figure out a way to deal with a break or an Exception raised inside the block. The core problem is that once a thread trying to push/pop from a ruby SizedQueue is blocking, there’s no way (I could find) to tell it to wake up and see if there’s an error from another thread floating around that needs to be addressed.

So, I got into a pattern of running my code with each for a while, debugging, and eventually doing the production run under threach. Which is just dumb. Then I’d try to re-write threach to deal with this stuff using different approach (mutexes, lightweight events), quickly (or not so quickly) fail, give up, and start again.

So…let’s not worry MRI for the moment. I run all my big jobs under JRuby these days anyway, and there I can take advantage of Java’s blocking queues that have timeouts. When a queue operation times out, I can check to see if there’s been a break or an exception thrown in the meantime and behave appropriately.

The result is the gem jruby_threach. It works just like threach, except that, you know, it actually works the way I’d like it to.

Looks familiar, doesn’t it.

But you can also break out of the loop.

Any exceptions that are rescued within the block are handled internally and don’t cause processing to stop. Any that are not handled within the block are noticed by threach, cause the processing to stop, and the re-raised so you can deal with them outside of threach

Dealing with the underlying Java data structures makes life a lot easier. To the point that I added an enhancement — threading production as well.

mthreach basically allows you to treat an array of Enumerables as a single logical entity, multithreading both the producer and consumer sides of the operation. There aren’t a whole lot of obvious use cases, but it can certainly come in handy.

You can also access the underlying class that aggregates multiple enumerables directly.

I spent way too long asking my friend, The Internet, how to get a normal DBI connection to SQLIte3 using JRuby. Apparently, everyone except me is using ActiveRecord and/or Rails and doesn’t want to just connect to the database.

But I do. Here’s how.

First, get the gems:

Then you’re ready to load it up into DBI.

For those of us that spend our days trying to tweak Mirlyn to make it better, one of the most important — and, in many ways, most opaque — questions is, “How good is our relevancy ranking?”

Research from the UMich Library’s Usability Group (pdf; 600k) points to the importance of relevancy ranking  for both known-item searches and discovery, but mapping search terms to the “best” results involves crawling deep inside the searcher’s head to know what she’s looking for.

So, what can we do?

Record interaction as a way of showing interest

One possibility is to look at those records that are somehow “touched” by a user in such a way that we can log it. If a user bothers to interact with an individual record, we’ll assume the record is interesting to her in the context of the current search.

There are three links associated with an individual record that a user can click on from the search results:

• (62% of all record interactions) The title
• (28%) An external link (HathiTrust, Google Books, or one of our vendors)
• (10%) The “see holdings” link for those items that have multiple holdings

Our first issue arises quickly: only about a quarter of Mirlyn sessions contain any of these actions. For a full 75% of sessions, we have no data about which records users are paying attention to. They get a call number — or determine they have a failed search — and move on.

Where on the page do users interact with items?

We don’t know how users that interact with items differ from those that don’t. But for those that do, more than half of all record interactions are with the first record.

Here are the numbers for the first five records:

• First record: 54%
• Second record: 12%
• Third record: 6%
• Fouth record: 3.7%
• Fifth record: 2.5%

More than 75% of all record interactions are with the first four items on the first page of results.

What does it all mean?

Frustratingly, we don’t know. Several possibilities are obvious:

• we’re doing a good job with relevancy ranking
• people do mostly known-item searches
• people don’t bother looking past the first few results
• excellent general search engines (e.g., Google) have trained people to believe that the first result is always worth a closer look.

The interactions between these (and unknown other) factors are likely complex.

In the meantime, though, to the extent these data can be extended to the general case (not at all obvious), we’re not doing too bad of a job.

I just released another (this time pretty good) version of my gem for normalizing/validating library standard numbers, library_stdnums (github source / docs).

The short version of the functions available:

• ISBN: get checkdigit, validate, convert isbn10 to/from isbn13, normalize (to 13-digit)
• ISSN: get checkdigit, validate, normalize
• LCCN: validate, normalize

Validation of LCCNs doesn’t involve a checkdigit; I basically just normalize whatever is sent in and then see if the result is syntactically valid.

My plan in my Copious Free Time is to do a Java version of these as well and then stick them into a new-style Solr v.3 filter so I (and, by extension, you, if you’re interested) can have Solr do normalization during both index and search time.

A couple days ago I decided to finally get back to working on threach to try to deal with problems it had — essentially, it didn’t deal well with non-local exits due to calls to break or even something simple like a NoMethodError.

[BTW, I think I managed it. As near as I can tell, threach version 0.4 won't deadlock anymore]

Along the way, while trying to figure out how threads affect the behavior of different non-local exits, I noticed that in some cases there was still work being done by one or more threads long after there was an exception raised.

I re-discovered something that a lot of people already know: raise/rescue under MRI is slow, and under JRuby can be unbearably slow. How slow?

Let’s look at four simple blocks that exercise four different block exit strategies: break, catch and throw, raise with the normal single (or zero) arguments, as well as the three-argument version of raise.

range.each do |i|      
  break          
end              
      
    

catch(:benchmarking) do  
 range.each do |i|      
   throw(:benchmarking) 
 end                    
end
      
    

 begin                  
   range.each do |i|    
     raise StandardError
   end                  
 rescue                 
  # do nothing                
 end                          
     
    

begin                  
  range.each do |i|
    raise StandardError, :hi, nil
  end
rescue 
 # do nothing
end
      
    

In each case, we immediately exit the block without doing any work; the idea is to measure how long it takes to break out for each case.

So....let's run them each 100K times and see what happens, shall we? Times are in seconds, averaged over two runs.

The first thing to note is that this is 100K iterations. Three of the strategies are fast enough that you'd have to work really, really hard to notice them. In terms of speed, raise (3 args), catch/throw, and break are fast enough that you shouldn't bother worrying about them (although you should choose the method that makes your code easy to understand).

The second things to note is Holy Camoli! JRuby is slow there!

This Jira ticket tells the tale: The creation of the backtrace is very, very expensive for JRuby. That nil at the end of the raise (3 args) call suppresses the creation of that backtrace, so the speed is fine.

Three things worth saying here:

• If you're using raise/rescue for flow control, you're already doing it wrong. Reserve exceptions for, well, exceptional conditions that are only going to be raised once or twice, not all the time.
• If you're writing code that, for some ungodly reason, is planning on raising a crapload of exceptions, use the three-arg version. I'm looking at you, gem authors.
• If you're writing your code without worrying about how it will work under multiple threads, well, please don't do that. Everyone has multi-core systems these days, and it's silly to not be able to use them. Plus, counting on Matz to never move to a VM with real threads is a big gamble.

Yesterday, I gave a brief overview of why free text is hard to deal with.

Today, I’m turning my attention to a concrete example that drives me absolutely batshit crazy: taking a perfectly good unique-id field (in this case, the ISBN in the 020) and appending stuff onto the end of it.

The point is not to mock anything. Mocking will, however, be included for free.

What’s supposed to be in the 020?

Well, for starters, an ISBN (10 or 13 digit, we’re not picky).

Let’s not worry, for the moment, about the actual ISBN and whether it’s valid or not.

Wait, no, let’s go ahead and worry about it. It’s an easy enough script to write, although it takes a while to run.

8,630,794  Total records
3,220,666  Total 020a's
    6,498  020a's that don't obviously contain an ISBN
    8,407  that look like an ISBN but fail checksum test:
... so 0.26% of the ISBNs have invalid checksums

So, not bad at all, especially considering some of those are known to be bad, but are transcribed dutifully from the actual (mis-)printed book.

A lot of the malformed data (anything from which I can’t seem to extract something that looks like an ISBN) is pricing data, and most of it appears in system numbers that are close enough to each other that I presume it was just a bad batch.

What’s goes after the ISBN in the 020?

I’m no cataloger, of course, but it looks to me like the answer is “Something about how the book is bound together, or the publisher, unless you want to put something else there, and then, really, go ahead, because it’s not like anyone is ever going to want to parse this out, all we need to do is print cards with it for god’s sake.”

No, I kid, I kid! The actual rules are in Library of Congress Rule Interpretation 1.8, which reads, in part:

For a hardbound resource, there is no attempt to use a consistent term other than to use one that conveys the condition intelligibly.

I think it’s important to read that a second time, because it succinctly conveys the culture in which these rules were devised.

• Don’t worry about consistency, because your only reader is human.
• Defer to the cataloger.
• Being complete is more important than being consistent.
• Base your notes on your subjective view of the actual, physical item you’re presumed to be holding in your hands.

Interestingly (to me, anyway), it looks like the OCLC once had a (now deprecated) $$b subfield for binding information. Apparently it didn’t catch on.

What did I find?

So, let’s pretend I’d like to be able to differentiate between paperback and hardbound books. Probably useful, yes?

I went ahead and took all parenthetical notes from any field in the 020, split them on colon (’cause that seems to be the way they roll) and did some basic normalization:

• Eliminate numbers (so ‘vol. 1′ and ‘vol. 2′ count as only one pattern)
• Lowercase everything
• Turn runs of spaces into a single space
• Trim leading/trailing spaces
• Remove any trailing punctuation

I found 1,506,729 parenthetical remarks in the 020 subfields of our catalog.

The top twenty most common entries using those normalizations are:

1. 402537 pbk
2. 387406 alk. paper
3. 99260 v # (e.g., “v. 1″, “v. 22″, etc.)
4. 82918 cloth
5. 51125 hbk
6. 42036 electronic bk
7. 41360 acid-free paper
8. 38792 hardcover
9. 28913 set
10. 20358 hardback
11. 19160 ebook
12. 16264 paper
13. 15269 u.s
14. 12770 hd.bd
15. 11793 print
16. 10625 lib. bdg
17. 10520 hc
18. 8772 est
19. 7767 pb
20. 7639 hard

The kicker? These are the top twenty of 13,374 unique parenthetical strings found in the 020 field. Many of them are publishers, or cities, or whatnot, but an awful lot of them are variations on “hardcover” and “paperback.”

For example, a quick search for anything that might be “hard” (regexp: /h[ar]{0,2}d/) got me started on a list. Here’s just the 90 examples from that list that start with ‘h’:

hard | hard adhesive | hard back | hard bd | hard book | hard bound | hard bound book | hard boundhard case | hard casehard copy | hard copy | hard copy set | hard cov | hard cover | hard covers | hard sewn | hard signed | hard-backhard-backcased | hard-bound | hard-cover | hard-cover acid-free | hardb | hard\cover | hardbach | hardback | hardback book | hardback cover | hardbackcased | hardbd | hardbk | hardbond | hardbook | hardboubd | hardbound | hardboundhardboundtion | hardc | hardcase | hardcopy | hardcopy publication | hardcov | hardcov er | hardcovcer | hardcove | hardcover | hardcover-alk. paper | hardcovercloth | hardcoverflexibound | hardcoverhardcoverwith cd | hardcoverr | hardcovers | hardcoversame | hardcoversame as above | hardcoverset | hardcovertion | hardcver | hardcvoer | hardcvr | harddback | harde | hardocover | hardover | hardpack | hardpaper | hardvocer | hardware | hd | hd bd | hd. bd | hd. bd. in slip case | hd. bd.in sl.cs | hd. bk | hd. cover | hd.bd | hd.bd. in box | hdb | hdbd | hdbk | hdbkb | hdbkhdbk | hdbnd | hdc | hdcvr | hdk | hdp | hdpk | hradback | hradcover | hrd | hrdbk | hrdcver | hrdcvr

And that’s after eliminating things like places of publication, strings like “with…”, “plus…”, “alk. paper”, etc.

“Yeah, but you have to understand that historically…”

Stop hiding behind that.

I understand that at one point in time it probably made sense (to someone at least) to do it this way. I can deal with that.

What I can’t accept is that as I type this there’s a cataloger doing this in this way. Today. April 2011. Some, what? maybe thirty years since computer-based OPACs became prevalent?

These sorts of problems were recognized ages ago and should have been dealt with. Add a subfield. Invent a controlled vocabulary. Don’t worry about the legacy data; it’s always going to suck.

But why are we still producing sucky data???

To sum up

The point is that there’s a better way to do this stuff. Lots and lots of better ways, in fact. Time I spend dealing with crappy data is time I don’t spend making relevancy raking better, or building a better command language search option for my librarians, or working on ways to get a decent “more like this”.

The need is both dire and urgent; the latter because sooner or later we’re going to have to go to a “two state solution” with traditional MARC21 for many of our records and whatever comes next (RDA?) for the newer stuff. And every day we wait, that first category grows, and the growth rate keeps increasing.

And then there’s serials. Don’t talk to me about serials.

One of the frustrating things about dealing with MARC (nee AACR2) data is how much nonsense is stored in free text when a unique identifier in a well-defined place would have done a much better job.

A lot of people seem to not understand why.

This post, then, is for all the catalogers out there who constantly answer my questions with, “Well, it depends” and don’t understand why that’s a problem.

Description vs Findability

I’m surprised — and a little dismayed — by how often I talk to people in the library world who don’t understand the difference between description and findability. AACR2 is clearly designed for description; once you’ve found a record, it does a pretty good job telling a human being what she’s looking at. With respect to a person who’s already got a copy of the record in her (virtual) hand, strings of text and reasonable abbreviations are…well, often good enough, let’s say.

But much of AACR2 is a giant mountain of fail when it comes to supporting findability — the ability for a machine to slice and dice the data in ways that can be mapped onto searches and transformations. What those of us on the business end of the computer need are well-defined values stuck into well-defined places that represent well-defined relationships.

Free text stuck on the end of a field fails all three of those criteria.

Machine Reasoning vs. Machine Parsing

When many people look at something like RDF, their first reaction is, “Great Googally Moogally! Just tell me the language! I don’t want to follow a chain of reasoning that’s seventeen steps long just to figure out the damn thing is in English!!!”

Of course you don’t. And you don’t have to. Someone — hopefully someone smarter than me — needs to write a program to do it. And we can.

Following all that logic — deriving relationships, figuring out eventual values, determining how to convert between various forms — is what I’ll call (for simplicity’s sake) machine reasoning. And machine reasoning — for the purposes of this discussion, anyway — is a solved problem. I’m not saying it’s not hard, and I’m not saying it might not take gobs of hardware resources. But we, the collective of humanity, know how to do it.

On the other hand, machine parsing — looking at all that free text that is sprinkled throughout our records and trying to turn it into something that is susceptible to machine reasoning — is vehemently not a solved problem. Even if you ignore all the misspellings, we’re still stuck with one-off abbreviations, lack of ordering, gobs of “local practice,” and iffy punctuation.

And, come to think of it, you can’t ignore the misspellings, either.

The point is this: good data trumps everything else. If there’s good, solid, well-defined data in computable places, we can (given some time) do damn near anything with it. If there’s human-entered, free-text, parenthetical-remark-type data, we’re pretty much stuck.

Examples?

Jonathan Rochkind just did a great post looking at LC call numbers, and how, well, they might be in a few different places, and may or may not be valid LC call numbers, and so on and on and on and on.

And my next post (hopefully tomorrow) will be an analysis of the first freetext in MARC I ever tried to deal with — the parenthetical remarks in the 020 (ISBN) field. If that doesn’t keep you up all night, well, I don’t know what will.

…at the same URL.

I was, to put it mildly, incredibly excited about code4lib this year because, for once, I thought I had something to say. And I did have something to say. And I said it. But it was wrong.

I presented a bunch of statistics drawn from nearly a year of Mirlyn logs. The most outlandish of my assertions, and the one that eventually turned out to be the most incorrect, was that some 45% of all our user sessions consist of only one action: a search.

Unfortunately, I’d missed a whole swath of things I should have excluded. I’d remembered robots and stuff coming in from our link resolver and so on. I hadn’t counted on having to fight my own stupidity.

In short: catalog.hathitrust.org and mirlyn.lib.umich.edu share a common code base, as well as a Solr backend. I was correctly excluding all the HathiTrust stuff from my stats except for simple searches. What I ended up with was a whole lotta sessions with nothing in them but that search. Luckily, I noticed waaaay too many people coming in via the HathiTrust site (which I know doesn’t have a link to Mirlyn) and did more digging.

The slides have been updated with correct numbers. Luckily, even though the adjustment was pretty extreme, I don’t think many of my conclusions are invalidated, especially given corroborating evidence from an extensive survey conducted by our usability team (PDF). They conclude, among other things, that known-item searching is prevalent and relevancy raking is important across task boundaries.

The basic stats from the powerpoint, for those who don’t want to read all my notes:

• 17% of all sessions have one action: a search
• —In only 28% of all sessions does the user see the Record View
• 75% of all logged actions that target an individual record (see the full record view, look at extended holdings, etc.) happen with a record in the top 6 search results
• 7% of sessions involve a user adding a facet
• 2% of sessions involve a user exporting records