[Note: this isn't so much a Stupid Solr Trick as a Thing You Should Probably Know; consider it required reading for the next SST. If you're just joining us, check out the introduction to the Stupid Solr Tricks series]

What the heck is a localparams query?

A garden-variety Solr query URL looks something like this:

  http://localhost:8983/solr/select?
    defType=dismax
    &qf=name^2 place^1
    &q=Dueber

Which is fine, as far as it goes. But it’s easy to run into the limits of the standard query plugins (e.g., Dismax).

Say, for example, you want something like this:

  title:Constructivism AND author:Dueber

And furthermore, you have multiple underlying fields (title1, title2, title3, author1, author2).

The naïve approach would be to just do this:

  defType=dismax
  &qf=title1 title2 title3 author1 author2
  &q=Constructivism Dueber

But you can’t construct a dismax query with the boolean AND. You can with edismax, but even then you’ve got no way of telling (e)dismax that Constructivism must be found in the title fields, and Dueber must be found in the author fields. Dismax doesn’t do that.

Solution: Build a query of queries

The solution is to build a query made up of fully-encapsulated sub-queries. A localparams query has two forms (note that, of course, you’d need to URL-Escape the values):

  _query_:"{!dismax qf='field^2 otherfield^4'}my search terms"
or
  _query_:"{!dismax qf='field^2 otherfield^4' v=$q1}"&q1=my search terms

I far prefer the second form (which uses a second URL parameter q1 instead of sticking the search right in there), because I don’t have to worry about escaping double-quotes in the query terms (as you would if there’s a phrase as part of the query).

Once you’ve got these things, you can combine them with booleans.

    q=query:"{!dismax qf='title1 title2 title3' v=$q1}" AND
      query:"{!dismax qf='author1 author2' v=$q2}"
  &q1=Constructivism
  &q2=Dueber

[Note: be careful with solr booleans!!!]

You can add any local parameters you need (for dismax, stuff like mm, qs, pf, and ps) and you can use any query parser you want by changing what comes after the bang (e.g., {!lucene ...} or {!edismax...}).

In this way, you can build up arbitrarily complex queries using any available query parsers in combination with each other. Very powerful.

An example: boost records that contain all terms

Just about everything in a localparams query can be pulled out in the way I pulled out the search terms above. Here’s a fairly-complex example (which, let’s be honest, would be a lot more complex if you were trying to inline and escape everything).

Scenario: We want to do a logical-OR search (mm=0%), but want to make sure we boost documents that contain all the search terms. This is necessary because sometimes a very long document with all the terms will have a lower score than a very short document with most of the terms.

Having short document with a few keywords show up before long documents with all the keywords will drive your librarians CraZy!!! So it’s tempting to just leave it alone. But let’s fix it anyway.

The gist of it is as follows:

• Query against title and author
• Use an mm of 0% (logical OR) for the main query
• Use a pf to boost on a phrase in those same two fields (just common sense)
• Set up a boost query (bq) to boost the score if all the search terms are present

To accomplish this, we’re going to have two localparams queries: one to be the main query, and another that we’re going to use as the boost query. This works in much the same way as our previous “AND-together two localparams queries” did.

[Presenting the URL parameters as a ruby hash to make it easier to read]

What’s nice about this is that I’m reusing the search terms (for the main query and the boost query) and field list (for the query field and the phrase fields) so I don’t have to repeat them.

Try along at home

First off, if you don’t have a browser that does nice XML and JSON formatting, well, get one. I use Chrome with JSONView and XMLTree, but I’m sure there are equivalents for Firefox. They’ll make your life easier.

By now you know the drill:

We’ll want to empty out the index and put in some documents to work with. I’m presuming you have curl installed. If not…well, you’re on your own.

You might want to take a look at the localparams.json file, which contains a set of documents in the new JSON update structure. The full Solr JSON Update structure allows repeated keys. Apparently, so does the JSON RFC:

> 2.2. Objects > An object structure is represented as a pair of curly brackets > surrounding zero or more name/value pairs (or members). A name is a > string. A single colon comes after each name, separating the name > from the value. A single comma separates a value from a following > name. The names within an object SHOULD be unique. (emphasis mine)

“SHOULD”. Not “MUST”. I don’t care if it’s legal. It still weirds me out.

Once you’ve got solr running in the background, you can go ahead and try our query!

• If you’re really lazy, just click the link
• If you’re slightly less lazy, and you’ve got ruby installed, take a look in the new ruby directory. You can run ruby browse.rb localparams_query.rb to run the query and have it automatically open up in your browser.
• If you’re ambitious, you might want to actually mess with the localparams_query.rb file so you can try things out.

As a longish side note, we’ll probably use browse.rb in the future of this series as well, so you might want to go ahead and get ruby installed if you don’t already. RVM is the easiest route if you’re on linux/OSX. You can also just install JRuby, seeing as how you’re running java anway (just make sure to use 1.9 mode by calling stuff as jruby --1.9 myscript.rb or setting the environment variable export JRUBY_OPTS=--1.9).

Special Stupid Solr Trick: Make a special query handler for a complex query

OK, so I said I wouldn’t have a real SST in this episode, but it’s so damn long at this point I figure I’ve lost everyone except Rochkind (Hey, Jonathan!), so let’s throw one in.

The Solr configuration file solrconfig.xml is where you can configure custom search handlers. In such a custom handler, you can specify defaults (which, by default, can be overridden by passed-in parameters, although you can control that, too) — this is commonly used to, say, put in a q.alt or a filter query that will always be applied.

But we can use it to put in our special query defaults that boosts when a document contains all the terms:

If you look closely, you’ll see that everything you need is defined in this requestHandler in the solrconfig.xml file, except for $fields and $qwords. You could also override mymm by passing in an argument with that name, if the default ’0%’ isn’t to your liking.

If you’ve been following along at home, this requestHandler is already in the solrconfig.xml file that you’re running right now. Go ahead and try it! Let’s search for the terms ‘dueberb’ and ‘penn’ and see if the correct record floats to the top.

http://localhost:8983/solr/edismaxplus/?qwords=dueber penn&fields=author title

Nifty, huh?

Next time we’ll use a local params query to get around something about dismax that drives me crazy: preventing (or penalizing) matches that go across a field’s multiple values.

[For the introduction to this series, take a quick gander at the introduction]

Like everyone else in the library world, I’ve got a bunch of well-defined, well-controlled standard identifiers I need to keep track of and allow searching on.

You know, well-vetted stuff like this:

• 1234-5678
• 123-4567-890
• 12-34-567-X
• 0012-0045
• ISBN13: 1234567890123
• ISSN: 1234567X (1998-99)
• ISSN (1998-99): 1234567X
• 1234567890 (hdk. 22 pgs)
• 9
• Behind the 3rd floor desk
• Henry VIII

[Note: some of these may be a titch exaggerated]

How does your system deal with these on index? How about on query?

Here’s an idea of how to use a custom solr fieldtype to do the heavy lifting.

What we’re shooting for

I’d like to be able to send in a text string as follows:

• The input can contain other text besides the id
• The ID starts with a digit and consists solely of digits and (optional) dashes, then ends with a digits and possibly a trailing ‘X’ or ‘x’ so we can deal with ISBN/ISSN
• The ID has to be at least N characters long (for this example, I’m using N=8); this helps us avoid other text that might trivially look like an ID but isn’t.
• Only the ID itself is indexed
• If no valid ID is identified, nothing is indexed

The numericID field, suitable for ISBN/ISSN/OCLC/etc.

Let’s take a look at the end product and then walk through it.

Things we’ll be learning about today

NOTE: I really, really recommend taking a look at Scaling Lucene and Solr by the good folks over at Lucid Imagination for great, short explanations of omitNorms, term frequencies, etc.

Since this is the first post, I’ll go over some stuff that’s probably a little too basic for any audience that’s likely to show up here, but what the heck.

• KeywordTokenizer
• PatternReplaceFilterFactory
• LowerCaseFilterFactory
• LengthFilterFactory

Step 1: “Tokenize” to a single token

The job of a tokenizer is to decide how to split your input into individual tokens (often “words”), which are then munged by any filters you’re applying.

For the case of an ID, we don’t want to tokenize. At least at this juncture, I’m not trying to extract multiple valid IDs out of a single string; I’m just trying to determine if there’s a valid ID in there somewhere and throwing everything else away.

In other words, I’m going to treat the input as a single token, and then munge the bejeebers out of it in order to get what I want.

In the Solr world, that leads us to the confusingly-named KeywordTokenizer.

What we have now: exactly what we started with

Step 2: Find the first thing that looks like an ID and mark it

I primarily work in Ruby and Perl, which means the dramatic abuse of regular expressions is just part of my daily life.

Line 5 is our first use of a regexp in the filter chain via PatternReplaceFilterFactory.

The idea is to:

1. Find something that looks like a match
2. If found, get rid of everything else, and throw a ‘***’ onto the beginning so later on I can tell if I matched or not.

The second step is a little…odd…but necessary because I need a way to know if I found a candidate ID or not. If I did, well, there will be three asterisks on the front of the string from here on out. If not, there won’t.

This is a little confusing as these things go, so I’ll break it down.

Line 6: the match:

• Skip any amount of stuff we don’t care about (.*?)
• Match a number (\p{N}) (that’s unicode regexp syntax, if you haven’t seen it)
• Match a string of at least 6 numbers and dashes
• Close with an optional X or x [Xx]?
• …and any trailing bits until the end of the string.

So…[number][six numbers/dashes][optional X]

At minimum, that’s seven digits/dashes.

Line 7: replacement

• Replace the whole string (note how I anchored the match with ^ and $?) with whatever was matched inside the parentheses (represented here by $1) after prepending a set of three asterisks ‘***’

What we have now: If we found a candidate ID, we have that string prepended by ‘***’. Otherwise, we have exactly what we started with.

Step 3: If we didn’t find a match, throw it all away

Line 9 shows an attempt to match on any string that start with an asterisk (which we’re pretty sure we won’t see because that’s illegal lucene wildcard syntax). If we have a string that doesn’t start with an asterisk, then throw the whole damn thing away because we don’t have a candidate ID anyway.

[There's a strong argument to be made that using an asterisk as the tagging character is a bad choice. Anyone have suggestions?]

What we have now: Either a candidate ID string preceded by ‘***’ or the empty string.

Step 4: Ditch the ‘***’ used to mark a candidate ID

Lines 10-11

Find the ‘***’ and throw it away.

What we have now: The raw candidate ID string or the empty string.

Step 5: Lowercase it

Line 12.

By ‘it’ I mean “any X that might be trailing the ID”; we should have thrown everything else away by now. (Note: could have done this with a PattenReplace as well, obviously; not sure why’d I’d choose one over the other).

What we have now: The raw candidate ID string with its optional trailing ‘X’ lowercased, or the empty string

Step 6: Get rid of everything that’s not a number or an ‘x’

Lines 13-15

Ditch any dashes that are remaining. I’m doing it like this instead of just ditching the dashes because I’ll likely modify this at some point to allow, e.g., periods between numbers, or maybe spaces. This is safer.

Note the extra parameter (replace=”all”), indicating that I want to replace all occurrences. This hasn’t been an issue until now because I’ve been careful to match the entire string by anchoring the pattern at the beginning (‘^’) and end (‘$’).

What we have now: A string of numbers possibly followed by an ‘x’, or the empty string.

Step 7: Make sure what we have is a reasonable length

Line 16

Now that we’ve gotten rid of the dashes, we need to make sure we have enough digits left to make a valid identifier.

If we didn’t match originally, it quickly got reduced to the empty string, and that will disappear here due to having length 0.

It’s also possible that our initial match was, say, ’1—-3—–6—7′, which will at this point have been reduced to just ’1367′ — too short for our taste.

In this version, I allow strings of any length between 7 (old OCLC number) and 14 (barcode).

What we have now: A string consisting purely of 7-14 characters, the last of which may be an ‘x’, or nothing at all (e.g., nothing will get indexed).

Step 8: Remove leading 0s

My ILS (Aleph) loves to zero-pad all its local identifiers. I’d rather get rid of them.

What we have now: What we had before, but with no leading zeros

Let’s try it!

If you’re following along at home, get the latest version of the schema and try it!

…and then:

• Go to the analysis page at http://localhost:8983/solr/admin/analysis.jsp?highlight=on
• Set the first line of the form to use Field: type and input numericID
• Check the “verbose output” checkbox under Field value: index
• Put in a test value and see what the analyzer gives you!

For those of you not following along at home, here are the examples from waaaaaay at the top of this post:

• 1234-5678 => 12345678
• 123-4567-890 => 1234567890
• 12-34-567-X => 1234567x
• 0012-0045 => 120045
• ISBN13: 1234567890123 => 1234567890123
• ISSN: 1234567X (1998-99) => 1234567x
• ISSN (1998-99): 1234567X => 199899
• 1234567890 (hdk. 22 pgs) => 1234567890
• 9 => [nothing]
• Behind the 3rd floor desk => [nothing]
• Henry VIII => [nothing]

So…not too bad. We did miss one, mistaking a year range for a numeric ID, but if your data are that bad, there’s only so much we can do.

Conclusions

Obviously, this is the tip of the iceberg with this sort of thing. And it can still be confused.

But it does follow our goal of having the exact same behavior on index and query, moving the logic to solr, and being pretty flexible.

Perfect? No. Useful? Yes.

Completed parts of the series:

1. A Solr Field Type for numeric(ish) IDs
2. Using localparams in Solr (or, how to boost records that contain all terms)
3. Requiring/Preferring searches that don’t span multiple values
4. Boosting on Exactish (anchored) phrase matching

Those of you who read this blog regularly (Hi Mom!) know that while we do a lot of stuff at the University of Michigan Library, our bread-and-butter these days are projects that center around Solr.

Right now, my production Solr is running an ancient nightly of version 1.4 (i.e., before 1.4 was even officially released), and reflects how much I didn’t know when we first started down this path. My primary responsibility is for Mirlyn, our catalog, but there’s plenty of smart people doing smart things around here, and I’d like to be one of them.

Solr has since advanced to 3.x (with version 4 on the horizon), and during that time I’ve learned a lot more about Solr and how to push it around. More importantly, I’ve learned a lot more about our data, the vagaries in the MARC/AACR2 that I process and how awful so much of it really is.

So…starting today I’m going to be doing some on-the-blog experiments with a new version of Solr, reflecting some of the problems I’ve run into and ways I think we can get more out of Solr.

Premise 1: put all the logic you possible can into Solr

Much of what I’ll be doing is looking at new field type definitions that are appropriate (in my mind, anyway) for library data. Some of this stuff (e.g., normalizing ISBNs) would be a lot easier to do in your indexing code.

But then you’d have to do it again in your application to munge whatever is entered in the search box. And maybe it won’t be the same every time. Or maybe you don’t want to write a freakin’ parser to try to find anything that might look like an ISBN and mess with it.

I take it as gospel that you should put all your logic into the solr field analysis chain, so the exact same thing is happening on index and on query. That way, even if it’s wrong, at least it’ll be wrong in the exact same way and your users will find the stuff they’re looking for.

Premise 2: Doing it crappily is better than not doing it at all.

Look, the right way to do much of this stuff is by hacking on Solr itself, building custom field analyzers or filters or tokenizers that mess with the token chain and…

Wait. I already lost myself, and probably you, too. At some point, I’m going to do an actual sample custom filter for the new Solr codebase (the stuff I did once before is out-of-date); the example will be LCCN normalization and you’ll be able to follow along with me on this blog.

But in the meantime, we can do a lot of fairly ambitious stuff just by using and abusing the out-of-the-box stuff: pattern replacement filters, the existing tokenizers, etc. It might be ugly, and not very fast, but if I start getting the 200 hits a second that mean this is a bottleneck for me, I’ll be happy to deal with it then.

Premise 3: It’s always better to put something out there so smart people can tell you how to do it right

One of the disappointments in my life right now is that there isn’t more formal and informal discussion about what people are doing/trying. I’m sure it’s out there, but some of it is buried in a sea of application-level crap, and much of it is ignored by the people that really understand the data.

With luck, I’ll get comments from folks who really know their stuff and can tell me, in excruciating detail, exactly how I don’t. Please: correct me. I might not be the brightest guy in the room, but I know enough to try to outsource my thinking.

Follow along at home!

Option 1: Build your own current-trunk Solr

If you want to follow along at home, you’ll need a copy of the current source (not the 3.5 stable, since I use things like the ICUTokenizer coming in 3.6 / 4.0), which you can find and build from the Solr site.

Option 2: Just use what I’m using

Alternately, if you’re lazy (and who isn’t??), I’ve provided a github repo of the standard solr “example” directory you can nab and run on your own java-equipped machine.

Warning: the git repo is currently 60MB or so.

…and then head to your local Solr Admin page page on port 8983 to check things out. We’ll be spending most of our time in the analysis tab.

I’ll get the first post in the series up later today, and then every few days as I think of more things to talk about. I hope you’ll join me!