Comments for Robot Librarian

Comment on Why bother with threading in jruby? Because it’s easy. by Bill

Bill — Fri, 12 Mar 2010 04:04:19 +0000

The assumption is that the producer is faster than the consumer (otherwise, why bother to have multiple consumers). A regular Queue (not sized) would grow without bound based on the speed difference between consumption and production. We don’t, for example, want 10K lines in memory while we’re waiting for consumers to turn them into MARC objects or whatnot.

A SizedQueue will block on both enqueue (if it’s full) and dequeue (if there’s nothing in it), so it’s exactly what we need for this kind of thing.

Comment on Why bother with threading in jruby? Because it’s easy. by Jonathan Rochkind

Jonathan Rochkind — Fri, 12 Mar 2010 03:41:11 +0000

What’s the purpose of using a SizedQueue instead of an ordinary Queue? What if the producer produces so much faster than the consumers consume, that the threads*4 size is exhausted, what happens? Does the producer just block waiting for there to be room to enqueue?

Comment on Why bother with threading in jruby? Because it’s easy. by Jonathan Rochkind

Jonathan Rochkind — Fri, 12 Mar 2010 03:35:12 +0000

Nice. You wrote that one? Ruby’s pretty sweet, huh?

Comment on Ask, and you shall receive, and it shall be AWESOME! by Santy Chumbe

Santy Chumbe — Wed, 10 Mar 2010 13:09:22 +0000

Is the API of journaltocs (www.journaltocs.hw.ac.uk) the Shetland pony you were waiting for?

Comment on New interest in MARC-HASH / JSON by GregPendlebury

GregPendlebury — Sun, 07 Mar 2010 23:10:30 +0000

Bill, I’d be interested in hearing if anyone has put up a central site for this in terms of schema definition and/or collaboration. This is an area I think could gain some traction here, but going off on our own without a public schema doesn’t seem productive.

Comment on Pushing MARC to Solr; processing times and threading and such by Jonathan Rochkind

Jonathan Rochkind — Fri, 05 Mar 2010 19:25:38 +0000

Hey, I should read more carefully before I post, but instead I’ll just multi-post.

I see the serialization to XML itself is non-trivial too.

json!

Comment on Pushing MARC to Solr; processing times and threading and such by Jonathan Rochkind

Jonathan Rochkind — Fri, 05 Mar 2010 19:24:02 +0000

Oh, I see, performance with toXML.

What i wonder/worry about, is if the added time for toXML isn't actually the serialization to xml, but simply that if you're pushing a larger stored field to solr, that's going to slow things down.

We still need to store our marc either way, of course. The UWisconsin approach of storing marc in an rdbms instead of a solr stored field may or may not speed up indexing, since it's still gonna take time to store it.

Comment on Pushing MARC to Solr; processing times and threading and such by Jonathan Rochkind

Jonathan Rochkind — Fri, 05 Mar 2010 19:16:25 +0000

What's HLB?

Both ruby-marc and marc4j will generate marc-xml, but do you mean optimizing speed of it? (Don't forget marc-json possibilities! heh).

Not sure if you're still happy with marc4j or might prefer ruby-marc, I realized one thing missing from the ruby stack (if you didn't want to use marc4j) (as far as I know) is the marc8-utf8 conversion stuff, and heuristic guess detection of marc records that aren't really the encoding they claim to be.

Comment on New interest in MARC-HASH / JSON by Bill

Bill — Thu, 04 Mar 2010 23:26:46 +0000

Heh. Yeah, the major element in the hash is a big ol’ array of fields, composed of arrays of subfields. My original take on MARC-Hash was mostly as a hash, and resulted in my first real, painful schooling in how batshit-insane MARC is.

Comment on New interest in MARC-HASH / JSON by Naomi Dushay

Naomi Dushay — Thu, 04 Mar 2010 22:57:14 +0000

by “hash” do you mean “array”? Because order matters in Marc, but ruby hashes do not guarantee order, right?