Comments on: Still another look at MARC parsing in ruby and jruby http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/ Disclaimer: I'm not actually a robot. Thu, 08 Jul 2010 20:37:05 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Bill http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/comment-page-1/#comment-292 Bill Mon, 01 Feb 2010 15:08:59 +0000 http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/#comment-292 <p>The XML parsing code for ruby-marc in jruby is a stax-based implementation I wrote from a position of intense ignorance. Nabbing the xml-parser out of the marc4j code is probably a next step when I get time. For the moment I've just re-opened the various marc4j classes to add syntactic sugar where necessary.</p> <p>BTW, can I tell you how much jruby rocks????</p> The XML parsing code for ruby-marc in jruby is a stax-based implementation I wrote from a position of intense ignorance. Nabbing the xml-parser out of the marc4j code is probably a next step when I get time. For the moment I’ve just re-opened the various marc4j classes to add syntactic sugar where necessary.

BTW, can I tell you how much jruby rocks????

]]> By: Charles Oliver Nutte http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/comment-page-1/#comment-291 Charles Oliver Nutte Mon, 01 Feb 2010 14:35:30 +0000 http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/#comment-291 <p>The XML performance in ruby-marc is almost certainly due to whatever XML parsing library it uses. Do you know what library that might be?</p> The XML performance in ruby-marc is almost certainly due to whatever XML parsing library it uses. Do you know what library that might be?

]]> By: Jonathan Rochkind http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/comment-page-1/#comment-289 Jonathan Rochkind Fri, 29 Jan 2010 22:36:25 +0000 http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/#comment-289 <p>So one thing about marc-ruby, not in parsing but in analyzing/processing, is every time you ask for a tag (say, '245'), it's got to iterate through every field in the record, and match each one to see if it's a 245. If you're doing a lot of 'mapping' to a lot of records.... I've wondered for a while if there's a bottleneck there, and how much difference it would make make to build a hash 'index' of tags. Almost all 'mapping' operations begin by looking up tag numbers, and the majority pretty much end there too.</p> <p>But I've never gotten around to trying to profile it.</p> <p>Curious how marc4j is implemented in terms of access to marc record fields by tag.</p> <p>Since your tests reveal that even ruby-marc is faster under jruby than mri, there's something just about the environment that is speeding things up. But I wonder if there are optmizations (perhaps simple ones) that could be made to ruby-marc to make it catch up with marc4j.</p> So one thing about marc-ruby, not in parsing but in analyzing/processing, is every time you ask for a tag (say, ‘245′), it’s got to iterate through every field in the record, and match each one to see if it’s a 245. If you’re doing a lot of ‘mapping’ to a lot of records…. I’ve wondered for a while if there’s a bottleneck there, and how much difference it would make make to build a hash ‘index’ of tags. Almost all ‘mapping’ operations begin by looking up tag numbers, and the majority pretty much end there too.

But I’ve never gotten around to trying to profile it.

Curious how marc4j is implemented in terms of access to marc record fields by tag.

Since your tests reveal that even ruby-marc is faster under jruby than mri, there’s something just about the environment that is speeding things up. But I wonder if there are optmizations (perhaps simple ones) that could be made to ruby-marc to make it catch up with marc4j.

]]> By: Jonathan Rochkind http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/comment-page-1/#comment-288 Jonathan Rochkind Fri, 29 Jan 2010 22:12:11 +0000 http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/#comment-288 <p>Oh wait, I see it now! Nevermind.</p> Oh wait, I see it now! Nevermind.

]]> By: Jonathan Rochkind http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/comment-page-1/#comment-287 Jonathan Rochkind Fri, 29 Jan 2010 22:10:48 +0000 http://robotlibrarian.billdueber.com/still-another-look-at-marc-parsing-in-ruby-and-jruby/#comment-287 <p>These kinds of numbers always make my head swim a bit.</p> <p>The original question you had, if I understand right, was if you could parse MARC at 250/300 records per second. But your findings aren't expressed in records per second. Is it possible to say how they add up in records per second, so we know which methods are fast enough to meet your original criteria and which aren't?</p> <p>(You must have a really fast machine to get 250/300 records per second from SolrMarc. I only get 100 records per second on my machine.)</p> These kinds of numbers always make my head swim a bit.

The original question you had, if I understand right, was if you could parse MARC at 250/300 records per second. But your findings aren’t expressed in records per second. Is it possible to say how they add up in records per second, so we know which methods are fast enough to meet your original criteria and which aren’t?

(You must have a really fast machine to get 250/300 records per second from SolrMarc. I only get 100 records per second on my machine.)

]]>