Very interesting to read about your success rate compared with VIAF’s AutoSuggest. I wonder if AutoSuggest is optimized for fuzzy matches, making it work less well with names that are already fairly close matches. I know I’ve experienced odd results with ElasticSearch and Apache Solr if you don’t specifically tune things for the data you’re working with. If VIAF uses one of those search products, that might explain it.

I’m so pleased to have discovered your incredibly thoughtful blog. You have a new regular reader.

The close correspondence between records and named graphs (as units of data management) makes named graphs (especially via the SPARQL 1.1 Graph Store protocol) a very convenient and useful technology for bridging legacy metadata stores to the Linked Data world.

You may find some of Jörg Prante’s explorations with Elasticsearch and bibliographic data helpful: http://jprante.github.io/

george

just another great post. I have some comments:

1) you are right, that XC is (partly) a middleware, but it has a discovery interface as well embeded into Drupal

2) I beleive that classification systems provide a wider network for each books, thus making connections between distinct books. And discovery systems could provide a number of links (e.g. other works of the same author), each give additional contextual information for the users. Facets do the same, they are not just “dressing”, but contextual information as well.

3) I totally agree with the importance of text mining.

4) You did not mention the importance of semantic web. The promise is to give additional information to catalog records (or even full text), which are not inside the original data (for example: Niagra Falls’ coordinates, or the name of its mayor in 1884). As I see it is as much important as text minig, and it could improve the effectiveness of text minig.

I am not fully agree with this. For me the direct purpose of code4lib (not of the conference, because I haven’t been there, but of the site, the listserv, the IRC channel and the journal) is to sharing ideas and practice, and this effects the daily work of code4libbers (at least mine life). But the grater effect on the whole library community is the improving quality of the softwares we wrote, and thus the hopefully improving quality of the library services. The wider audience of librarians does not care about the technical details of communication standards and Perl regex tricks, and it is not needed at all I guess. So it is the products which qualifies the effectivity of the community, not the size of the membership, according to my point of view.

Anyway: thanks for the travelouge!

FYI – it’s “Flesch” readability (not Flesh)

Look forward to seeing more of this.

Laurie

force.label.add(pv.Label).font(’14px sans-serif’).textStyle(‘bold’);

to:

force.label.add(pv.Label).font(’14px sans-serif’);

the problem seems to have gone away on my part. Fixed?

when I see the examples none of the words appear as label, only the nodes and axes, while in your screeshot there are labels as well. The JS API would be interesting in using XC UI for facets or for visualizing subjects. Thank you for the post!

Péter

Check it out: Click on ‘Publication Year’ on any result set after doing a search [or click on the graph on the main page, but I think it’s more useful after a search], and see a timeline; you can also ‘drill down’ into the timeline, and you’ll then see a sort of ‘zoomed in’ timeline of the range you selected.

https://blacklight.mse.jhu.edu/demo

The resolution of the bars is currently set at around 10, but I believe the underlying Solr search engine powering this could handle higher — 20, certainly — 100, maybe but not sure. Haven’t played with it, since this seemed good enough for now and I had other things to work on. It’s actually just a parameter in the code though, could be set to any number. And of course the graph could be displayed larger, as on the first page, but didn’t seem useful in this interface to do so.

I think it turned out pretty slick.

That’s an under development demo of the blacklight/solr powered opac replacement we’re working on, it’s not in production and is a bit wonky occasionally. And yes, it’s got a non-trusted SSL cert sorry, need to fix it soon.

I wonder if it would be better done in reverse. You need all the info to know where the terms will locate. Run it backwards with fixed term locations, reducing size of terms accordingly. Then re-run the sequence forwards. I have no idea how to do that!

I failed at algebra and calculus but got straight A’s in geometry. Too bad the former were shoved down our throats, otherwise I’d have had a better impression of math.

The primary purpose of the project is to determine whether or not “greatness” can be measured. I am using the Great Books of the Western World as a sort of control group for this study. Alas, Atlas Shrugged is not apart of the Great Books. The survey is a way of testing my mathematical model. So far the results are not conclusive.

In a previous posting I describe how I counted the number of times each of the Great Ideas appeared in each of the Great Books. [1] From that analysis it would be possible for me to objectively add tags to the books based on the “Great Ideas Coefficient”. Thus, I would be able to assign the word “love” to Shakespeare’s Sonnets and “war” to Aristophanes’s Peace. I would then be able to juxtapose works associated with any of the Great Ideas.

But according to Hutchins, one of the characteristics of “greatness” is the ability of a work to describe and discuss a wide range of ideas, not just a few. If this is the case, then I need to juxtapose every work to every other work with every idea. Doing the random thing covers all of those options.

Furthermore, there is absolutely nothing wrong with answering “I don’t know and/or neither”. Since I am only in interested in which work is greater than the other a non-response does not effect the tabulation. I don’t need to know what people don’t know. I only need to know what people do know or think they know.

Finally, the purpose of the survey is to determine if there is a correlation between my Great Ideas Coefficient and people’s opinions. So far the correlation is weak but I figure I don’t have nearly enough survey data to make a real evaluation.

Again, thanks for the feedback. The whole thing is an digital humanities investigation in a crowd sourced context.

[1] – http://bit.ly/cMBjEO

I work in reference at a library, and I’m also an alumna of St. John’s College, which is based on the Great Books program, so I found your experiment interesting on two levels.

Thanks much

Thank you so much for your highlighting the Friends of the Library Celebration of the 100th Birthday of the Roanoke Public Library! As president of the Roanoke Public Library Board, I appreciate what you have written and how the Michael Hart presentation impacted you. I too find it fascinating that we are so tied to physical books that we tend to forget that the purpose of libraries is to provide information retrieval of various media and not just one particular kind to as many people as possible. Even library boards don’t get that concept too well!

Just by way of clarification, the Evergreen Indiana system provides us access to over 72 libraries, including the State Library, at no charge to our patrons. Individuals may request materials not in our library from any one of those libraries, have them delivered to our site, pick them up here and then return them here after a three week period. In addition, Evergreen Indiana provides computer cataloguing and check out systems at no charge to the local library. Not only does it enhance our collection and provide greater access to materials, I believe it is the wave of the future in Indiana as we move to an integrated shared system of materials in this state.

Also, our current library has 1400 square feet of space and about 14,000 volumes in house. Evergreen Indiana gives our patrons access to over 3 million items across the state.

Again, thanks for being at our program and for the video. It and the text were great!

Sincerely,

Brian W. Secor
President, Roanoke Public Library Board

Mind you, I am certainly not saying that information retrieval and metadata issues are not important. They most certainly are. But in a time when people can easily acquire such large sets of relevant information than they know what to with, I think it is more important to discover ways to make useful and understandable. Find is only the very beginning of the research process. It is not an end in itself. I believe digital humanities computing techniques are one way to make that information more useful. Consequently, I am encouraged by your comments, and I hope I can make it to TPDL 2011 in Berlin.

If you do go about trying to integrate Alex with Facebook using Open Graph Protocol definitely check out their URL Linter, which lets you see how they are extracting information from your page…which is very useful indeed. Open Graph Protocol is actually using RDFa for the metadata embedding pattern, which is neat. The use of RDFa plus their opening up the code kinda makes me want to use Facebook more :-)

And a note — my last name has only one ‘u’. :)

—
ELM

very interesting post and demonstration. One suggestion: maybe you should use a synonym dictionary (and a formal taxanomy) to transform those words in the text which has synomys for the words in the __DATA__ section. Eg. algebra is a child concept of mathematics, but if the text won’t mention math, just algebra, the text won’t match the idea.

Regards,
Péter

Has me thinking about other disciplines and what kinds of tools that exist to mine their data. It’s been my problem with OpenCalias – it has such a business focus that it is great for an analysis of data where the user wants to use the results in business.. but not so great for other disciplines. I think librarians need to look toward using textual analysis tools from the disciplines they serve and incorporate it into their “beyond discovery” layers.

It’s a pity that often subject liaison librarians in universities do this job because they are not so interested in “techie things”….How to get them to understand that this is their role will be interesting.

This kind of discovery also depends obviously on full text. It creates a population bias that would need to be somehow overcome. For example if fewer books by female authors are digitised (as fewer books by female authors were added to canons of “great works” in the past) then an attempt to discover what “love is” may produce a very skewed view …

BTW, while great-books.xml is the canonical file representing the collection/investigation, most people will want to use .csv version for analysis in things like R. I plan on elaborating on this idea in the next posting, but in the meantime, you might want to try http://infomotions.com/sandbox/great-books/etc/great-books.csv

Again, thank you

Thanks for your inputs. However I still unable get to work. I tried both methods. Did it works for you? My search found 0 document when I multiple fqs.

# build an array of facets
my @fqs = ( ‘systype:LA’, ‘location:US’ );

# build the search options
my %search_options = ();
$search_options{ ‘start’ } = $start_record,
$search_options{ ‘rows’ } = $maximum_records;
$search_options{ ‘fq’ } = [ @fqs ];

# do the work; search
my $response = $solr->search( $query, \%search_options );

I had similar issues in my Alex Catalogue. Specifically, it is possible to enter a query, get a search result, and refine it through any number of facets (Solr fq values). To accommodate this functionality I wrote the following to build my Solr query:

# build the search options
my %search_options = ();
$search_options{ 'start' }          = $start;
$search_options{ 'rows' }           = ROWS;
$search_options{ 'facet' }          = 'true';
$search_options{ 'facet.field' }    = [ 'type', 'facet_creator', 'facet_subject', 'repository' ];
$search_options{ 'facet.mincount' } = MINCOUNT;
$search_options{ 'sort' }           = $sort;
if ( $cgi->param( 'fq' ) ) { $search_options{ 'fq' } = [ $cgi->param( 'fq') ] };

# do the work; search
my $response = $solr->search( $query, \%search_options );

Notice how $search_options{ ‘fq’ } points to a reference to an array. And notice later how the search method includes a reference to %search_options.

Maybe the following will work for you:

# build an array of facets
my @fqs = ( 'systype:LA', 'location:US' );

# build the search options
my %search_options         = ();
$search_options{ 'start' } = $start_record,
$search_options{ 'rows' }  = $maximum_records;
$search_options{ 'fq' }    = [ @fqs ];

# do the work; search
my $response = $solr->search( $query, \%search_options );

Good luck.

Upon closer examination of the definition of Cosine Similarity he realized that my compare subroutine included too many cosine functions. After editing the subroutine and duplicating a document in the corpus, a correct value of 1000 is returned for exactly similar documents. (Actually, it sometimes returns scores of 999 which I’m going to chalk up to rounding errors.)

“Thank you, Allen.”

Now a new problem presents itself. Specifically, the similarity scores for all the other documents are upside down:

  Comparison: scores closer to 1000 approach similarity

      d1    d2   d3   d4   d5   d6

  d1   -   396  459  538  541  320
  d2   -    -   478  247  334  240
  d3   -    -    -   312  304  265
  d4   -    -    -    -   694  438
  d5   -    -    -    -    -   367
  d6   -    -    -    -    -    - 

  d1 = aristotle.txt
  d2 = hegel.txt
  d3 = kant.txt
  d4 = librarianship.txt
  d5 = mississippi.txt
  d6 = plato.txt

Previously, hegel.txt (d2) and plato.txt (d6) where considered very similar, but now they are almost opposites. Something is still not correct, and I sincerely have no idea where to begin looking for a solution.

I have updated the downloadable scripts, but as far as the compare subroutine goes, they are still not perfect (broken).

Here are the changes made:
# require/include
changed use “./lib”;

use MARC::File::SAX; # I manually copied the sax.pm to the proper folder.

I also needed to cpan>install MARC::Charset

Thanks!
Dean

On my previous post, I forgot the put multiple fq. Basic I want to do multiple fq on webservice::solr. Is that possible? If so can you give me some advise.

my $response = $solr->search( $query, {
’start’ => $start_record,
‘rows’ => $maximum_records ,
‘fq’ => ’systype:LA’,
‘fq’ => ‘location:US’
});

This code failed.

I have problem with solr fq (facet query). If I use q=system&fq=systype:LA on SOLR admin page, it works.

I tried both of these methods using Webservice and it failed . Can you give some inputs to solve this problem.

Thanks

my $response = $solr->search( $query, {
‘start’ => $start_record,
‘rows’ => $maximum_records ,
‘fq’ => ‘systype:LA’
});

my $response = $solr->search( $query, {
‘start’ => $start_record,
‘rows’ => $maximum_records ,
‘facet.query’ => ‘systype’,
‘systype’ => LA
});

I can’t figure out this error. Can you help? I didn’t get any error on install of WebService::Solr.

Thanks

./trivial-search.pl Hello
Apr 6, 2010 11:09:40 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={wt=json&q=Hello} hits=2 status=0 QTime=1
Could not parse JSON response: malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before “\n\n<met…") at /usr/lib/perl5/site_perl/5.8.8/WebService/Solr/Response.pm line 42.

Error 400

HTTP ERROR: 400undefined field name
RequestURI=/solr/selectPowered by Jetty://

I can’t say I’m totally on board with the iPad personally (as you’ve probably seen (repeatedly) on Facebook, I’m getting the Notion Ink Adam this year pretty much no matter what: http://www.slashgear.com/notion-ink-adam-hands-on-0969281/).

But I am really interested in what the iPad is going to do for the popular perception of digital reading. I hope it’ll at least bust up the “I can’t read on a screen!” statement a little.

I remember your HyperCard stack. I suspect it was the first user-created stack I’d ever encountered. Thanks for the flashback to an earlier century :-)

You can keep mowing the grass. I’m allergic.

Yes, an additional next step would be to map the automatically generated subject “tags” to Library of Congress Subject Headings or some other controlled vocabulary. This has been done a number of times previously with various degrees of success, but compared to the other problems at hand — such as making the documents themselves more useful and interrelated — I plan to forgo this option at the present time. I understand the value of a controlled vocabulary, and that is why I have retained it whenever possible.

You might want to take a look at the open source Kea software, which does something similar (It uses tf*idf and some other heuristics).

http://www.nzdl.org/Kea/

Tom

Interesting. When you say “Third, automatically generated keywords and phrases were many times just as useful as the librarian-assigned Library of Congress Subject headings.”

I grow concerned though, because the strings:

Universalism United States History
Unitarian Universalist churches United States

…seem to me to be much more fulsome and useful. I’m not saying your automatically generated list isn’t useful, or that it isn’t complementary to LCSH – I’m just concerned with your phrase “just as useful” which, pardon me – seems to give you away as someone who, when weighing the options, will be more than willing to let LCSH go when the time comes.

And that, I think, is not going to serve us well.

~Nathan

Many of the items harvested from the Internet Archive were complete with MARC records. Some of those records included subject headings. During Step #5 (above), I spent time observing the output and comparing it to previously assigned terms. Take for example a work called Universalism in America: A History by Richard Eddy. Its assigned headings included:

Universalism United States History
Unitarian Universalist churches United States

I read with interest your article. Here are few points worth to mention:

1. IDF is defined as log(D/d) where D is number of documents in a collection and d is the number of documents mentioning a given term, regardless if the documents are relevant to said term. The base of the log does not matter (it can be base 10, 2, etc). The reason for taking logs is because most scoring functions in IR are assumed to be additive and because terms are assumed independent form one another (even when often this is not exactly the case).

2. IDF is a measure of the discriminatory power of a term (term specificity), but it does not relevancy. Indeed, IDF is a term weight score in the absence of relevance information.

3. IDF is a small pixel in the bigger picture of Robertson-Sparck Jones Probabilistic Model (RSJ-PM). A tutorial on the RSJ-PM Model explaining this model is available at http://www.miislita.com/.

4. With unstructured, unfocused, and generic collections at the scale of the Web (e.g. commercial search engines like Google), the stability of IDF and this as a reliable scoring function has been put into question by several authors.

Regards

Dr. Edel Garcia
http://www.miislita.com/

Umlaut does not have an index. It assumes users are finding texts and articles in some other software somewhere else. But once they do find a known item, Umlaut — via either a link or content embedded on the other software’s page via javascript — provides services for that item. Fulltext; location in library stacks; inter-library loan; search inside the book from Google, Amazon, HathiTrust; citations and similar items from ISI or Scopus; related information from Wikipedia, Ulrich’s, Amazon, Journal Citation Reports, Worldcat Identities.

This is just a start. I can think of so many more services that could be provided — some which actually exist, and Umlaut needs to have plug-ins written for, others which don’t exist yet and someone needs to create. But Umlaut is a platform for supporting plug-ins for arbitrary services for known items.

One project I’d like to get to in the future is working with LibX 2.0 to allow more pages on the web, without their actual cooperation, to link to Umlaut for library-provided services for items users find wherever.

The more I work with it and think about it, the more I think Umlaut is an important infrastructural building block, for exactly what you’re talking about.

But I’m having trouble getting anyone else to see this!

And if Code4Lib can’t say what the best open source software of the year is in the library community, then who can? Besides, maybe the award winner can do one of the keynotes on one of the days.

Mostly, I’m writing to connect your post up with the schedule (presentations and eventually videos): http://code4lib.org/conference/2009/schedule

It was nice to meet you in person, and to chat about the Journal! Thrilled that the FRBR presentation clicked!

It might be a good idea, but maybe not with the Code4Lib name. But I worry in general we don’t collectively know enough about what makes good software to give a Software of the Year honor reliably.

There has always been a bit of an elephant in the room with the issue of identifying schemas, from the “XML” and element set name wranglings of Z3950 to the advanced SRU “It’s just a URI (Lets hope naming the problem happens to solve it too)”

If the answer to these problems is “Just give everything a mime type” then thats cool (We certainly need some registry) but what should the mime type for mods actually look like?

I’ve been quite impressed with it. I think the idea of merging SRU into OpenSearch is a good one.

It’s not really true to say that “OpenSearch returns an RSS-like data stream” — rather an OpenSearch description can specify exactly what format is returned using a MIME type, and can even specify different URLs for retrieving different formats. So I think it’s rather compatible with SRU here. There’s no reason an OpenSearch description can’t declare results returned in MARC-XML or anything else. (I hope MARCXML, MODS, etc, have declared mime types? If not they need to asap for a billion reasons!)

Also, while an OpenSearch description document doesn’t declare any particular syntax for it’s query, there’s no reason an OpenSearch query _couldn’t_ be in CQL.

It would seem useful to me to extend OpenSearch to allow a description document to specify that CQL is supported, and specify what search indexes the server provides, or any other CQL-related metadata.

One of the most useful parts of OpenSearch is how it allows easy extensibility using custom namespaces. I am actually not that familiar with SRU, and haven’t read that OASIS document yet, but I hope they take the approach of trying to fit SRU into the existing OpenSearch standard, rather than creating a new ‘umbrella’ standard on top of it. I think the OpenSearch standard would be quite suited to this.

I perused the OpenSearch listserv recently, and I didn’t see any mention of OASIS or SRU. I hope the OASIS folks are actually talking to the OpenSearch folks about this, rather than just doing things in their own silo. The OpenSearch folks seem to me to be quite on top of things, and interested in making sure OpenSearch supports new use cases in the ‘right’ way.

Alas, no and yes regarding autocommit. No, my solrconfig.xml has not turned on autocommit. Yes, despite what the WebService::Solr pod says about the commit method, the underlying WebService::Solr code turns on autocommit by default. For better or for worse, WebService::Solr gives you autocommit for free.

I believe the best course of action here is to update the WebService::Solr pod to reflect what the code actually does.

I just noticed your 1st of a series of articles on WebService::Solr. Looks great so far! :)

Here are some comments about the code:

In the first script, you could probably use the much simpler syntax for add() rather than then full-on OOP way with Document and Field objects:

my $id  = WebService::Solr::Field->new( id  => $index );
my $title = WebService::Solr::Field->new( title => $_ );
my $doc = WebService::Solr::Document->new;
$doc->add_fields(( $id, $title ));
$solr->add( $doc );
$solr->commit;

could be as simple as:

$solr->add( { id => $index, title => $_ } );

The full object notation is available when you need it, but for simple docs, I wouldn’t bother with it.

Also, note that I’ve left off the call to commit(). WebService::Solr has an “autocommit” bit that is on by default, so, after operations that require a commit, it’s already done for you.

I should probably also mention that it would be more efficient to send multiple documents over the line rather than one at a time — though this case wouldn’t show much difference.

In the searching script you write:

print "Your search ($query) found " . ( $#hits + 1 ) . " document(s).\n\n";

While this is true for this particular index, in reality

my @hits = $response->docs;

only returns the documents attached to this particular response — which could be, for example, 10 out of 100 actual results.

The true number of matches can be found with the pager object:

$response->pager->total_entries

Hopefully I’m not giving you a bunch of details that you’re already aware of — If i am, I apologize.

Anyway, I can’t wait for the rest of the series! :)

“Thanks, Brian!

For some materials, there are probably alternative ways of doing this (say, getting a dataset from someone who already has work-based records and just matching them to manifestation records), but not probably for the more obscure stuff. We either try to get it out of the MARC records automatically or we do it manually or we stick with what we have now (which doesn’t work so well).

Thank you very much for trying out Google Research Datasets, providing interesting datasets, and giving us extremely useful feedback. We have learned a lot about the issues facing researchers and dataset producers from this testing period.

As you know, Google is a company that promotes experimentation with innovative new products and services. At the same time, we have to carefully balance that with ensuring that our resources are used in the most effective possible way to bring maximum value to our users.

It has been a difficult decision, but we have decided not to continue work on Google Research Datasets, but to instead focus our efforts on other activities such as Google Scholar, our Research Programs, and publishing papers about research here at Google.

The Google Research Datasets service will remain active until the end of January 2009 during which time any datasets may be downloaded. For those datasets that are impractical to download, we will also happily provide interested users with a copy via hard drive shipment.

Once again, we’d like to thank you for helping us test Google Research Datasets, it’s been a very useful experience, and we look forward to finding new ways to provide you with useful services in the future.

The Google Research Datasets team

Copyright 2008. Google Inc, 1600 Ampitheatre Parkway, Mountain View, CA 94043.

Thanks!

WorldCat Hackathon photos are out on Flickr, too. I see Ray has uploaded his already. I’ll bet Afelonne will post hers up shortly.

First, “something like Jangle” means. Implement Jangle but more so implement as many standards-driven things as possible. For example, while it was not discussed, I would imagine that if an OpenSearch and/or SRU interface were suggested people would have said, “‘Sounds like a good idea.”

Regarding authority records, yes, the largest problem was finding authority records in the wild.

Yes, I believe the idea of importing holdings information from an ERM was mentioned.

Last, yes, “working on the easier things first” would be a better way of prioritizing the items that were outlined during the meeting.

Thanks for posting this. I wish I could have been there, but the OLE Project meetings had to take priority. Keep us posted on your incorporation of VUFind into the Catholic Research Resources Alliance.

Tim

I think there’s still a need for the discovery tool to tell people, in a list, the full extent of what issues of Time Magazine the library has, in print, or in electronic.

Maybe it would do this working _with_ the link resolver, as it’s already assumed the discovery tool has to work with the ILS, naturally. But, in an academic library, or at least in my academic library, this is clearly a need.

“There was a lot of discussion whether or not this plug-in should be extended to include other data types, such as the ones outlined above, or to distribute Solrmarc as-is, more akin to a GNU “do one thing and one thing well” type of tool.”

Jeez, this seems obvious to me. No, Solrmarc shoudln’t do things that aren’t MARC. Solrmarc is a plug-in for SOLR to index MARC. You want to write a plug-in for SOLR to index other things, you can. Why oh why would you want a plug-in for SOLR to index half a dozen things all wrapped into one plug-in? That doesn’t make any sense at all. Ross suggests the sense of “a pluggable framework of different sorts of metadata parsers”–well, SOLR already IS this, and SolrMARC is a plug-in written for SOLR’s ‘pluggable framework’, to do MARC!

I don’t want to be mean, but that this was a lot of discussion doesn’t give me confidence in the software engineering experience represented in the room–making software engineering decisions about VuFind?

“In general it was agreed that this holdings information ought to be indexed to enable searches such as “Time Magazine 2004″, but displaying the results was seen as problematic.”

Well, the problem here is that most of our catalogs don’t actually contain sufficient semantic metadata to answer this question, regardless of what a discovery layer does. That means that until that problem is fixed, you won’t be able to have your discovery layer answer that question. I still think it’s important to have your discovery layer _list_ what issues of Time Magazine you hold, even if it can’t actually operate on the listing semantically.

“Why not use your link resolver to address this problem?” was asked.”

Oh boy, I think this was asked by someone with no experience with link resolvers. 1) Because, again, this data, for print, doesn’t exist anywhere that the link resolver can use to get the semantic info to answer the question. 2) If it did exist in your ILS, the link resolver would have to get it from the ILS. I wouldn’t hold my breath for most of our commercial link resolver vendors to provide this functionality, VuFind could provide it a lot quicker. 3) If you’re talking electronic, then, yes, our link resolvers can generally do it.

I think they missed the boat on this one. In general, from your review of the discussion, I see people rationalizing hard-but-important problems as “well, gee, that’s not really neccesary after all.” It’s one thing to say “It’s hard, let’s work on easier stuff first.” But don’t convince yourself it doesn’t really matter just because it’s hard.

1) “[I]mplementing something like Jangle was endorsed.” Out of curiosity, why “something like” Jangle instead of Jangle itself? Jangle is still a clay that can be molded to meet the desires of what people need, not hardened stone. Any and all suggestions are welcome to make it do what developers need.

2) I think I lean towards keeping SolrMARC, well, SolrMARC, although I can see the argument for a pluggable framework of different sorts of metadata parsers, as well. I still think most of our other formats don’t have the immediate technical and non-technical “problems” that MARC carries with it.

3) There are a few problems that I see with authority records. The first is the lack of name authorities in the wild (the subject authorities are all I know of that are available). The second is the fundamental problem of matching the authority to records, since it’s just string matching.

4) I’m still not sure why, even in an increasingly electronic environment, you shouldn’t be able search for “Time Magazine 2004”. Couldn’t the electronic holdings be imported from the link resolver knowledgebase or ERMS?

5) One of the plans I had when I still worked at Georgia Tech was to create a consortium-wide “cache” for the federated search project (the major universities in Georgia consortially use Metalib), using something like a Solr, or even Sphinx store to keep recent results as a place that the federated search searches “first” while federating through the licensed targets in the background. With around 80,000 FTE (GT, UGA, Georgia State, and Emory) contributing to the cache, I think you’d have more than enough search results in there to make it work. The biggest hurdle would be working out who has access to what, but I still think that’s pretty doable (since they’d all be using the same search engine in the first place).

Based on my experience, there are probably two ways to incorporate MyLibrary (reading list) content into a system like Blackboard. The first and probably most straight-forwarward way to is create pages using MyLibrary, have them be associated with URLs, and put those URLs into Blackboard. This is a more brute force method but also probably the quickest.

The second and more elegate solution but also more complicated is to use some sort of application programmer interface (API). For better or worse MyLibrary has a Perl API not a Web Services based API. I believe Blackboard has some sort of Java or C API and not some sort of Web Services API. Give this state of affairs I suppose you could:

• Put content into MyLibray
• Use the MyLibrary Perl API to save content to a file system
• Use the Blackboard Java API to read the content
• Display

Not straight-forward but scalable and would make it easy to keep content up-to-date. A better solution would be communication via Wewb Services on both sides. Alas.

Signed, Mr. Bar.

Thanks for your presentation at Access 2008! I’ve been wanting to incorporate something similar into BlackBoard for a while now and this may be the piece that’s been missing… do or anyone else know about anyone who’s attempted to integrated a mylibrary reading list into the proprietary LMS?

Cheers,\Jane

http://mndoci.com/blog/2007/08/09/scifoo-google-and-large-scientific-datasets/

I just finished harvesting content again, and I downloaded about 115,000 records. When I indexed the data, the indexer thought some of the data was invalid around the 113,000 record mark. (No puns intended.) Maybe your import routine accepted these extra records, and maybe the record mentioned above fell into that category. Strange.

http://skyline.cudenver.edu/record=b2221498

I realize your database is an experiment, but I was having trouble finding things with it. It appears to index titles on the article.

Thanks.

If I understand your question correction, then I know for certain MARCXML is not designed to contain full-text mark-up. I’m pretty sure MODS is the same way. Both are intended to contain bibliographic metadata. On the other hand, it would be entirely possible to included a link (think “call number”) in either a MARCXML or MODS file pointing to the full-text of a journal article.

As for the mark-up of the journal article itself, I would advocate the use of TEI. Many might think this is overkill since TEI leans towards the very analytic and scholarly, but in reality, TEI is well-suited to general mark-up of text — prose or poetry.

HTH. –ELM

However, I’m wondering if there’s any discussion of a “full-text” tag for journal articles, etc. I don’t mean articles that an author has posted, but something more general, like the PLoS or preprint servers. In my case, there’s no URI because the text doesn’t exist as separate from its metadata, the MODS record *is* the online version. Any guidance for me?

Yes, the word “properties” would have been better than “elements” to describe Dublin Core items. Yes, exactly, RDF, just like METS, can be used to mix content from various vocabularies together into a single XML file. That is what I meant. Regarding SKOS, again, your distinction is more precise than my description.

The devil is in the details.

1) “Dublin Core does not define how data should be encoded. It is simply a list of elements.”

So you’re using a concept from XML (“element”) to describe a format that is independent of XML, and so in fact distorting its purpose in the process.

I’d say the core DC terms (title, etc.) are data properties or attributes. But DC is of course much more than just those core terms thee days.

2) “RDF is similar to METS”

“Similar” how?? I see very little similarity, except at the most superficial level that both allow mixing of different metadata structures. RDF, however, is a data model; METS is not.

3) “SKOS is an XML format for thesauri”

SKOS is an RDF vocabulary that can (like any RDF) be serialized as XML. But it is NOT fundamentally an “XML format.”

I don’t mean to pick on you, but if you’re making sweeping suggestions like this, you need to be more careful about the details.

I’ve read many tributes to my father since his death, but this was one of the most moving.