|
Thanks Matt.
"* When calling solr:marc:index without a MARC_FILE argument, it'd be nice to get a message saying that the MARC_FILE arg is missing above the config dump. " The config dump does list "MARC_FILE: [marc file needed]" already. I was worrying that screen was getting too long already, but if you really think it needs an extra line at the very top saying the same thing, that can be done. You do? "* When attempting to get this going in the root of the plugin (for development) I created a new config/SolrMarc/development.properties file. To run this, I executed: " Do what works, but I think there's a confusion over multiple jetty instances being there. I was targetting the jetty instance that is optionally installed with the template installer, for your own data -- NOT the jetty instance (currently) included for testing purposes. So finding the "jetty at the root of the plugin" was not the intended goal, I would consider that a bug, heh. I think you must have been finding two different instances of jetty/solr with ../ vs ../..? It may be hard to get the default config.properties to work for all cases, in some cases you'll just have to write your own -- all you've got to do is put a config/SolrMarc in your own Rails app including the BL plugin, and the rake task, when executed from your app (NOT from inside the plugin) will find that. But here is the idea. The relative paths in config.properties basically end up being relative to config.properties itself. So the scenario I was coding the default relative paths for was: 1) You install the BL plugin via template installer. 2) You tell the template installer that, yes, you do want to install a jetty. 3) You then execute rake solr:marc:index from your app (NOT cd'ing into the plugin; that confuses rake, since rake will recognize the plugin AS a Rails app. 4) Outcome => It finds the jetty/solr installed by the template installer, at the relative paths specified. As far as I could tell, I had the right relative paths for that. i tried to do that scenario as best i could, although it wasn't exact since I had to point at the existing 'template' using the existing trunk Blacklight. But can you try out that scenario and see if it works as intended? But I think you'll get more ideas of possibilities for doing what you really want from the next point... "I'd be interesting in knowing if you can override the solrmarc.solr.war.path and solr.path when executing the SolrMarc jar file? Along with that, the solr.hosturl and solr.indexer.properties?" So solrmarc itself right now won't really allow an "over-ride", no. I tested that. If the paths were NOT in the config.properties, then they could be passed on the command line -- the rake task can't do that now, but it hypothetically could be written to do so. However, if the paths ARE in the config.properties, then SolrMarc doesn't allow a command line 'override'. The anticipated use case is that rather than pass them on the command line to the rake task, you'll just make a different config.properties file and pass THAT on the command line, as you've done. However, for the basic expected use cases, you don't even need to pass CONFIG_FILE on the command line. In your example, you had a "development.properties" config file. If you instead name this "config-development.properties", then the rake task will automatically find and use it if you execute: rake solr:marc:index RAILS_ENV=development So perhaps we should bundle and included "config-test.properties" that has the relative paths set to the embedded test data jetty/solr instead of the template installer optionally installed jetty/solr? Then you could just execute "rake solr:marc:index RAILS_ENV=test" to get the proper paths for the embedded test data jetty. Hope this helps clear things up. Let me know what you think appropriate next steps are. Actually, since I think "development" is the default Rails env, if you had a config-development.rb, then simply running "rake solr:marc:index" would use that in preference over a generic "config.properties". If you instead manually specify RAILS_ENV=production, then the rake task would use config.properties -- so long as a config-production.properties didn't exist.
You could theoretically use this with any arbitrary 'environment' name. rake solr:marc:index RAILS_ENV=my_made_up_env will indeed find config-my_made_up_env.properties
Matt Mitchell made changes - 09/Feb/10 02:47 PM
Thanks Jonathan,
For the MARC_FILE, or any other required param, I'd prefer an explicit error report than a config dump/view. We can always get the config dump in the "info" task. The paths/properties files to me are confusing, not because of your patch though. I did find out that overriding SolrMarc params from the command line does indeed work: http://groups.google.com/group/solrmarc-tech/browse_thread/thread/12005daeb1c50043 In my opinion, since we're using Rake to execute this stuff, we should completely override the path based properties that are in the config files using the command line opts. We could keep the path based properties in the config files, but add comments, "These paths are not used when running the solr:marc:* tasks" for example. This would also allow developers to test within the plugin directory and not worry about relative path stuff. And the same code would work when used in an application... because we have a dynamic relative base: Rails.root I'd do something similar for the SM solr.hosturl, fetching the value from Blacklight.solr_config[:url]. Of course this doesn't prevent people from using the properties files, it's only that our solr:marc:* Rake tasks would override in a useful way. I like the SOLRMARC_MEM_ARGS, but how about a JAVA_OPTS variable instead so you could append arbitrary java opts: rake solr:marc:index MARC_FILE=data.marc JAVA_OPTS=xxx -- just a thought? It's be nice also, to specify a MARC_FILE with a glob/* so you could index an entire directory of marc files Definitely like the RAILS_ENV stuff you have in there. Makes a lot of sense to switch based on that value. Matt, thanks for taking a look at this.
I think most of the things you're asking for are changes from what is currently in trunk for SolrMarc 2.0. Is it possible to apply this patch now -- as it gets up to SolrMarc 2.1 and takes advantage of some new features in 2.1 to make things more convenient -- and file your other ideas as 'enhancements' that can be added subsequently? The current behavior of spitting out the usage screen if you omit the MARC_FILE is actually there because it's what Bess requested when I wrote the previous version of original 2.0 task. It's easy enough to change though. What do you want it to look like? I don't really like having the rake task _require_ that certain properties NOT be in the config.properties file. I don't like changing SolrMarc default behavior like this, I think it will lead to confusion if a properties file that works with 'raw' SolrMarc does NOT work or works differently with the rake task. However, we could add a JAVA_OPTS env argument to allow arbitrary solrmarc args _if_ you choose to include them on execution. It would end up looking like... JAVA_OPTS="-Dsome.property=value -Dother.property=value". In general, I like including actual individual env args for particular arguments we actually have a known use case for, I think it's less confusing; but the JAVA_OPTS one could be there 'just in case'. But I do see what you mean about fetching the value from Blacklight.solr_config -- although that would require the rake task to load the Rails environment, whcih it doesn't do now, which adds 2-3 seconds to startup time for the rake task on my machine. Perhaps it needs more consideration -- can we go ahead and apply the patch, and then consider that enhancement subsequently? I _believe_ that you can supply a directory as the (possibly mis-named) MARC_FILE argument already, and it will Just Work to index all files in that directory, because SolrMarc itself is already prepared to recognize a directory in that slot. If you test this and it works, perhaps we should change the name of MARC_FILE or something. (Just "MARC_INPUT" ?). But since this wasn't there in the previous trunk version of the rake task either, again, I'd like it if we could go ahead and apply this patch and consider this subsequently. Hey Jonathan,
We can add enhancements later sure, this is a good base patch. BTW, I wasn't suggesting that the Rake task require that the SolrMarc properties file be setup any different. It would just override the path/url based variables. So it's not changing SolrMarc behavior. Rake is just introducing some dynamically generated paths/urls for convenience, to send to SolrMarc. Also, the solr.yml file wouldn't need to load the entire Rails environment. It could be parsed using the yaml lib, if start-up speed was a problem. Glad to know the sub-dir will work for MARC_FILE... should have tested this before I mentioned it :) Maybe the arg should just be named "MARC"? So I'll apply the patch and go from there. I have some ideas for getting multi-core working using some of this too. Thanks again for your patch! Great, thanks. If you outline the changes in another ticket (or tickets), I can try to get to writing the code; of course you (or anyone else!) is also welcome to.
I'm just back from a week away from work on "Snow Holiday" (Ie, crazy snow here everything was closed), so kind of overwhelmed catching up at the moment, sorry! I'm also not sure how long I'm going to stick with SolrMarc in general, I'm considering other indexing options, so if I end up not using SolrMarc at all here, my motivation to improve the rake task for SolrMarc will be somewhat diminished. :) Can you add a note on this ticket when you've applied the patch to trunk?
Naomi Dushay made changes - 05/Apr/10 12:14 PM
Bess Sadler made changes - 05/Apr/10 12:23 PM
Jonathan Rochkind made changes - 04/May/10 04:19 PM
Okay, this never got applied, but revisiting it now.
1. Added load of solr.yml file to use solr url for appropriate environment, put in -Dsolr.hosturl, over-riding config.properties. 2. Fixed paths in default config.properties to point to where the current template installer will install the optional jetty. config-test.properties points to the submodule testing jetty. The paths in config.properties are a little bit weird, because SolrMarc requires them to be relative to SolrMarc.jar, which we have in blacklight/solr_marc/SolrMarc.jar. 3. Updated included SolrMarc.jar to SolrMar tagged 2.1.1 4. Fixed solr:marc:index_test_data script to be written in terms of this solr:marc:index -- it simply sets the MARC_FILE arg and calls solr:marc:index. DRY, and more flexible -- now you can call index_test_data with whatever RAILS_ENV you want (test, development, production). 5. Fixed template.rb installer to NOT modify the config.properties solr.path before copying to local app. Is no longer neccesary and in fact messes things up. Also fixed template.rb to NOT make a copy of the distro sample data from plugin to local app, the one copy we already have in plugin is enough and rake task is perfectly happy eating it. My branch got all weird and out of sync with master, the change from direct files jetty/data to submodule confused it, and I couldn't figure out how to get it back in sync without causing weirdness in master.
So I am just committing these files straight, as one bulk commit. Fixed up specs to pass, and test as best as I could that solr:marc:index produced correct java command.
Jonathan Rochkind made changes - 13/May/10 03:03 PM
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Closed
Major
SolrMarc 2.1 update
* When calling solr:marc:index without a MARC_FILE argument, it'd be nice to get a message saying that the MARC_FILE arg is missing above the config dump.
* When attempting to get this going in the root of the plugin (for development) I created a new config/SolrMarc/development.properties file. To run this, I executed:
rake solr:marc:index MARC_FILE=../blacklight-data/test_data.utf8.mrc CONFIG_PATH=config/SolrMarc/development.properties
I got an error -- not able to find the Solr home and solr.war file (I expected that). I ended up getting the indexing to work when I set the paths like:
solrmarc.solr.war.path = ../../jetty/webapps/solr.war
solr.path = ../jetty/solr
Obviously this is a SolrMarc thing but, where is the reference root directory for SolrMarc?
Because you can see that the depth of those paths is not the same; "jetty" is at the root of the plugin, where I'm executing Rake.
* I'd be interesting in knowing if you can override the solrmarc.solr.war.path and solr.path when executing the SolrMarc jar file? Along with that, the solr.hosturl and solr.indexer.properties?