[#SOLRMARC-1] java.util.concurrent.RejectedExecutionException after indexing - Project Blacklight Jira

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.1 bug fixes
Fix Version/s: 2.1 bug fixes
Component/s: None

Description:

Hide
After indexing a file of MARC records, I sometimes (but not always) get this error:

Apr 12, 2010 10:12:49 AM org.apache.solr.common.SolrException log
SEVERE: java.util.concurrent.RejectedExecutionException
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1141)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
        at org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run(DirectUpdateHandler2.java:602)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

As far as I can tell, the indexing operation completed successfully in spite of this condition, and there are no other error messages in my import log.

Results of testing so far:

File 1: 800,000+ records; I have gotten the error both times I have indexed this file, but it takes hours, so I can't easily retest a large number of times.

File 2: 140,000+ records; I have gotten the error one out of two times when indexing this file. It's less time consuming to repeat this index, but results are already inconsistent, and I've only indexed it twice!

File 3: 1 record; I have indexed this quite a few times and never gotten the error.

I'm currently using the direct index writing method; I plan on retesting using the HTTP-based method to see if that makes any difference. I will post more details when I have had time to investigate further.

Show
After indexing a file of MARC records, I sometimes (but not always) get this error: Apr 12, 2010 10:12:49 AM org.apache.solr.common.SolrException log SEVERE: java.util.concurrent.RejectedExecutionException         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)         at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)         at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1141)         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)         at org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run(DirectUpdateHandler2.java:602)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         at java.util.concurrent.FutureTask.run(FutureTask.java:138)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)         at java.lang.Thread.run(Thread.java:619) As far as I can tell, the indexing operation completed successfully in spite of this condition, and there are no other error messages in my import log. Results of testing so far: File 1: 800,000+ records; I have gotten the error both times I have indexed this file, but it takes hours, so I can't easily retest a large number of times. File 2: 140,000+ records; I have gotten the error one out of two times when indexing this file. It's less time consuming to repeat this index, but results are already inconsistent, and I've only indexed it twice! File 3: 1 record; I have indexed this quite a few times and never gotten the error. I'm currently using the direct index writing method; I plan on retesting using the HTTP-based method to see if that makes any difference. I will post more details when I have had time to investigate further.

Environment:

VuFind integration under RedHat

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Demian Katz added a comment - 30/Apr/10 12:16 PM

I just tried switching to HTTP-based indexing in place of direct index writing. I was able to index all three of the sample files mentioned above without encountering the error. I then reverted to direct index writing, and reindexed files 1 and 2. File 2 indexed without error, but File 1 once again caused the exception -- except this time, the error repeated multiple times over a period of several seconds at the end of the index process; until this attempt, I had never seen the error more than once in a single indexing session.

It certainly seems that larger files are more likely to cause the error... and so far, I have only seen it when doing direct index writes (though I wouldn't completely rule out the possibility of errors in HTTP mode -- I would have to repeat my test a few more times to gain greater confidence).

Show

Demian Katz added a comment - 30/Apr/10 12:16 PM I just tried switching to HTTP-based indexing in place of direct index writing. I was able to index all three of the sample files mentioned above without encountering the error. I then reverted to direct index writing, and reindexed files 1 and 2. File 2 indexed without error, but File 1 once again caused the exception -- except this time, the error repeated multiple times over a period of several seconds at the end of the index process; until this attempt, I had never seen the error more than once in a single indexing session. It certainly seems that larger files are more likely to cause the error... and so far, I have only seen it when doing direct index writes (though I wouldn't completely rule out the possibility of errors in HTTP mode -- I would have to repeat my test a few more times to gain greater confidence).

Hide

Permalink

Demian Katz added a comment - 30/Apr/10 2:59 PM

I have repeated my HTTP mode test a second time, and once again, I did not encounter the error. I'm even more confident now that the problem is related to direct index writing.

Show

Demian Katz added a comment - 30/Apr/10 2:59 PM I have repeated my HTTP mode test a second time, and once again, I did not encounter the error. I'm even more confident now that the problem is related to direct index writing.

Hide

Permalink

Demian Katz added a comment - 01/Dec/10 1:38 PM - edited

I have begun seeing this same error when indexing authority records under Ubuntu, so this is not a strictly RedHat-based problem.

Courtesy of Anna Headley:

I think upgrading to solr 3.x will fix this issue. My experience mirrors Demian's, particularly in that I get the exception after larger imports (9240 and 21068 marc records).

I think this may be related to <https://issues.apache.org/jira/browse/SOLR-2008>.

Show

Demian Katz added a comment - 01/Dec/10 1:38 PM - edited I have begun seeing this same error when indexing authority records under Ubuntu, so this is not a strictly RedHat-based problem. Courtesy of Anna Headley: I think upgrading to solr 3.x will fix this issue. My experience mirrors Demian's, particularly in that I get the exception after larger imports (9240 and 21068 marc records). I think this may be related to <https://issues.apache.org/jira/browse/SOLR-2008>.

SolrMarc

java.util.concurrent.RejectedExecutionException after indexing

Details

Activity

People

Dates