27277

Java, Lucene : Set lock timeout for IndexWriter in Java.

Question:

I am working on integrating Lucene with our Spring-MVC based application. Currently we have it working, but rarely we get a cannot obtain lock error. After which I have to manually delete the lock file and then it works normally.

<strong>How can I set a timeout for Locking the index in Java?</strong> I don't have any XML configuration for Lucene. I added the project library in maven via POM.xml and instantiated the required classes.

Code :

public void saveIndexes(String text, String tagFileName, String filePath, long groupId, boolean type, int objectId) { try { // path is the indexing directory. File testDir; Path suggestionsPath; Directory suggestionsDir; Path phraseSuggestPath; Directory phraseSuggestDir; Directory directory = org.apache.lucene.store.FSDirectory.open(path); IndexWriterConfig config = new IndexWriterConfig(new SimpleAnalyzer()); IndexWriter indexWriter = new IndexWriter(directory, config); org.apache.lucene.document.Document doc = new org.apache.lucene.document.Document(); if (filePath != null) { File file = new File(filePath); // current directory doc.add(new TextField("path", file.getPath(), Field.Store.YES)); } doc.add(new StringField("id", String.valueOf(objectId), Field.Store.YES)); // doc.add(new TextField("id",String.valueOf(objectId),Field.Store.YES)); if (text == null) { if (filePath != null) { FileInputStream is = new FileInputStream(filePath); BufferedReader reader = new BufferedReader(new InputStreamReader(is)); StringBuilder stringBuffer = new StringBuilder(); String line; while ((line = reader.readLine()) != null) { stringBuffer.append(line).append("\n"); } stringBuffer.append("\n").append(tagFileName); reader.close(); doc.add(new TextField("contents", stringBuffer.toString(), Field.Store.YES)); } } else { FieldType fieldType = new FieldType(TextField.TYPE_STORED); fieldType.setTokenized(false); doc.add(new Field("contents", text+"\n"+tagFileName, fieldType)); } indexWriter.addDocument(doc); indexWriter.commit(); indexWriter.flush(); indexWriter.close(); directory.close(); StandardAnalyzer analyzer = new StandardAnalyzer(); AnalyzingInfixSuggester wordSuggester = new AnalyzingInfixSuggester(suggestionsDir, analyzer); ArrayList<String> words = new ArrayList<>(); if (text != null) { text = html2text(text); Pattern pt = Pattern.compile("[^\\w\\s]"); Matcher match = pt.matcher(text); while (match.find()) { String s = match.group(); text = text.replaceAll("\\" + s, ""); } if (text.contains(" ")) { Collections.addAll(words, text.split(" ")); } else { words.add(text); } SuggestionIterator suggestionIterator = new SuggestionIterator(words.iterator()); wordSuggester.build(suggestionIterator); wordSuggester.close(); suggestionsDir.close(); } AnalyzingInfixSuggester phraseSuggester = new AnalyzingInfixSuggester(phraseSuggestDir, analyzer); if (text != null) { text = html2text(text); ArrayList<String> phrases = new ArrayList<>(); phrases.add(text); SuggestionIterator suggestionIterator = new SuggestionIterator(phrases.iterator()); phraseSuggester.build(suggestionIterator); phraseSuggester.close(); phraseSuggestDir.close(); } } catch (Exception ignored) { } }

Thank you.

Answer1:

I quote two things from <a href="https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/index/IndexWriter.html" rel="nofollow">IndexWriter Documentation</a>,

<blockquote>

Opening an IndexWriter creates a lock file for the directory in use. Trying to open another IndexWriter on the same directory will lead to a LockObtainFailedException.

</blockquote>

and

<blockquote>

NOTE: IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexWriter instance as this may cause deadlock; use your own (non-Lucene) objects instead.

</blockquote>

So you can't open IndexWriter again if its already opened and not closed somewhere else. In your case, there happens to be some unlucky timing when two users are in same code block.

You can address this issue in two ways ,

1.<strong>Designate Critical Section:</strong> Mark code portion having writer opening , usage and close operation as critical section and apply Java synchronization on that critical section. Use some app Singleton bean to synchronize on. So when another user hits that block, he will wait till first user is done and lock is released.

2.<strong>Single Writer Instance</strong>:Develop a mechanism in your app to open and close writer only once for the life time of application and pass that single instance in service code so writer methods could get called by as many users as you wish since writer instance is made thread-safe by Lucene folks. I guess, this can be achieved by a Singleton Spring bean and by injecting that bean in your service.

Drawback in second approach is - multi server deployments for a single global index directory and if there are other applications trying to open writers on that global Index. This problem can be solved by wrapping your index writer instance creation code in some kind of global service that keeps returning the same instance to whichever application tries to use it.

This is not a simple issue that you are trying to solve by deleting lock files or by introducing time outs. You have to model your design as per IndexWriter documentation and not other way round.

Having single writer instance will introduce some performance improvements too.

Also, make a practice to do an empty commit just after creating the writer. This helped me in solving some issues in past.

Recommend

  • Squeryl custom field types
  • Setting date using DialogFragment and Android Annotations
  • Reading Content from File in J2ME
  • JSF PrimeFaces ajax request after logout and session invalidation
  • How to parse multiple, consecutive xml files in one document?
  • If my Servlet instance variable is of type StringBuffer, is it thread-safe? [duplicate]
  • Pinvoking adwapi.dll - cryptDecrypt and cryptEncrypt functions, weird problem
  • Finding if conditions in .java file
  • Java/ JDBC – Multi Parameter Search Best Practice
  • Setting JSON request header in Angular2 HTTP POST
  • android downloading multiple files with InputStream FileOutputStream
  • java.nio.file.FileSystemNotFoundException when getting file from resources folder
  • Help refreshing list with notifyDataSetChanged()
  • for loop iteration in django
  • Android: Alternative to onSensorChanged? no ACCELEROMETER data when phone is lying on its back on a
  • php: best way to validate POST
  • Client server multithread Socket
  • Is it possible to find the address of a non-primitive variable in Java?
  • Java out of memory errors on large WSDLs (play framework)
  • nested c:foreach in jstl
  • How to test method Jsoup.connect (static) using PowerMock?
  • C# Getting window's title by handle in Hebrew return question marks
  • Issue with static code in Android
  • Timezone display value with GMT offset from a given timezone?
  • java use Regular Expressions to generate a string [duplicate]
  • Why is JSON.NET adding all these backslashes
  • Put database into android project
  • How do I change content of ComboFieldEditor?
  • Retrieving value from sql ExecuteScalar()
  • Weird JavaScript statement, what does it mean?
  • Do I've to free mysql result after storing it?
  • How to delete a row from a dynamic generate table using jquery?
  • json Serialization in asp
  • SQL merge duplicate rows and join values that are different
  • KeystoneJS: Relationships in Admin UI not updating
  • Rails 2: use form_for to build a form covering multiple objects of the same class
  • Hits per day in Google Big Query
  • how does django model after text[] in postgresql [duplicate]
  • How to stop GridView from loading again when I press back button?
  • Checking variable from a different class in C#