#Apache lucene indexing example update

Some of the little ones already pulled out all the lego bricks and spread them all over the room. Next morning in the Kindergarten all kids are back and keen to play some games. OK let’s go back to the Kindergarten and see how that worked until yesterday… To make that work, Lucene maintains all delete terms and their total order with respect to the corresponding add. This guarantees that updating a document either in memory or already flushed and/or committed to disk at the current point in time, see the corresponding delete.

#Apache lucene indexing example update

Internally, an update is an atomic operation that simply adds a document and deletes all previously added documents with the same key ( delete term). Let me quickly explain how updates work in Apache Lucene…. Lockless concurrent updates with DocumentsWriterPerThread Lucene like other index engines offers an update procedure where you have to specify some kind of identifier so that IndexWriter knows which documents need updating. The problem here is that you could end up with duplicate documents in the index or multiple bricks of the same size and color. Unfortunately this was one part of the game which worked well for the task of cleaning up all lego bricks or indexing a bunch of documents. For the interested reader I explained this in a more technical manner in my previous post on this blog. Instead of writing several in-memory segments and merge them together on flush Lucene now uses a DocumentsWriterPerThread that can write and flush its own private segment to persistent storage. Now this is how Lucene indexing works as of yesterday. Voila! – this procedure scales up very well especially if your co-worker helps you merging the small piles into bigger ones and your kids can leave on time!

While kids arrive dropping their piles on the ground you pick those piles and make bigger ones out of them while your kids go off collecting more lego. Instead of letting kids wait for you to sort, you could tell each kid to pile the bricks up in their hands and once they have enough bricks collected they can sort them themselves and put them into a yet smaller pile next to you on the ground.

What could you do to make it more efficient not just from an educational point of view? Eventually those’flushed’ segments are merged together in the background.īack to the Kindergarten… your algorithm doesn’t really scale up if you suddenly have twice as many kids to pick up bricks since most of them will wait for you sorting the bricks and putting them on piles.

This is roughly how Lucene indexing worked until yesterday: several threadscoming in carrying documents that are indexed in a in-memory segment and oncewe reach a certain threshold (maxBufferedDocs or ramMaxBufferSizeMB) we mergethose in memory segments together and write them to disk. You turn around, and theprocess repeats, until all the bricks are piled up behind you merge themtogether into one pile and put them in to the box where they belong- Closingtime! While you are sorting the bricks, kids are still arriving and they must waitfor you to turn back around to accept more bricks. Once your hands are full, you turn around, sort them by color and carefully put them in the right colored piles. They all run around, bringing bricks to you one by one concurrently. Suddenly it’s almost 4pm and the big mess needs to be cleaned up, so you ask each kid to pick up one lego brick and put it in your hands.

Imagine you are a Kindergarten teacher and a whole bunch of kids are playing with lego.