NiFi Too many open files

Issue:

NiFi failed with following error:

2021-07-12 16:19:37,366 ERROR org.apache.nifi.provenance.index.lucene.EventIndexTask: Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1281)
	at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1257)
	at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
	at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
	at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /var/lib/nifi/provenance_repository/lucene-8-index-1625880478135/_p_Lucene50_0.tim: Too many open files
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010)
	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:575)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:514)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:470)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1284)
	... 9 common frames omitted

Caused:

NiFi process open files is over from ulimit

Resolution:

For NiFi Standalone

Add this to nifi-env.sh:

export MAX_FD=999999
export MAX_FD_LIMIT=999999

For NiFi in CDP

Change rlimit_fds value in CM > NiFi > Configuration