NiFi Too many open files

Issue:

NiFi failed with following error:

2021-07-12 16:19:37,366 ERROR org.apache.nifi.provenance.index.lucene.EventIndexTask: Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1281)
	at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1257)
	at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
	at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
	at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /var/lib/nifi/provenance_repository/lucene-8-index-1625880478135/_p_Lucene50_0.tim: Too many open files
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010)
	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:575)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:514)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:470)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1284)
	... 9 common frames omitted

Caused:

NiFi process open files is over from ulimit

Resolution:

For NiFi Standalone

Add this to nifi-env.sh:

export MAX_FD=999999
export MAX_FD_LIMIT=999999

For NiFi in CDP

Change rlimit_fds value in CM > NiFi > Configuration

NiFi instance disconnected

Issue:

2021-07-08 09:13:55,657 ERROR org.apache.nifi.controller.StandardFlowService: Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026)
	at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
	at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1043)
	at org.apache.nifi.NiFi.init(NiFi.java:158)
	at org.apache.nifi.NiFi.init(NiFi.java:72)
	at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows:
Local Fingerprint:
			"type": ["null","string"]
		},
		{
			"name": "FILE_SIZE",
			"type": ["null","string"]
		}
	]
}bbb0a0dc-413c-36d1-8464-05c1dfc0d1e8NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATION0487043f-6db3-3707
Cluster Fingerprint:
			"type": ["null","string"]
		},
		{
			"name": "FILE_SIZE",
			"type": ["null","string"]
		}
	]
}9a1935a1-3e91-1a85-9189-b185f4419739NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATION03e4304b-2263-1e0c
	at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:315)
	at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1408)
	at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
	at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1001)
	... 5 common frames omitted

Caused:

File "flow.xml.gz" is not sync with other nodes.

Resolution:

Copy file "flow.xml.gz" from other node then restart NiFi instance.