NiFi instance disconnected

Issue:

2021-07-08 09:13:55,657 ERROR org.apache.nifi.controller.StandardFlowService: Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1026)
	at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
	at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1043)
	at org.apache.nifi.NiFi.init(NiFi.java:158)
	at org.apache.nifi.NiFi.init(NiFi.java:72)
	at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows:
Local Fingerprint:
			"type": ["null","string"]
		},
		{
			"name": "FILE_SIZE",
			"type": ["null","string"]
		}
	]
}bbb0a0dc-413c-36d1-8464-05c1dfc0d1e8NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATION0487043f-6db3-3707
Cluster Fingerprint:
			"type": ["null","string"]
		},
		{
			"name": "FILE_SIZE",
			"type": ["null","string"]
		}
	]
}9a1935a1-3e91-1a85-9189-b185f4419739NO_VALUENO_VALUENO_VERSION_CONTROL_INFORMATION03e4304b-2263-1e0c
	at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:315)
	at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1408)
	at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
	at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:1001)
	... 5 common frames omitted

Caused:

File "flow.xml.gz" is not sync with other nodes.

Resolution:

Copy file "flow.xml.gz" from other node then restart NiFi instance.

Impala query memory limit exceeded

Issue:

Query submitted at: 2021-06-09 11:34:33 (Coordinator: https://node01:25000)
Query progress can be monitored at: https://node01:25000/query_plan?query_id=a546f1f46aed278a:8a9dcd1a00000000
WARNINGS: Memory limit exceeded: Error occurred on backend node06.example.com:22000 by fragment a546f1f46aed278a:8a9dcd1a0000000a
Memory left in process limit: 28.25 GB
Memory left in query limit: -1.39 MB
Query(a546f1f46aed278a:8a9dcd1a00000000): memory limit exceeded. Limit=3.00 GB Reservation=2.21 GB ReservationLimit=2.40 GB OtherMemory=809.39 MB Total=3.00 GB Peak=3.00 GB
  Fragment a546f1f46aed278a:8a9dcd1a00000041: Reservation=2.21 GB OtherMemory=14.91 MB Total=2.23 GB Peak=2.23 GB
    Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
    HASH_JOIN_NODE (id=2): Reservation=2.21 GB OtherMemory=92.25 KB Total=2.21 GB Peak=2.21 GB
      Exprs: Total=38.12 KB Peak=38.12 KB
      Hash Join Builder (join_node_id=2): Total=38.12 KB Peak=38.12 KB
        Hash Join Builder (join_node_id=2) Exprs: Total=38.12 KB Peak=38.12 KB
    HDFS_SCAN_NODE (id=0): Total=4.00 KB Peak=4.00 KB
      Exprs: Total=4.00 KB Peak=4.00 KB
    EXCHANGE_NODE (id=3): Reservation=12.52 MB OtherMemory=2.26 MB Total=14.78 MB Peak=14.78 MB
      KrpcDeferredRpcs: Total=2.26 MB Peak=2.26 MB
    KrpcDataStreamSender (dst_id=4): Total=912.00 B Peak=912.00 B
    CodeGen: Total=15.28 KB Peak=1.84 MB
  Fragment a546f1f46aed278a:8a9dcd1a0000000a: Reservation=0 OtherMemory=794.49 MB Total=794.49 MB Peak=808.90 MB
    HDFS_SCAN_NODE (id=1): Total=786.14 MB Peak=800.75 MB
      Exprs: Total=4.00 KB Peak=4.00 KB
      Queued Batches: Total=736.12 MB Peak=744.33 MB
    KrpcDataStreamSender (dst_id=3): Total=35.34 KB Peak=35.34 KB
    CodeGen: Total=3.26 KB Peak=444.00 KB

Kafka Flink credential revoked

Issue:

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
  at org.apache.flink.streaming.connectors.kafka.internals.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:55)
  at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94)
  at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:574)
  at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
  at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
  at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:426)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
  at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Clients credentials have been revoked (18)
  at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160)
  at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
  at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
  at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:112)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:726)
  ... 15 more

FileCrush replacement format

Error message:
-bash: --replacement=$1-${crush.timestamp}-${crush.task.num}-${crush.file.num}: bad substitution

Fix by using escape character in bash:
--replacement=\$1-\$\{crush.timestamp\}-\$\{crush.task.num\}-\$\{crush.file.num\}

Parse hosts file in Fabric

Add the following code to your fabfile.py

The "hosts_file" support comment and host sequence:
$ cat hosts_file
# This is comment. using # asprefix
hadoopdev01
hadoopprod[01-10]
#hadoopdev02
192.168.1.[1-200]

User sqoop2 cannot submit applications to queue root.sqoop2

Error messages:

sqoop:000> start job --jid 2
2014-11-05 12:49:31 WIB: FAILURE_ON_SUBMIT
Exception: java.io.IOException: Failed to run job : User sqoop2 cannot submit applications to queue root.sqoop2
Stack trace: java.io.IOException: Failed to run job : User sqoop2 cannot submit applications to queue root.sqoop2
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:247)
        at org.apache.sqoop.framework.JobManager.submit(JobManager.java:418)
        at org.apache.sqoop.handler.SubmissionRequestHandler.submissionSubmit(SubmissionRequestHandler.java:152)
        at org.apache.sqoop.handler.SubmissionRequestHandler.handleActionEvent(SubmissionRequestHandler.java:122)
        at org.apache.sqoop.handler.SubmissionRequestHandler.handleEvent(SubmissionRequestHandler.java:75)
        at org.apache.sqoop.server.v1.SubmissionServlet.handlePostRequest(SubmissionServlet.java:44)
        at org.apache.sqoop.server.SqoopProtocolServlet.doPost(SqoopProtocolServlet.java:63)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)

Fix:

CM -> yarn configuration -> Fair Scheduler Allocations

Add sqoop2 user to allow it to submit job:

https://gist.github.com/kholis/7c4597806f278233ee7c

Add Service HIVE Error on CDH5 Wizard

Command failed to run because service Hive has invalid configuration. Review and correct its configuration. First error: 'Hive Metastore Database Host' is required when using database type 'postgresql'

Fix: Goto Hive Service Configuration. Search "Hive Metastore Database Host" fill it with postgres server hostname.