Monday, June 14, 2021

Impala query memory limit exceeded

Issue:

Query submitted at: 2021-06-09 11:34:33 (Coordinator: https://node01:25000)
Query progress can be monitored at: https://node01:25000/query_plan?query_id=a546f1f46aed278a:8a9dcd1a00000000
WARNINGS: Memory limit exceeded: Error occurred on backend node06.example.com:22000 by fragment a546f1f46aed278a:8a9dcd1a0000000a
Memory left in process limit: 28.25 GB
Memory left in query limit: -1.39 MB
Query(a546f1f46aed278a:8a9dcd1a00000000): memory limit exceeded. Limit=3.00 GB Reservation=2.21 GB ReservationLimit=2.40 GB OtherMemory=809.39 MB Total=3.00 GB Peak=3.00 GB
  Fragment a546f1f46aed278a:8a9dcd1a00000041: Reservation=2.21 GB OtherMemory=14.91 MB Total=2.23 GB Peak=2.23 GB
    Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
    HASH_JOIN_NODE (id=2): Reservation=2.21 GB OtherMemory=92.25 KB Total=2.21 GB Peak=2.21 GB
      Exprs: Total=38.12 KB Peak=38.12 KB
      Hash Join Builder (join_node_id=2): Total=38.12 KB Peak=38.12 KB
        Hash Join Builder (join_node_id=2) Exprs: Total=38.12 KB Peak=38.12 KB
    HDFS_SCAN_NODE (id=0): Total=4.00 KB Peak=4.00 KB
      Exprs: Total=4.00 KB Peak=4.00 KB
    EXCHANGE_NODE (id=3): Reservation=12.52 MB OtherMemory=2.26 MB Total=14.78 MB Peak=14.78 MB
      KrpcDeferredRpcs: Total=2.26 MB Peak=2.26 MB
    KrpcDataStreamSender (dst_id=4): Total=912.00 B Peak=912.00 B
    CodeGen: Total=15.28 KB Peak=1.84 MB
  Fragment a546f1f46aed278a:8a9dcd1a0000000a: Reservation=0 OtherMemory=794.49 MB Total=794.49 MB Peak=808.90 MB
    HDFS_SCAN_NODE (id=1): Total=786.14 MB Peak=800.75 MB
      Exprs: Total=4.00 KB Peak=4.00 KB
      Queued Batches: Total=736.12 MB Peak=744.33 MB
    KrpcDataStreamSender (dst_id=3): Total=35.34 KB Peak=35.34 KB
    CodeGen: Total=3.26 KB Peak=444.00 KB

Kafka Flink credential revoked

Issue:


org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
  at org.apache.flink.streaming.connectors.kafka.internals.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:55)
  at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94)
  at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:574)
  at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
  at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
  at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:426)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
  at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
  at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:571)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Clients credentials have been revoked (18)
  at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160)
  at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
  at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
  at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:112)
  at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:726)
  ... 15 more