Impala query memory limit exceeded

Issue:

Query submitted at: 2021-06-09 11:34:33 (Coordinator: https://node01:25000)
Query progress can be monitored at: https://node01:25000/query_plan?query_id=a546f1f46aed278a:8a9dcd1a00000000
WARNINGS: Memory limit exceeded: Error occurred on backend node06.example.com:22000 by fragment a546f1f46aed278a:8a9dcd1a0000000a
Memory left in process limit: 28.25 GB
Memory left in query limit: -1.39 MB
Query(a546f1f46aed278a:8a9dcd1a00000000): memory limit exceeded. Limit=3.00 GB Reservation=2.21 GB ReservationLimit=2.40 GB OtherMemory=809.39 MB Total=3.00 GB Peak=3.00 GB
  Fragment a546f1f46aed278a:8a9dcd1a00000041: Reservation=2.21 GB OtherMemory=14.91 MB Total=2.23 GB Peak=2.23 GB
    Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
    HASH_JOIN_NODE (id=2): Reservation=2.21 GB OtherMemory=92.25 KB Total=2.21 GB Peak=2.21 GB
      Exprs: Total=38.12 KB Peak=38.12 KB
      Hash Join Builder (join_node_id=2): Total=38.12 KB Peak=38.12 KB
        Hash Join Builder (join_node_id=2) Exprs: Total=38.12 KB Peak=38.12 KB
    HDFS_SCAN_NODE (id=0): Total=4.00 KB Peak=4.00 KB
      Exprs: Total=4.00 KB Peak=4.00 KB
    EXCHANGE_NODE (id=3): Reservation=12.52 MB OtherMemory=2.26 MB Total=14.78 MB Peak=14.78 MB
      KrpcDeferredRpcs: Total=2.26 MB Peak=2.26 MB
    KrpcDataStreamSender (dst_id=4): Total=912.00 B Peak=912.00 B
    CodeGen: Total=15.28 KB Peak=1.84 MB
  Fragment a546f1f46aed278a:8a9dcd1a0000000a: Reservation=0 OtherMemory=794.49 MB Total=794.49 MB Peak=808.90 MB
    HDFS_SCAN_NODE (id=1): Total=786.14 MB Peak=800.75 MB
      Exprs: Total=4.00 KB Peak=4.00 KB
      Queued Batches: Total=736.12 MB Peak=744.33 MB
    KrpcDataStreamSender (dst_id=3): Total=35.34 KB Peak=35.34 KB
    CodeGen: Total=3.26 KB Peak=444.00 KB
WARNING: The following tables are missing relevant table and/or column statistics.
Memory limit exceeded: Error occurred on backend node06.example.com:22000 by fragment a546f1f46aed278a:8a9dcd1a0000000a
Memory left in process limit: 28.25 GB
Memory left in query limit: -1.39 MB
Query(a546f1f46aed278a:8a9dcd1a00000000): memory limit exceeded. Limit=3.00 GB Reservation=2.21 GB ReservationLimit=2.40 GB OtherMemory=809.39 MB Total=3.00 GB Peak=3.00 GB
  Fragment a546f1f46aed278a:8a9dcd1a00000041: Reservation=2.21 GB OtherMemory=14.91 MB Total=2.23 GB Peak=2.23 GB
    Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
    HASH_JOIN_NODE (id=2): Reservation=2.21 GB OtherMemory=92.25 KB Total=2.21 GB Peak=2.21 GB
      Exprs: Total=38.12 KB Peak=38.12 KB
      Hash Join Builder (join_node_id=2): Total=38.12 KB Peak=38.12 KB
        Hash Join Builder (join_node_id=2) Exprs: Total=38.12 KB Peak=38.12 KB
    HDFS_SCAN_NODE (id=0): Total=4.00 KB Peak=4.00 KB
      Exprs: Total=4.00 KB Peak=4.00 KB
    EXCHANGE_NODE (id=3): Reservation=12.52 MB OtherMemory=2.26 MB Total=14.78 MB Peak=14.78 MB
      KrpcDeferredRpcs: Total=2.26 MB Peak=2.26 MB
    KrpcDataStreamSender (dst_id=4): Total=912.00 B Peak=912.00 B
    CodeGen: Total=15.28 KB Peak=1.84 MB
  Fragment a546f1f46aed278a:8a9dcd1a0000000a: Reservation=0 OtherMemory=794.49 MB Total=794.49 MB Peak=808.90 MB
    HDFS_SCAN_NODE (id=1): Total=786.14 MB Peak=800.75 MB
      Exprs: Total=4.00 KB Peak=4.00 KB
      Queued Batches: Total=736.12 MB Peak=744.33 MB
    KrpcDataStreamSender (dst_id=3): Total=35.34 KB Peak=35.34 KB
    CodeGen: Total=3.26 KB Peak=444.00 KB (1 of 2 similar)

Caused:

Impalad memory is limited to 3GB. Adjust memory limit to fix this issue.

Resolution:

- in query:
Add: "set MEM_LIMIT=8g;" before the query.

- or in Impala pool in Dynamic Resource Pool Configuration:
Adjust MEM_LIMIT in particular pool you want.