Drill query fails with "AGGR OOM at First Phase" when doing Hash Aggregate

Symptom:

Drill query fails with "AGGR OOM at First Phase" when doing Hash Aggregate.
Sample error message or stacktrace is:

2018-01-01 11:11:11,111 [xxx:frag:5:6] INFO  o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes ran out of memory while executing the query. (AGGR OOM at First Phase. Partitions: 1. Estimated batch size: 57655296. values size: 65536. Output alloc size: 65536 Memory limit: 41943040 so far allocated: 262144. )
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

AGGR OOM at First Phase. Partitions: 1. Estimated batch size: 57655296. values size: 65536. Output alloc size: 65536 Memory limit: 41943040 so far allocated: 262144.

[Error Id: yyy ]
        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-mapr.jar:1.13.0-mapr]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: AGGR OOM at First Phase. Partitions: 1. Estimated batch size: 57655296. values size: 65536. Output alloc size: 65536 Memory limit: 41943040 so far allocated: 262144.
        at org.apache.drill.exec.test.generated.HashAggregatorGen5.spillIfNeeded(HashAggTemplate.java:1419) ~[na:na]
        at org.apache.drill.exec.test.generated.HashAggregatorGen5.doSpill(HashAggTemplate.java:1381) ~[na:na]
        at org.apache.drill.exec.test.generated.HashAggregatorGen5.checkGroupAndAggrValues(HashAggTemplate.java:1281) ~[na:na]
        at org.apache.drill.exec.test.generated.HashAggregatorGen5.doWork(HashAggTemplate.java:592) ~[na:na]
        at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:176) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_171]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_171]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633) ~[hadoop-common-2.7.0-mapr-1710.jar:na]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]

Hive: Different lock behaviors between DummyTxnManager and DbTxnManager

Goal

Understand the different lock behaviors between DummyTxnManager and DbTxnManager in Hive. The example query is INSERT into a partition table.

How to control the parallelism of Spark job

Goal:

This article explains how to control the parallelism of Spark job and uses a Spark on YARN job to demonstrate.

How to configure LDAP client by using SSSD for authentication on CentOS

Goal:

How to configure LDAP client by using SSSD(System Security Services Daemon) for authentication on CentOS.

How to install and configure MapR Hive ODBC driver on Linux

Goal:

How to install and configure MapR Hive ODBC driver on Linux.
This article gives detailed step-by-step instructions as a supplement to this MapR Documentation.

How to modify hbase thrift client code if Hbase Thrift Service enables framed transport and compact protocol

Goal:

How to modify hbase thrift client code if Hbase Thrift Service enables framed transportation and compact protocol.
The background is:
To avoid thrift service crash issue mentioned in HBASE-11052, we need to enable framed transport and compact protocol in hbase-site.xml and restart Hbase Thrift Service as below:

<property> 
  <name>hbase.regionserver.thrift.framed</name> 
  <value>true</value> 
</property> 
<property> 
  <name>hbase.regionserver.thrift.framed.max_frame_size_in_mb</name> 
  <value>2</value> 
</property> 
<property> 
  <name>hbase.regionserver.thrift.compact</name> 
  <value>true</value> 
</property>

After that, the old Hbase thrift client code need to be modified, otherwise it will fail with below error:

thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

This article explains what to modify in hbase thrift code to make the job compatible with framed transport and compact protocol.

How to install Thrift and run a sample Hbase thrift job towards Hbase Thrift Gateway on MapR Cluster

Goal:

How to install Thrift and run a sample Hbase thrift job towards Hbase Thrift Gateway on MapR Cluster.
The example job is written in Python, and it just scans a MapR-DB table.

How to modify max heap size for Impalad embedded JVM

Goal:

The impalad is a 'native' process that has an embedded JVM. The JVM is started from within c++ code.
 The -mem_limit startup option sets an overall limit for the impalad process (which handles multiple queries concurrently).
However the max heap size of impalad embedded JVM is much smaller than that limit.
Some impala queries may use up all embedded JVM heap size before reaching the limit set by "-mem_limit" startup option, so that it may cause impalad errors "OutOfMemoryError: Java heap space" or just get hung. In that situation, we need to increase the JVM max heap size.

This article shows how to check and modify the max heap size for impalad embedded JVM.

Hue could not show Hive tables after Hive enables PAM authentication

Symptom:

a. Hue could not show Hive tables after Hive enables PAM authentication, see below screenshot:

b. From /opt/mapr/hue/hue-<version>/logs/runcpserver.log, below error messages show up:

[19/Sep/2017 15:55:07 -0700] dbms         DEBUG    Query Server: {'server_name': 'beeswax', 'transport_mode': 'socket', 'server_host': 's4.poc.com', 'server_port': 10000, 'auth_password_used': False, 'http_url': 'http://s4.poc.com:10001/cliservice', 'auth_username': 'hue', 'principal': None}
[19/Sep/2017 15:55:10 -0700] thrift_util  INFO     Thrift saw a transport exception: Bad status: 3 (Error validating the login)

c. From HiveServer2 log /opt/mapr/hive/hive-<version>/logs/mapr/hive.log, below stacktrace shows up:

2017-09-19T15:57:11,046 ERROR [HiveServer2-Handler-Pool: Thread-60] transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: Error validating the login [Caused by javax.security.sasl.AuthenticationException: Error authenticating with the PAM service: login [Caused by javax.security.sasl.AuthenticationException: Error authenticating with the PAM service: login]]
 at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:110)
 at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
 at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
 at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.sasl.AuthenticationException: Error authenticating with the PAM service: login [Caused by javax.security.sasl.AuthenticationException: Error authenticating with the PAM service: login]
 at org.apache.hive.service.auth.PamAuthenticationProviderImpl.Authenticate(PamAuthenticationProviderImpl.java:54)
 at org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(PlainSaslHelper.java:119)
 at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:103)
 ... 8 more
Caused by: javax.security.sasl.AuthenticationException: Error authenticating with the PAM service: login
 at org.apache.hive.service.auth.PamAuthenticationProviderImpl.Authenticate(PamAuthenticationProviderImpl.java:48)
 ... 10 more
2017-09-19T15:57:11,046 ERROR [HiveServer2-Handler-Pool: Thread-60] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Error validating the login
 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Error validating the login
 at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
 at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
 at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
 at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
 ... 4 more

Hive 2.x queries got stuck when waiting for "tryAcquireCompileLock" in HS2 stacktrace

Env:

Hive 2.x

Symptom:

Hive 2.x queries got stuck when waiting for "tryAcquireCompileLock" in HS2 stacktrace.
When the issue happens, connecting using beeline to HS2 works.
However any queries, eg, "show databases" will hung.

Below is one example of the jstack output on HS2 process:

"7d88a5ad-cd2c-4c37-9025-8372164524fd HiveServer2-Handler-Pool: Thread-214" #214 prio=5 os_prio=0 tid=0x00007fdacc2d1800 nid=0x7af0 waiting on condition [0x00007fda9b6fa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005c1fb7f28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at org.apache.hadoop.hive.ql.Driver.tryAcquireCompileLock(Driver.java:1324)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1236)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1230)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:191)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:276)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:324)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:499)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:486)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
        at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
        at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
        at com.sun.proxy.$Proxy25.executeStatementAsync(Unknown Source)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:294)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)