Friday, November 15, 2019
Friday, November 1, 2019
Can not drop partition or drop Hive partition table due to UTF8 characters in partition value
Symptom:
Can not drop partition or drop Hive partition table due to UTF8 characters in partition value.The partition key value with UTF8 characters shows as "???" in Hive also in mysql(Hive Metastore backend database).
Monday, September 16, 2019
Failed to login a secured Drill on Yarn cluster with MapRSASL authentication
Symptom:
Failed to login a secured Drill on Yarn cluster with MapRSASL authentication.The sample stacktrace inside drillbit.log when trying to use sqlline to connect is:
2019-09-16 14:23:13,331 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.72.41:31010 <--> /10.10.72.41:48032 (user server). Closing connection. io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handleAuthFailure(ServerAuthenticationHandler.java:324) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:109) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:182) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:54) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [netty-codec-4.0.48.Final.jar:4.0.48.Final] ... 31 common frames omitted Caused by: javax.security.sasl.SaslException: Bad server key at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:190) ~[maprfs-6.1.0-mapr.jar:na] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:239) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_212] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_212] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) ~[hadoop-common-2.7.0-mapr-1808.jar:na] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.evaluateResponse(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.access$500(ServerAuthenticationHandler.java:53) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslInProgressProcessor.process(ServerAuthenticationHandler.java:176) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslStartProcessor.process(ServerAuthenticationHandler.java:164) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:107) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] ... 36 common frames omitted Caused by: javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:143) ~[maprfs-6.1.0-mapr.jar:na] ... 46 common frames omittedThe sample sqlline error message is:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] (state=,code=0) java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:174) at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) at org.apache.drill.jdbc.Driver.connect(Driver.java:72) at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130) at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179) at sqlline.Commands.connect(Commands.java:1247) at sqlline.Commands.connect(Commands.java:1139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) at sqlline.SqlLine.dispatch(SqlLine.java:722) at sqlline.SqlLine.initArgs(SqlLine.java:416) at sqlline.SqlLine.begin(SqlLine.java:514) at sqlline.SqlLine.start(SqlLine.java:264) at sqlline.SqlLine.main(SqlLine.java:195) Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:210) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402) at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:165) ... 18 more Caused by: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$SaslFailedProcessor.process(AuthenticationOutcomeListener.java:230) at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:128) at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:53) at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:293) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at java.lang.Thread.run(Thread.java:748)
Thursday, September 12, 2019
Step by step on how to install a MapR Client on Windows 7 to access multiple secured clusters
Goal:
This article explains the detailed steps to install and configure a MapR Client on Windows 7 to access multiple secured MapR Clusters.Wednesday, April 24, 2019
Some reducer output files are not moved into final output directory after a MapReduce Job with customized FileOutputCommitter is migrated from MRv1 to MRv2.
Symptom:
After a MapReduce Job with customized FileOutputCommitter got migrated from MRv1 to MRv2, some reducer output files are not moved into final output directory.Take a sample WordCount job with a customized FileOutputCommitter for example.
In this specific case, only 1st reducer's output files are moved into final output directory.
All other reducers' output files are still sitting in the temporary location:
/hao/wordfinal/part_00000/part-r-00000 /hao/wordfinal/part_00001/_temporary/1/task_1554837836642_0091_r_000001/part-r-00001 /hao/wordfinal/part_00002/_temporary/1/task_1554837836642_0091_r_000001/part-r-00002The expected final output files should be:
/hao/wordfinal/part_00000/part-r-00000 /hao/wordfinal/part_00001/part-r-00001 /hao/wordfinal/part_00002/part-r-00002Note: The same job works fine in MRv1 though before migrating to MRv2.
Tuesday, April 23, 2019
What is the difference between mapreduce.fileoutputcommitter.algorithm.version=1 and 2
Goal:
This article explains the difference between mapreduce.fileoutputcommitter.algorithm.version=1 and 2 using a sample wordcount job.How to customize FileOutputCommitter for MapReduce job by overwriting Output Format Class
Goal:
This article explains how to customize FileOutputCommitter for MapReduce job by overwriting Output Format Class.This can be used to change the output directory, customize the file name etc.
Monday, March 11, 2019
How to use snapshot feature of MapR-FS to isolate Hive reads and writes
Goal:
As we know, Hive locks can be used to isolate reads and writes.Please refer to my previous blogs for details:
http://www.openkb.info/2014/11/hive-locks-tablepartition-level.html
http://www.openkb.info/2018/07/hive-different-lock-behaviors-between.html
That lock mechanism is maintained by Hive and it could have performance overhead or unexpected lock behavior based on your application logic.
This article shows another way to use snapshot feature of MapR-FS to isolate Hive reads and writes.
Wednesday, February 6, 2019
"msck repair" of a partition table fails with error "Expecting only one partition but more than one partitions are found.".
Symptom:
In spark-shell, "msck repair" of a partition table named "database_name.table_name" fails with error "Expecting only one partition but more than one partitions are found.".Thursday, January 24, 2019
Subscribe to:
Posts (Atom)
Popular Posts
-
Many commands can check the memory utilization of JAVA processes, for example, pmap, ps, jmap, jstat. What are the differences? Before we ...
-
Hive table contains files in HDFS, if one table or one partition has too many small files, the HiveQL performance may be impacted. Sometime...
-
This is a cookbook for scala programming. 1. Define a object with main function -- Helloworld. object HelloWorld { def main(args: Array...
-
Hive is trying to embrace CBO(cost based optimizer) in latest versions, and Join is one major part of it. Understanding join best practices ...
-
This article shows a sample code to load data into Hbase or MapRDB(M7) using Scala on Spark. I will introduce 2 ways, one is normal load us...
-
Goal: How to build and use parquet-tools to read parquet files. Solution: 1. Download and Install maven. Follow below link: http://...
-
Goal: This article explains what is the difference between Spark HiveContext and SQLContext.
-
Goal: This article explains the configuration parameters for Oozie Launcher job.
-
Goal: This article provides the SQL to list table or partition locations from Hive Metastore. Env: Hive metastore 0.13 on MySQL Root ...
-
Goal: How to control the number of Mappers and Reducers in Hive on Tez.