Thursday, December 12, 2019
How to create a MapR PACC using mapr-setup.sh to submit a Spark sample job
Goal:
This article shares the detailed steps on how to create a MapR Persistent Application Client Container(PACC) using mapr-setup.sh to submit a Spark sample job towards a secured MapR cluster.Friday, December 6, 2019
Spark Streaming sample scala code for different sources
Goal:
This article shares some sample Spark Streaming scala code for different sources -- socket text, text files in MapR-FS directory, kafka broker and MapR Event Store for Apache Kafka(MapR Streams).These are wordcount code which can be run directly from spark-shell.
Wednesday, December 4, 2019
How to mount a PersistentVolume for Static Provisioning using MapR CSI in GKE
Goal:
This article explains the detailed steps on how to mount a PersistentVolume for Static Provisioning using MapR Container Storage Interface(CSI) in Google Kubernetes Engine(GKE).How to submit REST requests to a distributed Kafka Connect cluster
Goal:
This article shares the examples of curl commands to submit REST requests to a distributed Kafka Connect cluster.Thursday, November 28, 2019
Understanding different modes in kafka-connect using an example
Goal:
This article is to help understand different modes in kafka-connect using an example.The example will stream data from a mysql table to MapR Event Store for Apache Kafka(aka "MapR Streams") using different modes of kafka-connect -- incrementing, bulk, timestamp and timestamp+incrementing .
Wednesday, November 27, 2019
How to test MapR Data Access Gateway
Goal:
This article explains the steps on how to test MapR Data Access Gateway.Friday, November 15, 2019
Quick tip on how to install python3
Goal:
This is a quick tip on how to install python3 on CentOS 7, MacOS.Friday, November 1, 2019
Can not drop partition or drop Hive partition table due to UTF8 characters in partition value
Symptom:
Can not drop partition or drop Hive partition table due to UTF8 characters in partition value.The partition key value with UTF8 characters shows as "???" in Hive also in mysql(Hive Metastore backend database).
Monday, September 16, 2019
Failed to login a secured Drill on Yarn cluster with MapRSASL authentication
Symptom:
Failed to login a secured Drill on Yarn cluster with MapRSASL authentication.The sample stacktrace inside drillbit.log when trying to use sqlline to connect is:
2019-09-16 14:23:13,331 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.72.41:31010 <--> /10.10.72.41:48032 (user server). Closing connection. io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) [netty-codec-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.0.48.Final.jar:4.0.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handleAuthFailure(ServerAuthenticationHandler.java:324) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:109) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:182) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:54) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [netty-codec-4.0.48.Final.jar:4.0.48.Final] ... 31 common frames omitted Caused by: javax.security.sasl.SaslException: Bad server key at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:190) ~[maprfs-6.1.0-mapr.jar:na] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:239) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_212] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_212] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) ~[hadoop-common-2.7.0-mapr-1808.jar:na] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.evaluateResponse(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.access$500(ServerAuthenticationHandler.java:53) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslInProgressProcessor.process(ServerAuthenticationHandler.java:176) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslStartProcessor.process(ServerAuthenticationHandler.java:164) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:107) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] ... 36 common frames omitted Caused by: javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:143) ~[maprfs-6.1.0-mapr.jar:na] ... 46 common frames omittedThe sample sqlline error message is:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] (state=,code=0) java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:174) at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67) at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67) at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138) at org.apache.drill.jdbc.Driver.connect(Driver.java:72) at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130) at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179) at sqlline.Commands.connect(Commands.java:1247) at sqlline.Commands.connect(Commands.java:1139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38) at sqlline.SqlLine.dispatch(SqlLine.java:722) at sqlline.SqlLine.initArgs(SqlLine.java:416) at sqlline.SqlLine.begin(SqlLine.java:514) at sqlline.SqlLine.start(SqlLine.java:264) at sqlline.SqlLine.main(SqlLine.java:195) Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:210) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458) at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402) at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:165) ... 18 more Caused by: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$SaslFailedProcessor.process(AuthenticationOutcomeListener.java:230) at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:128) at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:53) at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:293) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at java.lang.Thread.run(Thread.java:748)
Subscribe to:
Posts (Atom)
Popular Posts
-
Many commands can check the memory utilization of JAVA processes, for example, pmap, ps, jmap, jstat. What are the differences? Before we ...
-
Hive table contains files in HDFS, if one table or one partition has too many small files, the HiveQL performance may be impacted. Sometime...
-
This is a cookbook for scala programming. 1. Define a object with main function -- Helloworld. object HelloWorld { def main(args: Array...
-
Hive is trying to embrace CBO(cost based optimizer) in latest versions, and Join is one major part of it. Understanding join best practices ...
-
This article shows a sample code to load data into Hbase or MapRDB(M7) using Scala on Spark. I will introduce 2 ways, one is normal load us...
-
Goal: How to build and use parquet-tools to read parquet files. Solution: 1. Download and Install maven. Follow below link: http://...
-
Goal: This article explains what is the difference between Spark HiveContext and SQLContext.
-
Goal: This article explains the configuration parameters for Oozie Launcher job.
-
Goal: This article provides the SQL to list table or partition locations from Hive Metastore. Env: Hive metastore 0.13 on MySQL Root ...
-
Goal: How to control the number of Mappers and Reducers in Hive on Tez.