Thursday, December 12, 2019

How to use nodeSelector to constrain POD csi-controller-kdf-0 to only be able to run on particular Node(s)

Goal:

This article explains how to use nodeSelector to constrain POD csi-controller-kdf-0 to only be able to run on particular Node(s).

How to create a MapR PACC using mapr-setup.sh to submit a Spark sample job

Goal:

This article shares the detailed steps on how to create a MapR Persistent Application Client Container(PACC) using mapr-setup.sh to submit a Spark sample job towards a secured MapR cluster.

Friday, December 6, 2019

Spark Streaming sample scala code for different sources

Goal:

This article shares some sample Spark Streaming scala code for different sources -- socket text, text files in MapR-FS directory, kafka broker and MapR Event Store for Apache Kafka(MapR Streams).
These are wordcount code which can be run directly from spark-shell.

Wednesday, December 4, 2019

How to mount a PersistentVolume for Static Provisioning using MapR CSI in GKE

Goal:

This article explains the detailed steps on how to mount a PersistentVolume for Static Provisioning using MapR Container Storage Interface(CSI) in Google Kubernetes Engine(GKE).

How to submit REST requests to a distributed Kafka Connect cluster

Goal:

This article shares the examples of curl commands to submit REST requests to a distributed Kafka Connect cluster.

Thursday, November 28, 2019

Understanding different modes in kafka-connect using an example

Goal:

This article is to help understand different modes in kafka-connect using an example.
The example will stream data from a mysql table to MapR Event Store for Apache Kafka(aka "MapR Streams") using different modes of kafka-connect -- incrementing, bulk, timestamp and timestamp+incrementing .

Wednesday, November 27, 2019

Friday, November 15, 2019

Friday, November 1, 2019

Can not drop partition or drop Hive partition table due to UTF8 characters in partition value

Symptom:

Can not drop partition or drop Hive partition table due to UTF8 characters in partition value.
The partition key value with UTF8 characters shows as "???" in Hive also in mysql(Hive Metastore backend database).

Monday, September 16, 2019

Failed to login a secured Drill on Yarn cluster with MapRSASL authentication

Symptom:

Failed to login a secured Drill on Yarn cluster with MapRSASL authentication.
The sample stacktrace inside drillbit.log when trying to use sqlline to connect is:
2019-09-16 14:23:13,331 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /10.10.72.41:31010 <--> /10.10.72.41:48032 (user server).  Closing connection.
io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key  [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: Bad server key  [Caused by javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handleAuthFailure(ServerAuthenticationHandler.java:324) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:109) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:182) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.BasicServer.handle(BasicServer.java:54) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) ~[drill-rpc-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [netty-codec-4.0.48.Final.jar:4.0.48.Final]
 ... 31 common frames omitted
Caused by: javax.security.sasl.SaslException: Bad server key
 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:190) ~[maprfs-6.1.0-mapr.jar:na]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:239) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$1.run(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_212]
 at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_212]
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) ~[hadoop-common-2.7.0-mapr-1808.jar:na]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.evaluateResponse(ServerAuthenticationHandler.java:236) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.access$500(ServerAuthenticationHandler.java:53) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslInProgressProcessor.process(ServerAuthenticationHandler.java:176) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler$SaslStartProcessor.process(ServerAuthenticationHandler.java:164) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.rpc.security.ServerAuthenticationHandler.handle(ServerAuthenticationHandler.java:107) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 ... 36 common frames omitted
Caused by: javax.security.sasl.SaslException: Error while trying to decrypt ticket: 2
 at com.mapr.security.maprsasl.MaprSaslServer.evaluateResponse(MaprSaslServer.java:143) ~[maprfs-6.1.0-mapr.jar:na]
 ... 46 common frames omitted
The sample sqlline error message is:
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:174)
 at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
 at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
 at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
 at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
 at sqlline.DatabaseConnection.connect(DatabaseConnection.java:130)
 at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
 at sqlline.Commands.connect(Commands.java:1247)
 at sqlline.Commands.connect(Commands.java:1139)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
 at sqlline.SqlLine.dispatch(SqlLine.java:722)
 at sqlline.SqlLine.initArgs(SqlLine.java:416)
 at sqlline.SqlLine.begin(SqlLine.java:514)
 at sqlline.SqlLine.start(SqlLine.java:264)
 at sqlline.SqlLine.main(SqlLine.java:195)
Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:210)
 at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
 at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
 at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:165)
 ... 18 more
Caused by: javax.security.sasl.SaslException: Authentication failed. Incorrect credentials? [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$SaslFailedProcessor.process(AuthenticationOutcomeListener.java:230)
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:128)
 at org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.success(AuthenticationOutcomeListener.java:53)
 at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:293)
 at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 at java.lang.Thread.run(Thread.java:748)

Popular Posts