How to create a MapR PACC using mapr-setup.sh to submit a Spark sample job

Goal:

This article shares the detailed steps on how to create a MapR Persistent Application Client Container(PACC) using mapr-setup.sh to submit a Spark sample job towards a secured MapR cluster.

Env:

MapR 6.1 (secured) with FUSE based posix client running.
Docker is installed and running on Mac

Solution:

1. Generate a service ticket for user "mapr" on the secured MapR cluster

maprlogin generateticket -type service -cluster my61.cluster.com -duration 30:0:0 -out /tmp/mapr_ticket -user mapr

2. Copy the service ticket to Mac where docker is running

Say this location on my mac is /Users/hzu/pacc/mapr_ticket

3. Download mapr-setup.sh on Mac

curl -O https://package.mapr.com/releases/installer/mapr-setup.sh
chmod +x ./mapr-setup.sh

4. Create MapR PACC Image Using mapr-setup.sh

./mapr-setup.sh docker client

Follow doc: https://mapr.com/docs/61/AdvancedInstallation/CreatingPACCImage.html
Note: We add spark client at least.

5. Edit ./docker_images/client/mapr-docker-client.sh

MAPR_CLUSTER=my61.cluster.com
MAPR_CLDB_HOSTS=v1.poc.com,v2.poc.com,v3.poc.com
MAPR_MOUNT_PATH=/maprfuse
MAPR_TICKET_FILE=/Users/hzu/pacc/mapr_ticket
MAPR_TICKETFILE_LOCATION="/tmp/$(basename $MAPR_TICKET_FILE)"
MAPR_CONTAINER_USER=mapr
MAPR_CONTAINER_UID=5000
MAPR_CONTAINER_GROUP=mapr
MAPR_CONTAINER_GID=5000
MAPR_MEMORY=0
MAPR_DOCKER_NETWORK=bridge

6. Run mapr-docker-client.sh to start the container

./docker_images/client/mapr-docker-client.sh

7. Verify the container have access to posix mount point

[mapr@8955101793bf ~]$ ls -altr  /maprfuse
total 1
drwxr-xr-x 11 mapr mapr 12 Dec  6 12:46 my61.cluster.com

[mapr@8955101793bf ~]$ rpm -qa|grep -i mapr-
mapr-client-6.1.0.20180926230239.GA-1.x86_64
mapr-posix-client-container-6.1.0.20180926230239.GA-1.x86_64
mapr-hive-2.3.201809220807-1.noarch
mapr-librdkafka-0.11.3.201803231414-1.noarch
mapr-spark-2.3.1.201809221841-1.noarch
mapr-kafka-1.1.1.201809281337-1.noarch
mapr-pig-0.16.201707251429-1.noarch

8. Verify that submitting spark job works in the container

/opt/mapr/spark/spark-2.3.1/bin/run-example --master yarn --deploy-mode cluster SparkPi 10

Common Issues:

If mapr-setup.sh fails with below error on Mac, please add Mac's IP address and hostname into /etc/hosts in advance.

ERROR: Hostname (mymacbook.local) cannot be resolved. Correct the problem and retry mapr-setup.sh

References:

https://mapr.com/products/persistent-application-client-container/
https://mapr.com/blog/persistent-storage-docker-containers-whiteboard-walkthrough/
https://mapr.com/docs/61/AdvancedInstallation/UsingtheMapRPACC.html
https://mapr.com/docs/61/AdvancedInstallation/CreatingPACCImage.html
https://mapr.com/docs/61/AdvancedInstallation/CustomizingaMapRPACC.html
https://mapr.com/blog/getting-started-mapr-client-container/
https://hub.docker.com/r/maprtech/pacc/tags/