Understand Decimal precision and scale calculation in Spark using GPU or CPU mode

Goal:

This article research on how Spark calculates the Decimal precision and scale using GPU or CPU mode.

Basically we will test Addition/Subtraction/Multiplication/Division/Modulo/Union in this post.

kubelet failed to start after rebooting

Symptom:

kubelet failed to start after rebooting.

How to use Spark Operator to run Spark job with Rapids Accelerator

Goal:

This article shares the steps on how to run Spark job with Rapids Accelerator using Spark Operator in a Kubernetes Cluster.

Rapids Accelerator compatibility related to spark.sql.legacy.parquet.datetimeRebaseModeInWrite

Goal:

This article talked about the compatibility of Rapids Accelerator for Spark regarding parquet writing related to parameters spark.sql.legacy.parquet.datetimeRebaseModeInWrite etc.

Spark Code -- Dig into SparkListenerEvent

Goal:

This article digs into different types of SparkListenerEvent in Spark event log with some examples. 

Understanding this can help us know how to pares Spark event log.

How to use latest version of Rapids Accelerator for Spark on EMR

Goal:

This article shows how to use latest version of Rapids Accelerator for Spark on EMR. 

Currently the latest EMR 6.2 only ships with Rapids Accelerator 0.2.0 with cuDF 0.15 jar.

However as of today, the latest Rapids Accelerator is 0.4.1 with cuDF 0.18 jar.

Note: This is NOT official steps on enabling rapids+Spark on EMR, but just some technical research.

How to use NVIDIA Nsight Systems to profile a Spark on K8s job with Rapids Accelerator

Goal:

This article explains how to use NVIDIA Nsight Systems to profile a Spark on K8s job with Rapids Accelerator.

This is a follow-up blog after How to use NVIDIA Nsight Systems to profile a Spark job on Rapids Accelerator. 

How to use NVIDIA Nsight Systems to profile a Spark job on Rapids Accelerator

Goal:

This article explains how to use NVIDIA Nsight Systems to profile a Spark job on Rapids Accelerator.

How to enable GpuKryoRegistrator on RAPIDS Accelerator for Spark

Goal:

This article shares the steps to enable GpuKryoRegistrator on RAPIDS Accelerator for Spark.

How to install a Kubernetes Cluster with NVIDIA GPU on AWS using DeepOps

Goal:

This article shares a step-by-step guide on how to install a Kubernetes Cluster with NVIDIA GPU on AWS using DeepOps.