Symptom:
When using Spark writing to S3, the insert query failed:
java.lang.NoSuchMethodError: org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
at org.apache.hadoop.fs.s3a.impl.StoreContext.createThrottledExecutor(StoreContext.java:292)
at org.apache.hadoop.fs.s3a.impl.DeleteOperation.<init>(DeleteOperation.java:206)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:2468)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:532)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.abortJob(FileOutputCommitter.java:551)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.abortJob(HadoopMapReduceCommitProtocol.scala:242)
at org.apache.spark.sql.rapids.GpuFileFormatWriter$.write(GpuFileFormatWriter.scala:262)
Env:
spark-3.2.1-bin-hadoop3.2
hadoop-aws-3.2.1.jar
aws-java-sdk-bundle-1.11.375.jar
Solution:
After upgrading hadoop-aws-3.2.1.jar to hadoop-aws-3.2.3.jar, it works fine.
$ ls -altr $SPARK_HOME/jars|grep -i aws
-rw-rw-r-- 1 xxx xxx 98732349 Jul 26 2018 aws-java-sdk-bundle-1.11.375.jar
-rw-rw-r-- 1 xxx xxx 506819 Jul 19 10:03 hadoop-aws-3.2.3.jar
Sohbet
ReplyDeleteMobil Sohbet
Sohbet Odaları
The article provides a useful solution to the NoSuchMethodError encountered when writing Spark data to Amazon S3. Identifying the Hadoop AWS library version mismatch and upgrading hadoop-aws from 3.2.1 to 3.2.3 is a practical troubleshooting approach for resolving compatibility issues in Spark-based data engineering workflows.
DeleteFor professionals looking to strengthen their Apache Spark skills, PySpark Online Training in India can help them understand distributed data processing, Spark configurations, and practical troubleshooting techniques.
Building expertise in data pipelines, storage integration, and large-scale data processing is also valuable for aspiring data engineers. Data Engineering Online Training in India is relevant for learners who want to develop practical skills in modern data engineering workflows.
Delete9B0F68F8EB
ReplyDeletekiralık hacker
hacker arıyorum
kiralık hacker
hacker arıyorum
belek
I love how practical and actionable your tips are. This blog post was exactly what I needed today!
ReplyDeletechampaign urbana food delivery
Very helpful troubleshooting guide! Issues with Spark writing to S3 can be frustrating, so the clear explanation and practical solution shared here are greatly appreciated. Thanks for making this easier to understand.
ReplyDeletetrucking business advisory consultant
Interesting issue—this Spark S3 failure with NoSuchMethodError often points to Hadoop dependency mismatch. Check your Hadoop/Spark versions and aligned AWS SDK jars. Check your Hadoop/Spark versions and aligned AWS SDK jars. amazon seller accountant
ReplyDelete9944509E
ReplyDeleteAdıyaman
Düzce
Batman
Siirt
Tokat
Isparta
Urfa
Çorum
Karaman