Spark writing to S3 failed: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument

Symptom:

When using Spark writing to S3, the insert query failed:

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
	at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:893)
	at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:869)
	at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1580)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:341)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
	at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
	at org.apache.spark.sql.execution.datasources.FindDataSourceTable.$anonfun$readDataSourceTable$1(DataSourceStrategy.scala:252)
	at org.sparkproject.guava.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
	at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
	at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
	at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
	at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)

Env:

spark-3.2.1-bin-hadoop3.2
hadoop-aws-3.2.3.jar
aws-java-sdk-bundle-1.11.375.jar
guava-14.0.1.jar

Solution:

Remove guava-14.0.1.jar from Spark and use the Hive's newer guava-27.0-jre.jar.

$  ls -altr $SPARK_HOME/jars/guava*.jar
lrwxrwxrwx 1 xxx xxx 46 Jul 19 09:39 /home/xxx/spark/myspark/jars/guava-27.0-jre.jar -> /home/xxx/hive/myhive/lib/guava-27.0-jre.jar