Thursday, June 25, 2015

TaskTracker goes down with error "Comparison method violates its general contract"


Hadoop version < 1.3


TaskTracker goes down with below stacktrace in tasktracker and jobtracker logs:

ERROR org.apache.hadoop.ipc.FailoverRPC: FailoverProxy: Failing this Call: heartbeat for error(RemoteException): 
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeHi(
at java.util.TimSort.mergeAt(
at java.util.TimSort.mergeForceCollapse(
at java.util.TimSort.sort(
at java.util.TimSort.sort(
at java.util.Arrays.sort(
at java.util.Collections.sort(
at org.apache.hadoop.mapred.PoolSchedulable.assignTask(
at org.apache.hadoop.mapred.FairScheduler.assignTasks(
at org.apache.hadoop.mapred.JobTracker.heartbeat(
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at org.apache.hadoop.ipc.WritableRpcEngine$Server$
at org.apache.hadoop.ipc.RPC$
at org.apache.hadoop.ipc.Server$Handler$
at org.apache.hadoop.ipc.Server$Handler$
at Method)
at org.apache.hadoop.ipc.Server$

Root Cause:

This is caused by issue MAPREDUCE-5966, If a custom weight adjuster is used, that may be called multiple times during a sort causing different values to return.
That causes a failure in sorting because the weight may change during the sort.
This issue is fixed in Hadoop 1.3.0.


Remove below 3 configurations from mapred-site.xml from all client nodes, if they are in use.
The client nodes mean the nodes where MapReduce jobs are submitted.
  • mapred.fairscheduler.weightadjuster
  • mapred.newjobweightbooster.factor
  • mapred.newjobweightbooster.duration
If other Hadoop ecosystem products are used to submit MapReduce jobs, make sure above 3 configurations are not set in client side.
Take Hive for example, make sure they are not set in hive-site.xml or Hive job level settings.

No comments:

Post a Comment

Popular Posts