tag:blogger.com,1999:blog-929270410515568702.post3708640521782432642..comments2024-03-09T22:16:11.028-08:00Comments on Open Knowledge Base: Difference between hive.mapjoin.smalltable.filesize and hive.auto.convert.join.noconditionaltask.sizeOpenKBhttp://www.blogger.com/profile/02892129494774761942noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-929270410515568702.post-6269589874149505062016-03-08T17:40:11.887-08:002016-03-08T17:40:11.887-08:00check my latest reply below.check my latest reply below.OpenKBhttps://www.blogger.com/profile/02892129494774761942noreply@blogger.comtag:blogger.com,1999:blog-929270410515568702.post-16296408177785387042016-03-07T18:02:39.440-08:002016-03-07T18:02:39.440-08:00If you do not want "conditional task", t...If you do not want "conditional task", the only way is to make sure hive.auto.convert.join.noconditionaltask.size is large enough. <br />Check the code ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java:<br /><br /> // If sizes of at least n-1 tables in a n-way join is known, and their sum is smaller than<br /> // the threshold size, convert the join into map-join and don't create a conditional task<br /> boolean convertJoinMapJoin = HiveConf.getBoolVar(conf,<br /> HiveConf.ConfVars.HIVECONVERTJOINNOCONDITIONALTASK);<br /><br />Then:<br />MapRedTask newTask = convertTaskToMapJoinTask(currTask.getWork(), bigTablePosition);<br /><br /> newTask.setTaskTag(Task.MAPJOIN_ONLY_NOBACKUP);<br /> replaceTask(currTask, newTask, physicalContext);<br /><br /><br />After that , the code logic is to check individual table size VS hive.mapjoin.smalltable.filesize. And add possible mapjoin task into "conditional task".<br />The reason why they did not remove "common join" is probably because the statistics may not be accurate during planning time. Hive want to make the final decision at runtime according to the size of each table(or results).<br /><br /><br />OpenKBhttps://www.blogger.com/profile/02892129494774761942noreply@blogger.comtag:blogger.com,1999:blog-929270410515568702.post-62101793719321631322016-03-06T12:08:44.905-08:002016-03-06T12:08:44.905-08:00Also, when we talk of the "conditional task&q...Also, when we talk of the "conditional task", do we mean checking the size of the small table or checking the size of the n-1 table/partitions? The reason why i'm confused with these 2 parameters is that i do not understand the need for hive.auto.convert.join.noconditionaltask.size, when hive.mapjoin.smalltable.filesize is already serving the purpose in an n way join.<br /><br />~AbhilashAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-929270410515568702.post-68196013190044818522016-03-06T12:02:24.490-08:002016-03-06T12:02:24.490-08:00Thanks for your prompt reply.
So why is it that i...Thanks for your prompt reply.<br /><br />So why is it that in the 2 way join example (with hive.auto.convert.join.noconditionaltask.size=10000000 and hive.mapjoin.smalltable.filesize=31000000) does the query plan output show both stage 1 and stage 5? Shouldn't the plan show only the stage 5?<br /><br />Thanks,<br />AbhilashAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-929270410515568702.post-63411325352621147052016-03-05T23:39:54.088-08:002016-03-05T23:39:54.088-08:00If you think like a query planner, of course, all ...If you think like a query planner, of course, all plans should be in consideration. <br />However the explain plan output may not print all possible plans due to the value of some parameters such as hive.auto.convert.join.noconditionaltask.size and hive.mapjoin.smalltable.filesize.<br /><br />Look at Plan a in 2-way join, the map join plan is not printed simply because: the smaller table size(30MB) > hive.mapjoin.smalltable.filesize.<br />OpenKBhttps://www.blogger.com/profile/02892129494774761942noreply@blogger.comtag:blogger.com,1999:blog-929270410515568702.post-83865552283163005512016-03-05T23:31:15.924-08:002016-03-05T23:31:15.924-08:00Shouldn't all the plans created be the same fo...Shouldn't all the plans created be the same for a particular query? Which means the Hive driver create permutation of all possible ways in which queries could be run? like the plan for the second query, which draws out plans for both common join and a map join. Of course, its only during execution, will a certain path in the query plan(map join or common join) be executed. the actual selection of the path could be based off of hive.auto.convert.join.noconditionaltask.size and <br />hive.mapjoin.smalltable.filesize<br /><br />Also, i'm a little confused with the differences in the above properties. Could you elaborate a little. I couldn't get any help online. Its the same copy pasted thing everywhere.<br /><br />Thanks<br />Abhilash<br />Anonymousnoreply@blogger.com