Thursday, July 10, 2014

[BUG] JDBC using PreparedStatement can not close the statements properly.

Env:

impalad version 1.3.1-cdh5 RELEASE (build )
hive-jdbc-0.10.0-cdh4.2.0.jar

Symptom:

When using "PreparedStatement" to execute SQLs in Impala in a loop, it can not close the statements properly.
From "http://<impalad IP>:25000/queries" page, you may see many queries are running in parallel although you are using one single connection.
Many of them are in "FINISHED" state, but not closed.

If the SQL count in the loop is above the resource pool capacity(default pool is 20), below error message shows:
java.sql.SQLException: Admission for query id=78415bdbab8f54e0:3cbe172156d17c99 exceeded timeout 60000ms
 at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:159)
 at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:147)
 at org.apache.hive.jdbc.HivePreparedStatement.executeImmediate(HivePreparedStatement.java:187)
 at org.apache.hive.jdbc.HivePreparedStatement.executeQuery(HivePreparedStatement.java:149)
 at ClouderaImpalaJdbcExample2.main(ClouderaImpalaJdbcExample2.java:40)

Reproduce:

ClouderaImpalaJdbcExample.java is using "java.sql.Statement" and it works fine.
 Class.forName(JDBC_DRIVER_NAME);
 con = DriverManager.getConnection(CONNECTION_URL);
            
            for (int i=1;i<=21;i++) {
             Statement stmt = con.createStatement();
             ResultSet rs = stmt.executeQuery(SQL_STATEMENT);
             System.out.println("\n== Begin Query Results ======================");
             while (rs.next()) {
              System.out.println(rs.getString(1));
             }
             System.out.println("== End Query Results =======================\n\n");
             stmt.close();
             rs.close();
            }
ClouderaImpalaJdbcExample2.java is using "java.sql.PreparedStatement" and it can reproduce the issue.
 Class.forName(JDBC_DRIVER_NAME);
 con = DriverManager.getConnection(CONNECTION_URL);
            for (int i=1;i<=21;i++) {
             PreparedStatement stmt=con.prepareStatement(SQL_STATEMENT);
                stmt.setString(1,"abc");
             ResultSet rs = stmt.executeQuery();
             System.out.println("\n== Begin Query Results ======================");
             while (rs.next()) {
              System.out.println(rs.getString(1));
             }
             System.out.println("== End Query Results =======================\n\n");
             stmt.close();
             rs.close();
            }

Fix:

This issue is fixed in latest hive-jdbc, for example, hive-jdbc-0.12.0-cdh5.0.2.jar.
If you are using CM5 parcels installation, you can specify below CLASSPATH:
export CLASSPATH=/opt/cloudera/parcels/CDH/lib/hive/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop/*:.

No comments:

Post a Comment

Popular Posts