[SPARK-13996] [SQL] Add more not null attributes for Filter codegen
## What changes were proposed in this pull request?
JIRA: https://issues.apache.org/jira/browse/SPARK-13996
Filter codegen finds the attributes not null by checking IsNotNull(a) expression with a condition if child.output.contains(a). However, the current approach to checking it is not comprehensive. We can improve it.
E.g., for this plan:
val rdd = sqlContext.sparkContext.makeRDD(Seq(Row(1, "1"), Row(null, "1"), Row(2, "2")))
val schema = new StructType().add("k", IntegerType).add("v", StringType)
val smallDF = sqlContext.createDataFrame(rdd, schema)
val df = smallDF.filter("isnotnull(k + 1)")
The code snippet generated without this patch:
/* 031 */ protected void processNext() throws java.io.IOException {
/* 032 */ /*** PRODUCE: Filter isnotnull((k#0 + 1)) */
/* 033 */
/* 034 */ /*** PRODUCE: INPUT */
/* 035 */
/* 036 */ while (!shouldStop() && inputadapter_input.hasNext()) {
/* 037 */ InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 038 */ /*** CONSUME: Filter isnotnull((k#0 + 1)) */
/* 039 */ /* input[0, int] */
/* 040 */ boolean filter_isNull = inputadapter_row.isNullAt(0);
/* 041 */ int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));
/* 042 */
/* 043 */ /* isnotnull((input[0, int] + 1)) */
/* 044 */ /* (input[0, int] + 1) */
/* 045 */ boolean filter_isNull3 = true;
/* 046 */ int filter_value3 = -1;
/* 047 */
/* 048 */ if (!filter_isNull) {
/* 049 */ filter_isNull3 = false; // resultCode could change nullability.
/* 050 */ filter_value3 = filter_value + 1;
/* 051 */
/* 052 */ }
/* 053 */ if (!(!(filter_isNull3))) continue;
/* 054 */
/* 055 */ filter_metricValue.add(1);
With this patch:
/* 031 */ protected void processNext() throws java.io.IOException {
/* 032 */ /*** PRODUCE: Filter isnotnull((k#0 + 1)) */
/* 033 */
/* 034 */ /*** PRODUCE: INPUT */
/* 035 */
/* 036 */ while (!shouldStop() && inputadapter_input.hasNext()) {
/* 037 */ InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 038 */ /*** CONSUME: Filter isnotnull((k#0 + 1)) */
/* 039 */ /* input[0, int] */
/* 040 */ boolean filter_isNull = inputadapter_row.isNullAt(0);
/* 041 */ int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));
/* 042 */
/* 043 */ if (filter_isNull) continue;
/* 044 */
/* 045 */ filter_metricValue.add(1);
## How was this patch tested?
Existing tests.
Author: Liang-Chi Hsieh <simonh@tw.ibm.com>
Closes #11810 from viirya/add-more-not-null-attrs.
c2f25b1a14