ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/sql/core/benchmarks
History
Karuppayya Rajendran 0a58029d52 [SPARK-31897][SQL] Enable codegen for GenerateExec 
### What changes were proposed in this pull request?
Enabling codegen for GenerateExec

### Why are the changes needed?
To leverage code generation for Generators

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- UT tests added

### Benchmark
```
case class Data(value1: Float, value2: Map[String, String], value3: String)
val path = "<path>"

val numRecords = Seq(10000000, 100000000)
numRecords.map {
  recordCount =>
    import java.util.concurrent.TimeUnit.NANOSECONDS

    val srcDF = spark.range(recordCount).map {
      x => Data(x.toFloat, Map(x.toString -> x.toString ), s"value3$x")
    }.select($"value1", explode($"value2"), $"value3")
    val start = System.nanoTime()
    srcDF
      .write
      .mode("overwrite")
      .parquet(s"$path/$recordCount")
    val end = System.nanoTime()
    val diff = end - start
    (recordCount, NANOSECONDS.toMillis(diff))
}
```
**With codegen**:
```
res0: Seq[(Int, Long)] = List((10000000,13989), (100000000,129625))
```
**Without codegen**:
```
res0: Seq[(Int, Long)] = List((10000000,15736), (100000000,150399))
```

Closes #28715 from karuppayya/SPARK-31897.

Lead-authored-by: Karuppayya Rajendran <karuppayya1990@gmail.com>
Co-authored-by: Karuppayya Rajendran <karuppayya.rajendran@apple.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
2021-03-18 20:50:28 -07:00
..
AggregateBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
AggregateBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BloomFilterBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BloomFilterBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BuiltInDataSourceWriteBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
BuiltInDataSourceWriteBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
CharVarcharBenchmark-results.txt [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too 2021-01-26 02:08:35 +08:00
ColumnarBatchBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ColumnarBatchBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
CompressionSchemeBenchmark-jdk11-results.txt [SPARK-32802][SQL] Avoid using SpecificInternalRow in RunLengthEncoding#Encoder 2020-09-12 22:19:30 -07:00
CompressionSchemeBenchmark-results.txt [SPARK-32802][SQL] Avoid using SpecificInternalRow in RunLengthEncoding#Encoder 2020-09-12 22:19:30 -07:00
CSVBenchmark-jdk11-results.txt [SPARK-30648][SQL] Support filters pushdown in JSON datasource 2020-07-17 00:01:13 +09:00
CSVBenchmark-results.txt [SPARK-30648][SQL] Support filters pushdown in JSON datasource 2020-07-17 00:01:13 +09:00
DatasetBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
DatasetBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
DataSourceReadBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
DataSourceReadBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
DateTimeBenchmark-jdk11-results.txt [SPARK-33404][SQL][FOLLOWUP] Update benchmark results for date_trunc 2020-11-11 08:50:43 -08:00
DateTimeBenchmark-results.txt [SPARK-33404][SQL][FOLLOWUP] Update benchmark results for date_trunc 2020-11-11 08:50:43 -08:00
DateTimeRebaseBenchmark-jdk11-results.txt [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing 2020-10-22 10:03:41 +09:00
DateTimeRebaseBenchmark-results.txt [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing 2020-10-22 10:03:41 +09:00
ExternalAppendOnlyUnsafeRowArrayBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ExternalAppendOnlyUnsafeRowArrayBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ExtractBenchmark-jdk11-results.txt [SPARK-31507][SQL] Remove uncommon fields support and update some fields with meaningful names for extract function 2020-04-22 10:24:49 +00:00
ExtractBenchmark-results.txt [SPARK-31507][SQL] Remove uncommon fields support and update some fields with meaningful names for extract function 2020-04-22 10:24:49 +00:00
FilterPushdownBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
FilterPushdownBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
GenerateExecBenchmark-jdk11-results.txt [SPARK-31897][SQL] Enable codegen for GenerateExec 2021-03-18 20:50:28 -07:00
GenerateExecBenchmark-results.txt [SPARK-31897][SQL] Enable codegen for GenerateExec 2021-03-18 20:50:28 -07:00
HashedRelationMetricsBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
HashedRelationMetricsBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
InExpressionBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
InExpressionBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
InsertTableWithDynamicPartitionsBenchmark-jdk11-results.txt [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct 2020-10-22 14:01:07 +00:00
InsertTableWithDynamicPartitionsBenchmark-results.txt [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct 2020-10-22 14:01:07 +00:00
IntervalBenchmark-jdk11-results.txt [SPARK-32071][SQL][TESTS] Add make_interval benchmark 2020-06-27 17:54:06 -07:00
IntervalBenchmark-results.txt [SPARK-32071][SQL][TESTS] Add make_interval benchmark 2020-06-27 17:54:06 -07:00
JoinBenchmark-jdk11-results.txt [SPARK-34620][SQL] Code-gen broadcast nested loop join (inner/cross) 2021-03-09 11:45:43 +00:00
JoinBenchmark-results.txt [SPARK-34620][SQL] Code-gen broadcast nested loop join (inner/cross) 2021-03-09 11:45:43 +00:00
JsonBenchmark-jdk11-results.txt [SPARK-30648][SQL] Support filters pushdown in JSON datasource 2020-07-17 00:01:13 +09:00
JsonBenchmark-results.txt [SPARK-30648][SQL] Support filters pushdown in JSON datasource 2020-07-17 00:01:13 +09:00
MakeDateTimeBenchmark-jdk11-results.txt [SPARK-32072][CORE][TESTS] Fix table formatting with benchmark results 2020-06-24 04:43:53 +00:00
MakeDateTimeBenchmark-results.txt [SPARK-32072][CORE][TESTS] Fix table formatting with benchmark results 2020-06-24 04:43:53 +00:00
MetricsAggregationBenchmark-jdk11-results.txt [SPARK-29562][SQL] Speed up and slim down metric aggregation in SQL listener 2019-10-24 22:18:10 -07:00
MetricsAggregationBenchmark-results.txt [SPARK-29562][SQL] Speed up and slim down metric aggregation in SQL listener 2019-10-24 22:18:10 -07:00
MiscBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
MiscBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcNestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcNestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcV2NestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcV2NestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
ParquetNestedPredicatePushDownBenchmark-jdk11-results.txt [SPARK-31364][SQL][TESTS] Benchmark Parquet Nested Field Predicate Pushdown 2020-04-24 22:10:58 +00:00
ParquetNestedPredicatePushDownBenchmark-results.txt [SPARK-31364][SQL][TESTS] Benchmark Parquet Nested Field Predicate Pushdown 2020-04-24 22:10:58 +00:00
ParquetNestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
ParquetNestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
PrimitiveArrayBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
PrimitiveArrayBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
RangeBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
RangeBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
SortBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
SortBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
SubExprEliminationBenchmark-jdk11-results.txt [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark 2020-11-25 15:22:47 -08:00
SubExprEliminationBenchmark-results.txt [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark 2020-11-25 15:22:47 -08:00
TPCDSQueryBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
TPCDSQueryBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UDFBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UDFBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UnsafeArrayDataBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
UnsafeArrayDataBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
UpdateFieldsBenchmark-results.txt [SPARK-32511][SQL] Add dropFields method to Column class 2020-10-06 08:53:30 +00:00
WideSchemaBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideSchemaBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideTableBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideTableBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00