ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/sql/core/benchmarks
History
Max Gekk 350aa859fe [SPARK-32006][SQL] Create date/timestamp formatters once before collect in hiveResultString() 
### What changes were proposed in this pull request?
1. Add method `getTimeFormatters` to `HiveResult` which creates timestamp and date formatters.
2. Move creation of `dateFormatter` and `timestampFormatter` from the constructor of the `HiveResult` object to `HiveResult. hiveResultString()` via `getTimeFormatters`. This allows to resolve time zone ID from Spark's session time zone `spark.sql.session.timeZone` and create date/timestamp formatters only once before collecting `java.sql.Timestamp`/`java.sql.Date` values.
3. Create date/timestamp formatters once in SparkExecuteStatementOperation.

### Why are the changes needed?
To fix perf regression comparing to Spark 2.4

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- By existing test suite `HiveResultSuite` and etc.
- Re-generate benchmarks results of `DateTimeBenchmark` in the environment:

| Item | Description |
| ---- | ----|
| Region | us-west-2 (Oregon) |
| Instance | r3.xlarge |
| AMI | ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 (ami-06f2f779464715dc5) |
| Java | OpenJDK 64-Bit Server VM 1.8.0_252 and OpenJDK 64-Bit Server VM 11.0.7+10 |

Closes #28842 from MaxGekk/opt-toHiveString-oss-master.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
2020-06-17 06:28:47 +00:00
..
AggregateBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
AggregateBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BloomFilterBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BloomFilterBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
BuiltInDataSourceWriteBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
BuiltInDataSourceWriteBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ColumnarBatchBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ColumnarBatchBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
CompressionSchemeBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
CompressionSchemeBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
CSVBenchmark-jdk11-results.txt [SPARK-31755][SQL][FOLLOWUP] Update date-time, CSV and JSON benchmark results 2020-05-25 15:00:11 +00:00
CSVBenchmark-results.txt [SPARK-31755][SQL][FOLLOWUP] Update date-time, CSV and JSON benchmark results 2020-05-25 15:00:11 +00:00
DatasetBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
DatasetBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
DataSourceReadBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
DataSourceReadBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
DateTimeBenchmark-jdk11-results.txt [SPARK-32006][SQL] Create date/timestamp formatters once before collect in hiveResultString() 2020-06-17 06:28:47 +00:00
DateTimeBenchmark-results.txt [SPARK-32006][SQL] Create date/timestamp formatters once before collect in hiveResultString() 2020-06-17 06:28:47 +00:00
DateTimeRebaseBenchmark-jdk11-results.txt [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode 2020-06-15 07:25:56 +00:00
DateTimeRebaseBenchmark-results.txt [SPARK-31992][SQL] Benchmark the EXCEPTION rebase mode 2020-06-15 07:25:56 +00:00
ExternalAppendOnlyUnsafeRowArrayBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ExternalAppendOnlyUnsafeRowArrayBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
ExtractBenchmark-jdk11-results.txt [SPARK-31507][SQL] Remove uncommon fields support and update some fields with meaningful names for extract function 2020-04-22 10:24:49 +00:00
ExtractBenchmark-results.txt [SPARK-31507][SQL] Remove uncommon fields support and update some fields with meaningful names for extract function 2020-04-22 10:24:49 +00:00
FilterPushdownBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
FilterPushdownBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
HashedRelationMetricsBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
HashedRelationMetricsBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
InExpressionBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
InExpressionBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
IntervalBenchmark-jdk11-results.txt [SPARK-31129][SQL][TESTS] Fix IntervalBenchmark and DateTimeBenchmark 2020-03-12 12:59:29 -07:00
IntervalBenchmark-results.txt [SPARK-31129][SQL][TESTS] Fix IntervalBenchmark and DateTimeBenchmark 2020-03-12 12:59:29 -07:00
JoinBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
JoinBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
JsonBenchmark-jdk11-results.txt [SPARK-31755][SQL][FOLLOWUP] Update date-time, CSV and JSON benchmark results 2020-05-25 15:00:11 +00:00
JsonBenchmark-results.txt [SPARK-31755][SQL][FOLLOWUP] Update date-time, CSV and JSON benchmark results 2020-05-25 15:00:11 +00:00
MakeDateTimeBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
MakeDateTimeBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
MetricsAggregationBenchmark-jdk11-results.txt [SPARK-29562][SQL] Speed up and slim down metric aggregation in SQL listener 2019-10-24 22:18:10 -07:00
MetricsAggregationBenchmark-results.txt [SPARK-29562][SQL] Speed up and slim down metric aggregation in SQL listener 2019-10-24 22:18:10 -07:00
MiscBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
MiscBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcNestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcNestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcV2NestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
OrcV2NestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
ParquetNestedPredicatePushDownBenchmark-jdk11-results.txt [SPARK-31364][SQL][TESTS] Benchmark Parquet Nested Field Predicate Pushdown 2020-04-24 22:10:58 +00:00
ParquetNestedPredicatePushDownBenchmark-results.txt [SPARK-31364][SQL][TESTS] Benchmark Parquet Nested Field Predicate Pushdown 2020-04-24 22:10:58 +00:00
ParquetNestedSchemaPruningBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
ParquetNestedSchemaPruningBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
PrimitiveArrayBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
PrimitiveArrayBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
RangeBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
RangeBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
SortBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
SortBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
TPCDSQueryBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
TPCDSQueryBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UDFBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UDFBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
UnsafeArrayDataBenchmark-jdk11-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
UnsafeArrayDataBenchmark-results.txt [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) 2019-10-03 08:58:25 -07:00
WideSchemaBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideSchemaBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideTableBenchmark-jdk11-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00
WideTableBenchmark-results.txt [SPARK-30409][SPARK-29173][SQL][TESTS] Use NoOp datasource in SQL benchmarks 2020-01-12 13:18:19 -08:00