spark-instrumented-optimizer/sql/hive
Bruce Robbins 558f31b31c [SPARK-23963][SQL] Properly handle large number of columns in query on text-based Hive table
## What changes were proposed in this pull request?

TableReader would get disproportionately slower as the number of columns in the query increased.

I fixed the way TableReader was looking up metadata for each column in the row. Previously, it had been looking up this data in linked lists, accessing each linked list by an index (column number). Now it looks up this data in arrays, where indexing by column number works better.

## How was this patch tested?

Manual testing
All sbt unit tests
python sql tests

Author: Bruce Robbins <bersprockets@gmail.com>

Closes #21043 from bersprockets/tabreadfix.
2018-04-13 14:05:04 -07:00
..
compatibility/src/test/scala/org/apache/spark/sql/hive/execution [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules 2018-01-22 04:31:24 -08:00
src [SPARK-23963][SQL] Properly handle large number of columns in query on text-based Hive table 2018-04-13 14:05:04 -07:00
pom.xml [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00