Nong Li 6d0848b53b [SPARK-11787][SQL] Improve Parquet scan performance when using flat schemas. ... This patch adds an alternate to the Parquet RecordReader from the parquet-mr project that is much faster for flat schemas. Instead of using the general converter mechanism from parquet-mr, this directly uses the lower level APIs from parquet-columnar and a customer RecordReader that directly assembles into UnsafeRows. This is optionally disabled and only used for supported schemas. Using the tpcds store sales table and doing a sum of increasingly more columns, the results are: For 1 Column: Before: 11.3M rows/second After: 18.2M rows/second For 2 Columns: Before: 7.2M rows/second After: 11.2M rows/second For 5 Columns: Before: 2.9M rows/second After: 4.5M rows/second Author: Nong Li <nong@databricks.com> Closes #9774 from nongli/parquet.		2015-11-18 18:38:45 -08:00
..
src	[SPARK-11787][SQL] Improve Parquet scan performance when using flat schemas.	2015-11-18 18:38:45 -08:00
pom.xml	Revert "[SPARK-11271][SPARK-11016][CORE] Use Spark BitSet instead of RoaringBitmap to reduce memory usage"	2015-11-16 14:50:38 -08:00