5cb5edaf9c
## What changes were proposed in this pull request? Currently, we use java HashMap for HashedRelation if the key could fit within a Long. The java HashMap and CompactBuffer are not memory efficient, the memory used by them is also accounted accurately. This PR introduce a LongToUnsafeRowMap (similar to BytesToBytesMap) for better memory efficiency and performance. This PR reopen #12190 to fix bugs. ## How was this patch tested? Existing tests. Author: Davies Liu <davies@databricks.com> Closes #12278 from davies/long_map3. |
||
---|---|---|
.. | ||
avro | ||
gen-java/org/apache/spark/sql/execution/datasources/parquet/test/avro | ||
java/test/org/apache/spark/sql | ||
resources | ||
scala/org/apache/spark/sql | ||
scripts | ||
thrift | ||
README.md |
Notes for Parquet compatibility tests
The following directories and files are used for Parquet compatibility tests:
.
├── README.md # This file
├── avro
│ ├── *.avdl # Testing Avro IDL(s)
│ └── *.avpr # !! NO TOUCH !! Protocol files generated from Avro IDL(s)
├── gen-java # !! NO TOUCH !! Generated Java code
├── scripts
│ ├── gen-avro.sh # Script used to generate Java code for Avro
│ └── gen-thrift.sh # Script used to generate Java code for Thrift
└── thrift
└── *.thrift # Testing Thrift schema(s)
To avoid code generation during build time, Java code generated from testing Thrift schema and Avro IDL are also checked in.
When updating the testing Thrift schema and Avro IDL, please run gen-avro.sh
and gen-thrift.sh
accordingly to update generated Java code.
Prerequisites
Please ensure avro-tools
and thrift
are installed. You may install these two on Mac OS X via:
$ brew install thrift avro-tools