f2ee0ed4b7
## What changes were proposed in this pull request? Currently SparkSession.Builder use SQLContext.getOrCreate. It should probably the the other way around, i.e. all the core logic goes in SparkSession, and SQLContext just calls that. This patch does that. This patch also makes sure config options specified in the builder are propagated to the existing (and of course the new) SparkSession. ## How was this patch tested? Updated tests to reflect the change, and also introduced a new SparkSessionBuilderSuite that should cover all the branches. Author: Reynold Xin <rxin@databricks.com> Closes #13200 from rxin/SPARK-15075. |
||
|---|---|---|
| .. | ||
| avro | ||
| gen-java/org/apache/spark/sql/execution/datasources/parquet/test/avro | ||
| java/test/org/apache/spark/sql | ||
| resources | ||
| scala/org/apache/spark/sql | ||
| scripts | ||
| thrift | ||
| README.md | ||
Notes for Parquet compatibility tests
The following directories and files are used for Parquet compatibility tests:
.
├── README.md # This file
├── avro
│ ├── *.avdl # Testing Avro IDL(s)
│ └── *.avpr # !! NO TOUCH !! Protocol files generated from Avro IDL(s)
├── gen-java # !! NO TOUCH !! Generated Java code
├── scripts
│ ├── gen-avro.sh # Script used to generate Java code for Avro
│ └── gen-thrift.sh # Script used to generate Java code for Thrift
└── thrift
└── *.thrift # Testing Thrift schema(s)
To avoid code generation during build time, Java code generated from testing Thrift schema and Avro IDL are also checked in.
When updating the testing Thrift schema and Avro IDL, please run gen-avro.sh and gen-thrift.sh accordingly to update generated Java code.
Prerequisites
Please ensure avro-tools and thrift are installed. You may install these two on Mac OS X via:
$ brew install thrift avro-tools