ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/sql/core/src/test
History
Davies Liu 8a94eb23d5 [SPARK-9069] [SPARK-9264] [SQL] remove unlimited precision support for DecimalType 
Romove Decimal.Unlimited (change to support precision up to 38, to match with Hive and other databases).

In order to keep backward source compatibility, Decimal.Unlimited is still there, but change to Decimal(38, 18).

If no precision and scale is provide, it's Decimal(10, 0) as before.

Author: Davies Liu <davies@databricks.com>

Closes #7605 from davies/decimal_unlimited and squashes the following commits:

aa3f115 [Davies Liu] fix tests and style
fb0d20d [Davies Liu] address comments
bfaae35 [Davies Liu] fix style
df93657 [Davies Liu] address comments and clean up
06727fd [Davies Liu] Merge branch 'master' of github.com:apache/spark into decimal_unlimited
4c28969 [Davies Liu] fix tests
8d783cc [Davies Liu] fix tests
788631c [Davies Liu] fix double with decimal in Union/except
1779bde [Davies Liu] fix scala style
c9c7c78 [Davies Liu] remove Decimal.Unlimited
2015-07-23 18:31:13 -07:00
..
avro [SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility 2015-07-08 15:51:01 -07:00
gen-java/org/apache/spark/sql/parquet/test/avro [SPARK-8959] [SQL] [HOTFIX] Removes parquet-thrift and libthrift dependencies 2015-07-09 17:09:16 -07:00
java/test/org/apache/spark/sql [SPARK-9069] [SPARK-9264] [SQL] remove unlimited precision support for DecimalType 2015-07-23 18:31:13 -07:00
resources [SPARK-8959] [SQL] [HOTFIX] Removes parquet-thrift and libthrift dependencies 2015-07-09 17:09:16 -07:00
scala/org/apache/spark/sql [SPARK-9069] [SPARK-9264] [SQL] remove unlimited precision support for DecimalType 2015-07-23 18:31:13 -07:00
scripts [SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility 2015-07-08 15:51:01 -07:00
thrift [SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility 2015-07-08 15:51:01 -07:00
README.md [SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility 2015-07-08 15:51:01 -07:00

README.md

Notes for Parquet compatibility tests

The following directories and files are used for Parquet compatibility tests:

.
├── README.md                   # This file
├── avro
│   ├── parquet-compat.avdl     # Testing Avro IDL
│   └── parquet-compat.avpr     # !! NO TOUCH !! Protocol file generated from parquet-compat.avdl
├── gen-java                    # !! NO TOUCH !! Generated Java code
├── scripts
│   └── gen-code.sh             # Script used to generate Java code for Thrift and Avro
└── thrift
    └── parquet-compat.thrift   # Testing Thrift schema

Generated Java code are used in the following test suites:

  • org.apache.spark.sql.parquet.ParquetAvroCompatibilitySuite
  • org.apache.spark.sql.parquet.ParquetThriftCompatibilitySuite

To avoid code generation during build time, Java code generated from testing Thrift schema and Avro IDL are also checked in.

When updating the testing Thrift schema and Avro IDL, please run gen-code.sh to update all the generated Java code.

Prerequisites

Please ensure avro-tools and thrift are installed. You may install these two on Mac OS X via:

$ brew install thrift avro-tools