236dfac676
Many users have reported being confused by the distinction between the `sql` and `hql` methods. Specifically, many users think that `sql(...)` cannot be used to read hive tables. In this PR I introduce a new configuration option `spark.sql.dialect` that picks which dialect with be used for parsing. For SQLContext this must be set to `sql`. In `HiveContext` it defaults to `hiveql` but can also be set to `sql`.
The `hql` and `hiveql` methods continue to act the same but are now marked as deprecated.
**This is a possibly breaking change for some users unless they set the dialect manually, though this is unlikely.**
For example: `hiveContex.sql("SELECT 1")` will now throw a parsing exception by default.
Author: Michael Armbrust <michael@databricks.com>
Closes #1746 from marmbrus/sqlLanguageConf and squashes the following commits:
ad375cc [Michael Armbrust] Merge remote-tracking branch 'apache/master' into sqlLanguageConf
20c43f8 [Michael Armbrust] override function instead of just setting the value
7e4ae93 [Michael Armbrust] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
|
||
|---|---|---|
| .. | ||
| blank_maven_build | ||
| blank_sbt_build | ||
| maven_app_core | ||
| sbt_app_core | ||
| sbt_app_ganglia | ||
| sbt_app_graphx | ||
| sbt_app_hive | ||
| sbt_app_kinesis | ||
| sbt_app_sql | ||
| sbt_app_streaming | ||
| .gitignore | ||
| audit_release.py | ||
| README.md | ||
Test Application Builds
This directory includes test applications which are built when auditing releases. You can run them locally by setting appropriate environment variables.
$ cd sbt_app_core
$ SCALA_VERSION=2.10.4 \
SPARK_VERSION=1.0.0-SNAPSHOT \
SPARK_RELEASE_REPOSITORY=file:///home/patrick/.ivy2/local \
sbt run