f08f6f4314
## What changes were proposed in this pull request? This PR fixes an exception during the compilation of generated code of `mapEntry`. This error occurs since the current code uses `key` type to store a `value` when `key` and `value` types are primitive type. ``` val mid0 = Literal.create(Map(1 -> 1.1, 2 -> 2.2), MapType(IntegerType, DoubleType)) checkEvaluation(MapEntries(mid0), Seq(r(1, 1.1), r(2, 2.2))) ``` ``` [info] Code generation of map_entries(keys: [1,2], values: [1.1,2.2]) failed: [info] java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: No applicable constructor/method found for actual parameters "int, double"; candidates are: "public void org.apache.spark.sql.catalyst.expressions.UnsafeRow.setInt(int, int)", "public void org.apache.spark.sql.catalyst.InternalRow.setInt(int, int)" [info] java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: No applicable constructor/method found for actual parameters "int, double"; candidates are: "public void org.apache.spark.sql.catalyst.expressions.UnsafeRow.setInt(int, int)", "public void org.apache.spark.sql.catalyst.InternalRow.setInt(int, int)" [info] at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306) [info] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293) [info] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) [info] at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) [info] at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410) [info] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380) [info] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) [info] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) [info] at com.google.common.cache.LocalCache.get(LocalCache.java:4000) [info] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) [info] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) [info] at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1290) ... ``` ## How was this patch tested? Added a new test to `CollectionExpressionsSuite` Closes #22033 from kiszk/SPARK-23935-followup. Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Takuya UESHIN <ueshin@databricks.com> |
||
---|---|---|
.. | ||
catalyst | ||
core | ||
hive | ||
hive-thriftserver | ||
create-docs.sh | ||
gen-sql-markdown.py | ||
mkdocs.yml | ||
README.md |
Spark SQL
This module provides support for executing relational queries expressed in either SQL or the DataFrame/Dataset API.
Spark SQL is broken up into four subprojects:
- Catalyst (sql/catalyst) - An implementation-agnostic framework for manipulating trees of relational operators and expressions.
- Execution (sql/core) - A query planner / execution engine for translating Catalyst's logical query plans into Spark RDDs. This component also includes a new public interface, SQLContext, that allows users to execute SQL or LINQ statements against existing RDDs and Parquet files.
- Hive Support (sql/hive) - Includes an extension of SQLContext called HiveContext that allows users to write queries using a subset of HiveQL and access data from a Hive Metastore using Hive SerDes. There are also wrappers that allow users to run queries that include Hive UDFs, UDAFs, and UDTFs.
- HiveServer and CLI support (sql/hive-thriftserver) - Includes support for the SQL CLI (bin/spark-sql) and a HiveServer2 (for JDBC/ODBC) compatible server.
Running sql/create-docs.sh
generates SQL documentation for built-in functions under sql/site
.