spark-instrumented-optimizer/docs/_data/menu-sql.yaml

280 lines
12 KiB
YAML
Raw Normal View History

- text: Getting Started
url: sql-getting-started.html
subitems:
- text: "Starting Point: SparkSession"
url: sql-getting-started.html#starting-point-sparksession
- text: Creating DataFrames
url: sql-getting-started.html#creating-dataframes
- text: Untyped Dataset Operations (DataFrame operations)
url: sql-getting-started.html#untyped-dataset-operations-aka-dataframe-operations
- text: Running SQL Queries Programmatically
url: sql-getting-started.html#running-sql-queries-programmatically
- text: Global Temporary View
url: sql-getting-started.html#global-temporary-view
- text: Creating Datasets
url: sql-getting-started.html#creating-datasets
- text: Interoperating with RDDs
url: sql-getting-started.html#interoperating-with-rdds
- text: Scalar Functions
url: sql-getting-started.html#scalar-functions
- text: Aggregations
url: sql-getting-started.html#aggregations
- text: Data Sources
url: sql-data-sources.html
subitems:
- text: "Generic Load/Save Functions"
url: sql-data-sources-load-save-functions.html
- text: "Generic File Source Options"
url: sql-data-sources-generic-options.html
- text: Parquet Files
url: sql-data-sources-parquet.html
- text: ORC Files
url: sql-data-sources-orc.html
- text: JSON Files
url: sql-data-sources-json.html
- text: CSV Files
url: sql-data-sources-csv.html
- text: Text Files
url: sql-data-sources-text.html
- text: Hive Tables
url: sql-data-sources-hive-tables.html
- text: JDBC To Other Databases
url: sql-data-sources-jdbc.html
- text: Avro Files
url: sql-data-sources-avro.html
- text: Whole Binary Files
url: sql-data-sources-binaryFile.html
- text: Troubleshooting
url: sql-data-sources-troubleshooting.html
- text: Performance Tuning
url: sql-performance-tuning.html
subitems:
- text: Caching Data In Memory
url: sql-performance-tuning.html#caching-data-in-memory
- text: Other Configuration Options
url: sql-performance-tuning.html#other-configuration-options
- text: Join Strategy Hints for SQL Queries
url: sql-performance-tuning.html#join-strategy-hints-for-sql-queries
- text: Coalesce Hints for SQL Queries
url: sql-performance-tuning.html#coalesce-hints-for-sql-queries
- text: Adaptive Query Execution
url: sql-performance-tuning.html#adaptive-query-execution
- text: Distributed SQL Engine
url: sql-distributed-sql-engine.html
subitems:
- text: "Running the Thrift JDBC/ODBC server"
url: sql-distributed-sql-engine.html#running-the-thrift-jdbcodbc-server
- text: Running the Spark SQL CLI
url: sql-distributed-sql-engine.html#running-the-spark-sql-cli
- text: PySpark Usage Guide for Pandas with Apache Arrow
url: sql-pyspark-pandas-with-arrow.html
- text: Migration Guide
[SPARK-29052][DOCS][ML][PYTHON][CORE][R][SQL][SS] Create a Migration Guide tap in Spark documentation ### What changes were proposed in this pull request? Currently, there is no migration section for PySpark, SparkCore and Structured Streaming. It is difficult for users to know what to do when they upgrade. This PR proposes to create create a "Migration Guide" tap at Spark documentation. ![Screen Shot 2019-09-11 at 7 02 05 PM](https://user-images.githubusercontent.com/6477701/64688126-ad712f80-d4c6-11e9-8672-9a2c56c05bf8.png) ![Screen Shot 2019-09-11 at 7 27 15 PM](https://user-images.githubusercontent.com/6477701/64689915-389ff480-d4ca-11e9-8c54-7f46095d0d23.png) This page will contain migration guides for Spark SQL, PySpark, SparkR, MLlib, Structured Streaming and Core. Basically it is a refactoring. There are some new information added, which I will leave a comment inlined for easier review. 1. **MLlib** Merge [ml-guide.html#migration-guide](https://spark.apache.org/docs/latest/ml-guide.html#migration-guide) and [ml-migration-guides.html](https://spark.apache.org/docs/latest/ml-migration-guides.html) ``` 'docs/ml-guide.md' ↓ Merge new/old migration guides 'docs/ml-migration-guide.md' ``` 2. **PySpark** Extract PySpark specific items from https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html ``` 'docs/sql-migration-guide-upgrade.md' ↓ Extract PySpark specific items 'docs/pyspark-migration-guide.md' ``` 3. **SparkR** Move [sparkr.html#migration-guide](https://spark.apache.org/docs/latest/sparkr.html#migration-guide) into a separate file, and extract from [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) ``` 'docs/sparkr.md' 'docs/sql-migration-guide-upgrade.md' Move migration guide section ↘ ↙ Extract SparkR specific items docs/sparkr-migration-guide.md ``` 4. **Core** Newly created at `'docs/core-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 5. **Structured Streaming** Newly created at `'docs/ss-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 6. **SQL** Merged [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) and [sql-migration-guide-hive-compatibility.html](https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) ``` 'docs/sql-migration-guide-hive-compatibility.md' 'docs/sql-migration-guide-upgrade.md' Move Hive compatibility section ↘ ↙ Left over after filtering PySpark and SparkR items 'docs/sql-migration-guide.md' ``` ### Why are the changes needed? In order for users in production to effectively migrate to higher versions, and detect behaviour or breaking changes before upgrading and/or migrating. ### Does this PR introduce any user-facing change? Yes, this changes Spark's documentation at https://spark.apache.org/docs/latest/index.html. ### How was this patch tested? Manually build the doc. This can be verified as below: ```bash cd docs SKIP_API=1 jekyll build open _site/index.html ``` Closes #25757 from HyukjinKwon/migration-doc. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2019-09-15 14:17:30 -04:00
url: sql-migration-old.html
- text: SQL Reference
url: sql-ref.html
subitems:
- text: ANSI Compliance
url: sql-ref-ansi-compliance.html
subitems:
- text: Arithmetic Operations
url: sql-ref-ansi-compliance.html#arithmetic-operations
- text: Type Conversion
url: sql-ref-ansi-compliance.html#type-conversion
- text: SQL Keywords
url: sql-ref-ansi-compliance.html#sql-keywords
[SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref ### What changes were proposed in this pull request? Fix a few issues in SQL Reference ### Why are the changes needed? To make SQL Reference look better ### Does this PR introduce _any_ user-facing change? Yes. before: <img width="189" alt="Screen Shot 2020-05-21 at 11 41 34 PM" src="https://user-images.githubusercontent.com/13592258/82639052-d0f38a80-9bbc-11ea-81a4-22def4ca5cc0.png"> after: <img width="195" alt="Screen Shot 2020-05-21 at 11 41 17 PM" src="https://user-images.githubusercontent.com/13592258/82639063-d5b83e80-9bbc-11ea-84d1-8361e6bee949.png"> before: <img width="763" alt="Screen Shot 2020-05-21 at 11 45 22 PM" src="https://user-images.githubusercontent.com/13592258/82639252-3e9fb680-9bbd-11ea-863c-e6a6c2f83a06.png"> after: <img width="724" alt="Screen Shot 2020-05-21 at 11 45 02 PM" src="https://user-images.githubusercontent.com/13592258/82639265-42cbd400-9bbd-11ea-8df2-fc5c255b84d3.png"> before: <img width="437" alt="Screen Shot 2020-05-21 at 11 41 57 PM" src="https://user-images.githubusercontent.com/13592258/82639072-db158900-9bbc-11ea-9963-731881cda4fd.png"> after <img width="347" alt="Screen Shot 2020-05-21 at 11 42 26 PM" src="https://user-images.githubusercontent.com/13592258/82639082-dfda3d00-9bbc-11ea-9bd2-f922cc91f175.png"> ### How was this patch tested? Manually build and check Closes #28608 from huaxingao/doc_fix. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
2020-05-22 19:43:16 -04:00
- text: Data Types
url: sql-ref-datatypes.html
- text: Datetime Pattern
url: sql-ref-datetime-pattern.html
- text: Functions
url: sql-ref-functions.html
subitems:
- text: Built-in Functions
url: sql-ref-functions-builtin.html
- text: Scalar UDFs (User-Defined Functions)
url: sql-ref-functions-udf-scalar.html
- text: UDAFs (User-Defined Aggregate Functions)
url: sql-ref-functions-udf-aggregate.html
- text: Integration with Hive UDFs/UDAFs/UDTFs
url: sql-ref-functions-udf-hive.html
- text: Identifiers
url: sql-ref-identifier.html
- text: Literals
url: sql-ref-literals.html
- text: Null Semantics
url: sql-ref-null-semantics.html
- text: SQL Syntax
url: sql-ref-syntax.html
subitems:
- text: Data Definition Statements
url: sql-ref-syntax-ddl.html
subitems:
- text: ALTER DATABASE
url: sql-ref-syntax-ddl-alter-database.html
- text: ALTER TABLE
url: sql-ref-syntax-ddl-alter-table.html
- text: ALTER VIEW
url: sql-ref-syntax-ddl-alter-view.html
- text: CREATE DATABASE
url: sql-ref-syntax-ddl-create-database.html
- text: CREATE FUNCTION
url: sql-ref-syntax-ddl-create-function.html
- text: CREATE TABLE
url: sql-ref-syntax-ddl-create-table.html
- text: CREATE VIEW
url: sql-ref-syntax-ddl-create-view.html
- text: DROP DATABASE
url: sql-ref-syntax-ddl-drop-database.html
- text: DROP FUNCTION
url: sql-ref-syntax-ddl-drop-function.html
- text: DROP TABLE
url: sql-ref-syntax-ddl-drop-table.html
- text: DROP VIEW
url: sql-ref-syntax-ddl-drop-view.html
- text: TRUNCATE TABLE
url: sql-ref-syntax-ddl-truncate-table.html
- text: REPAIR TABLE
url: sql-ref-syntax-ddl-repair-table.html
- text: USE DATABASE
[SPARK-32171][SQL][DOCS] Change file locations for use db and refresh table ### What changes were proposed in this pull request? docs/sql-ref-syntax-qry-select-usedb.md -> docs/sql-ref-syntax-ddl-usedb.md docs/sql-ref-syntax-aux-refresh-table.md -> docs/sql-ref-syntax-aux-cache-refresh-table.md ### Why are the changes needed? usedb belongs to DDL. Its location should be consistent with other DDL commands file locations similar reason for refresh table ### Does this PR introduce _any_ user-facing change? before change, when clicking USE DATABASE, the side bar menu shows select commands <img width="1200" alt="Screen Shot 2020-07-04 at 9 05 35 AM" src="https://user-images.githubusercontent.com/13592258/86516696-b45f8a80-bdd7-11ea-8dba-3a5cca22aad3.png"> after change, when clicking USE DATABASE, the side bar menu shows DDL commands <img width="1120" alt="Screen Shot 2020-07-04 at 9 06 06 AM" src="https://user-images.githubusercontent.com/13592258/86516703-bf1a1f80-bdd7-11ea-8a90-ae7eaaafd44c.png"> before change, when clicking refresh table, the side bar menu shows Auxiliary statements <img width="1200" alt="Screen Shot 2020-07-04 at 9 30 40 AM" src="https://user-images.githubusercontent.com/13592258/86516877-3d2af600-bdd9-11ea-9568-0a6f156f57da.png"> after change, when clicking refresh table, the side bar menu shows Cache statements <img width="1199" alt="Screen Shot 2020-07-04 at 9 35 21 AM" src="https://user-images.githubusercontent.com/13592258/86516937-b4f92080-bdd9-11ea-8ad1-5f5a7f58d76b.png"> ### How was this patch tested? Manually build and check Closes #28995 from huaxingao/docs_fix. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Huaxin Gao <huaxing@us.ibm.com>
2020-07-04 22:01:07 -04:00
url: sql-ref-syntax-ddl-usedb.html
- text: Data Manipulation Statements
url: sql-ref-syntax-dml.html
subitems:
- text: INSERT
url: sql-ref-syntax-dml-insert.html
- text: LOAD
url: sql-ref-syntax-dml-load.html
- text: Data Retrieval(Queries)
url: sql-ref-syntax-qry.html
subitems:
- text: SELECT
url: sql-ref-syntax-qry-select.html
subitems:
- text: WHERE Clause
url: sql-ref-syntax-qry-select-where.html
- text: GROUP BY Clause
url: sql-ref-syntax-qry-select-groupby.html
- text: HAVING Clause
url: sql-ref-syntax-qry-select-having.html
- text: ORDER BY Clause
url: sql-ref-syntax-qry-select-orderby.html
- text: SORT BY Clause
url: sql-ref-syntax-qry-select-sortby.html
- text: CLUSTER BY Clause
url: sql-ref-syntax-qry-select-clusterby.html
- text: DISTRIBUTE BY Clause
url: sql-ref-syntax-qry-select-distribute-by.html
- text: LIMIT Clause
url: sql-ref-syntax-qry-select-limit.html
- text: Common Table Expression
url: sql-ref-syntax-qry-select-cte.html
- text: Hints
url: sql-ref-syntax-qry-select-hints.html
- text: Inline Table
url: sql-ref-syntax-qry-select-inline-table.html
- text: File
url: sql-ref-syntax-qry-select-file.html
- text: JOIN
url: sql-ref-syntax-qry-select-join.html
- text: LIKE Predicate
url: sql-ref-syntax-qry-select-like.html
- text: Set Operators
url: sql-ref-syntax-qry-select-setops.html
- text: TABLESAMPLE
url: sql-ref-syntax-qry-select-sampling.html
[SPARK-31419][SQL][DOCS] Document Table-valued Function and Inline Table ### What changes were proposed in this pull request? Document Table-valued Function and Inline Table ### Why are the changes needed? To make SQL Reference complete ### Does this PR introduce any user-facing change? Yes <img width="1050" alt="Screen Shot 2020-04-11 at 5 34 25 PM" src="https://user-images.githubusercontent.com/13592258/79057852-cedff880-7c1a-11ea-9e1e-7882594ab573.png"> <img width="1050" alt="Screen Shot 2020-04-11 at 5 34 46 PM" src="https://user-images.githubusercontent.com/13592258/79057854-d4d5d980-7c1a-11ea-94cc-92ef1121fa43.png"> <img width="1050" alt="Screen Shot 2020-04-10 at 7 36 00 PM" src="https://user-images.githubusercontent.com/13592258/79033391-c2986480-7b62-11ea-9d0a-6c60de823256.png"> <img width="1051" alt="Screen Shot 2020-04-10 at 7 36 21 PM" src="https://user-images.githubusercontent.com/13592258/79033392-c5935500-7b62-11ea-88d4-e7d7812a7add.png"> <img width="1051" alt="Screen Shot 2020-04-11 at 5 09 48 PM" src="https://user-images.githubusercontent.com/13592258/79057555-6ba09700-7c17-11ea-9683-16bbde63a529.png"> Also, linked the newly added pages to select statement <img width="1050" alt="Screen Shot 2020-04-10 at 3 27 59 PM" src="https://user-images.githubusercontent.com/13592258/79027245-5147ba00-7b40-11ea-9b10-527fd9639958.png"> ### How was this patch tested? Manually build and check Closes #28185 from huaxingao/tvf. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Sean Owen <srowen@gmail.com>
2020-04-13 00:39:27 -04:00
- text: Table-valued Function
url: sql-ref-syntax-qry-select-tvf.html
- text: Window Function
url: sql-ref-syntax-qry-select-window.html
- text: CASE Clause
url: sql-ref-syntax-qry-select-case.html
- text: LATERAL VIEW Clause
url: sql-ref-syntax-qry-select-lateral-view.html
- text: PIVOT Clause
url: sql-ref-syntax-qry-select-pivot.html
- text: TRANSFORM Clause
url: sql-ref-syntax-qry-select-transform.html
- text: EXPLAIN
url: sql-ref-syntax-qry-explain.html
- text: Auxiliary Statements
url: sql-ref-syntax-aux.html
subitems:
- text: ANALYZE
url: sql-ref-syntax-aux-analyze.html
subitems:
- text: ANALYZE TABLE
url: sql-ref-syntax-aux-analyze-table.html
- text: ANALYZE TABLES
url: sql-ref-syntax-aux-analyze-tables.html
- text: CACHE
url: sql-ref-syntax-aux-cache.html
subitems:
- text: CACHE TABLE
url: sql-ref-syntax-aux-cache-cache-table.html
- text: UNCACHE TABLE
url: sql-ref-syntax-aux-cache-uncache-table.html
- text: CLEAR CACHE
url: sql-ref-syntax-aux-cache-clear-cache.html
- text: REFRESH TABLE
[SPARK-32171][SQL][DOCS] Change file locations for use db and refresh table ### What changes were proposed in this pull request? docs/sql-ref-syntax-qry-select-usedb.md -> docs/sql-ref-syntax-ddl-usedb.md docs/sql-ref-syntax-aux-refresh-table.md -> docs/sql-ref-syntax-aux-cache-refresh-table.md ### Why are the changes needed? usedb belongs to DDL. Its location should be consistent with other DDL commands file locations similar reason for refresh table ### Does this PR introduce _any_ user-facing change? before change, when clicking USE DATABASE, the side bar menu shows select commands <img width="1200" alt="Screen Shot 2020-07-04 at 9 05 35 AM" src="https://user-images.githubusercontent.com/13592258/86516696-b45f8a80-bdd7-11ea-8dba-3a5cca22aad3.png"> after change, when clicking USE DATABASE, the side bar menu shows DDL commands <img width="1120" alt="Screen Shot 2020-07-04 at 9 06 06 AM" src="https://user-images.githubusercontent.com/13592258/86516703-bf1a1f80-bdd7-11ea-8a90-ae7eaaafd44c.png"> before change, when clicking refresh table, the side bar menu shows Auxiliary statements <img width="1200" alt="Screen Shot 2020-07-04 at 9 30 40 AM" src="https://user-images.githubusercontent.com/13592258/86516877-3d2af600-bdd9-11ea-9568-0a6f156f57da.png"> after change, when clicking refresh table, the side bar menu shows Cache statements <img width="1199" alt="Screen Shot 2020-07-04 at 9 35 21 AM" src="https://user-images.githubusercontent.com/13592258/86516937-b4f92080-bdd9-11ea-8ad1-5f5a7f58d76b.png"> ### How was this patch tested? Manually build and check Closes #28995 from huaxingao/docs_fix. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Huaxin Gao <huaxing@us.ibm.com>
2020-07-04 22:01:07 -04:00
url: sql-ref-syntax-aux-cache-refresh-table.html
- text: REFRESH FUNCTION
url: sql-ref-syntax-aux-cache-refresh-function.html
- text: REFRESH
url: sql-ref-syntax-aux-cache-refresh.html
- text: DESCRIBE
url: sql-ref-syntax-aux-describe.html
subitems:
- text: DESCRIBE DATABASE
url: sql-ref-syntax-aux-describe-database.html
- text: DESCRIBE TABLE
url: sql-ref-syntax-aux-describe-table.html
- text: DESCRIBE FUNCTION
url: sql-ref-syntax-aux-describe-function.html
- text: DESCRIBE QUERY
url: sql-ref-syntax-aux-describe-query.html
- text: SHOW
url: sql-ref-syntax-aux-show.html
subitems:
- text: SHOW COLUMNS
url: sql-ref-syntax-aux-show-columns.html
- text: SHOW CREATE TABLE
url: sql-ref-syntax-aux-show-create-table.html
- text: SHOW DATABASES
url: sql-ref-syntax-aux-show-databases.html
- text: SHOW FUNCTIONS
url: sql-ref-syntax-aux-show-functions.html
- text: SHOW PARTITIONS
url: sql-ref-syntax-aux-show-partitions.html
- text: SHOW TABLE
url: sql-ref-syntax-aux-show-table.html
- text: SHOW TABLES
url: sql-ref-syntax-aux-show-tables.html
- text: SHOW TBLPROPERTIES
url: sql-ref-syntax-aux-show-tblproperties.html
- text: SHOW VIEWS
url: sql-ref-syntax-aux-show-views.html
- text: CONFIGURATION MANAGEMENT
url: sql-ref-syntax-aux-conf-mgmt.html
subitems:
- text: SET
url: sql-ref-syntax-aux-conf-mgmt-set.html
- text: RESET
url: sql-ref-syntax-aux-conf-mgmt-reset.html
- text: SET TIME ZONE
url: sql-ref-syntax-aux-conf-mgmt-set-timezone.html
- text: RESOURCE MANAGEMENT
url: sql-ref-syntax-aux-resource-mgmt.html
subitems:
- text: ADD FILE
url: sql-ref-syntax-aux-resource-mgmt-add-file.html
- text: ADD JAR
url: sql-ref-syntax-aux-resource-mgmt-add-jar.html
- text: ADD ARCHIVE
url: sql-ref-syntax-aux-resource-mgmt-add-archive.html
- text: LIST FILE
url: sql-ref-syntax-aux-resource-mgmt-list-file.html
- text: LIST JAR
url: sql-ref-syntax-aux-resource-mgmt-list-jar.html
- text: LIST ARCHIVE
url: sql-ref-syntax-aux-resource-mgmt-list-archive.html