ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-26918][DOCS] All .md should have ASF license header

Browse source 
## What changes were proposed in this pull request?

Add AL2 license to metadata of all .md files.
This seemed to be the tidiest way as it will get ignored by .md renderers and other tools. Attempts to write them as markdown comments revealed that there is no such standard thing.

## How was this patch tested?

Doc build

Closes #24243 from srowen/SPARK-26918.

Authored-by: Sean Owen <sean.owen@databricks.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
Sean Owen 2019-03-30 19:49:45 -05:00
parent 88ea319871
commit 754f820035
91 changed files with 1380 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
R/CRAN_RELEASE.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# SparkR CRAN Release # SparkR CRAN Release
To release SparkR as a package to CRAN, we would use the `devtools` package. Please work with the To release SparkR as a package to CRAN, we would use the `devtools` package. Please work with the

18
R/DOCUMENTATION.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# SparkR Documentation # SparkR Documentation
SparkR documentation is generated by using in-source comments and annotated by using SparkR documentation is generated by using in-source comments and annotated by using

18
R/WINDOWS.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
## Building SparkR on Windows ## Building SparkR on Windows
To build SparkR on Windows, the following steps are required To build SparkR on Windows, the following steps are required

18
core/src/main/scala/org/apache/spark/deploy/security/README.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# Delegation Token Handling In Spark # Delegation Token Handling In Spark
This document aims to explain and demystify delegation tokens as they are used by Spark, since This document aims to explain and demystify delegation tokens as they are used by Spark, since

18
dev/appveyor-guide.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# AppVeyor Guides # AppVeyor Guides
Currently, SparkR on Windows is being tested with [AppVeyor](https://ci.appveyor.com). This page describes how to set up AppVeyor with Spark, how to run the build, check the status and stop the build via this tool. There is the documentation for AppVeyor [here](https://www.appveyor.com/docs). Please refer this for full details. Currently, SparkR on Windows is being tested with [AppVeyor](https://ci.appveyor.com). This page describes how to set up AppVeyor with Spark, how to run the build, check the status and stop the build via this tool. There is the documentation for AppVeyor [here](https://www.appveyor.com/docs). Please refer this for full details.

18
docs/README.md
View file

@ -1,3 +1,21 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
Welcome to the Spark documentation! Welcome to the Spark documentation!
This readme will walk you through navigating and building the Spark documentation, which is included This readme will walk you through navigating and building the Spark documentation, which is included

15
docs/api.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark API Documentation title: Spark API Documentation
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Here you can read API docs for Spark and its submodules. Here you can read API docs for Spark and its submodules.

15
docs/building-spark.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Building Spark title: Building Spark
redirect_from: "building-with-maven.html" redirect_from: "building-with-maven.html"
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

18
docs/cloud-integration.md
View file

@ -3,20 +3,22 @@ layout: global
displayTitle: Integration with Cloud Infrastructures displayTitle: Integration with Cloud Infrastructures
title: Integration with Cloud Infrastructures title: Integration with Cloud Infrastructures
description: Introduction to cloud storage support in Apache Spark SPARK_VERSION_SHORT description: Introduction to cloud storage support in Apache Spark SPARK_VERSION_SHORT
--- license: |
<!--- Licensed to the Apache Software Foundation (ASF) under one or more
Licensed under the Apache License, Version 2.0 (the "License"); contributor license agreements. See the NOTICE file distributed with
you may not use this file except in compliance with the License. this work for additional information regarding copyright ownership.
You may obtain a copy of the License at The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file. limitations under the License.
--> ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/cluster-overview.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Cluster Mode Overview title: Cluster Mode Overview
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
This document gives a short overview of how Spark runs on clusters, to make it easier to understand This document gives a short overview of how Spark runs on clusters, to make it easier to understand

15
docs/configuration.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: Spark Configuration displayTitle: Spark Configuration
title: Configuration title: Configuration
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/contributing-to-spark.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Contributing to Spark title: Contributing to Spark
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The Spark team welcomes all forms of contributions, including bug reports, documentation or patches. The Spark team welcomes all forms of contributions, including bug reports, documentation or patches.

15
docs/graphx-programming-guide.md
View file

@ -3,6 +3,21 @@ layout: global
displayTitle: GraphX Programming Guide displayTitle: GraphX Programming Guide
title: GraphX title: GraphX
description: GraphX graph processing library guide for Spark SPARK_VERSION_SHORT description: GraphX graph processing library guide for Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/hadoop-provided.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: Using Spark's "Hadoop Free" Build displayTitle: Using Spark's "Hadoop Free" Build
title: Using Spark's "Hadoop Free" Build title: Using Spark's "Hadoop Free" Build
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Spark uses Hadoop client libraries for HDFS and YARN. Starting in version Spark 1.4, the project packages "Hadoop free" builds that lets you more easily connect a single Spark binary to any Hadoop version. To use these builds, you need to modify `SPARK_DIST_CLASSPATH` to include Hadoop's package jars. The most convenient place to do this is by adding an entry in `conf/spark-env.sh`. Spark uses Hadoop client libraries for HDFS and YARN. Starting in version Spark 1.4, the project packages "Hadoop free" builds that lets you more easily connect a single Spark binary to any Hadoop version. To use these builds, you need to modify `SPARK_DIST_CLASSPATH` to include Hadoop's package jars. The most convenient place to do this is by adding an entry in `conf/spark-env.sh`.

15
docs/hardware-provisioning.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Hardware Provisioning title: Hardware Provisioning
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
A common question received by Spark developers is how to configure hardware for it. While the right A common question received by Spark developers is how to configure hardware for it. While the right

15
docs/index.md
View file

@ -3,6 +3,21 @@ layout: global
displayTitle: Spark Overview displayTitle: Spark Overview
title: Overview title: Overview
description: Apache Spark SPARK_VERSION_SHORT documentation homepage description: Apache Spark SPARK_VERSION_SHORT documentation homepage
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Apache Spark is a fast and general-purpose cluster computing system. Apache Spark is a fast and general-purpose cluster computing system.

15
docs/job-scheduling.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Job Scheduling title: Job Scheduling
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/ml-advanced.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Advanced topics title: Advanced topics
displayTitle: Advanced topics displayTitle: Advanced topics
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/ml-ann.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Multilayer perceptron classifier title: Multilayer perceptron classifier
displayTitle: Multilayer perceptron classifier displayTitle: Multilayer perceptron classifier
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
> This section has been moved into the > This section has been moved into the

15
docs/ml-classification-regression.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Classification and regression title: Classification and regression
displayTitle: Classification and regression displayTitle: Classification and regression
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---

15
docs/ml-clustering.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Clustering title: Clustering
displayTitle: Clustering displayTitle: Clustering
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
This page describes clustering algorithms in MLlib. This page describes clustering algorithms in MLlib.

15
docs/ml-collaborative-filtering.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Collaborative Filtering title: Collaborative Filtering
displayTitle: Collaborative Filtering displayTitle: Collaborative Filtering
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/ml-datasource.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Data sources title: Data sources
displayTitle: Data sources displayTitle: Data sources
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
In this section, we introduce how to use data source in ML to load data. In this section, we introduce how to use data source in ML to load data.

15
docs/ml-decision-tree.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Decision trees title: Decision trees
displayTitle: Decision trees displayTitle: Decision trees
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
> This section has been moved into the > This section has been moved into the

15
docs/ml-ensembles.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Tree ensemble methods title: Tree ensemble methods
displayTitle: Tree ensemble methods displayTitle: Tree ensemble methods
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
> This section has been moved into the > This section has been moved into the

15
docs/ml-features.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Extracting, transforming and selecting features title: Extracting, transforming and selecting features
displayTitle: Extracting, transforming and selecting features displayTitle: Extracting, transforming and selecting features
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
This section covers algorithms for working with features, roughly divided into these groups: This section covers algorithms for working with features, roughly divided into these groups:

15
docs/ml-frequent-pattern-mining.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Frequent Pattern Mining title: Frequent Pattern Mining
displayTitle: Frequent Pattern Mining displayTitle: Frequent Pattern Mining
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Mining frequent items, itemsets, subsequences, or other substructures is usually among the Mining frequent items, itemsets, subsequences, or other substructures is usually among the

15
docs/ml-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: "MLlib: Main Guide" title: "MLlib: Main Guide"
displayTitle: "Machine Learning Library (MLlib) Guide" displayTitle: "Machine Learning Library (MLlib) Guide"
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
MLlib is Spark's machine learning (ML) library. MLlib is Spark's machine learning (ML) library.

15
docs/ml-linear-methods.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Linear methods title: Linear methods
displayTitle: Linear methods displayTitle: Linear methods
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
> This section has been moved into the > This section has been moved into the

15
docs/ml-migration-guides.md
View file

@ -3,6 +3,21 @@ layout: global
title: Old Migration Guides - MLlib title: Old Migration Guides - MLlib
displayTitle: Old Migration Guides - MLlib displayTitle: Old Migration Guides - MLlib
description: MLlib migration guides from before Spark SPARK_VERSION_SHORT description: MLlib migration guides from before Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The migration guide for the current Spark version is kept on the [MLlib Guide main page](ml-guide.html#migration-guide). The migration guide for the current Spark version is kept on the [MLlib Guide main page](ml-guide.html#migration-guide).

15
docs/ml-pipeline.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: ML Pipelines title: ML Pipelines
displayTitle: ML Pipelines displayTitle: ML Pipelines
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
`\[ `\[

15
docs/ml-statistics.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Basic Statistics title: Basic Statistics
displayTitle: Basic Statistics displayTitle: Basic Statistics
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---

15
docs/ml-survival-regression.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Survival Regression title: Survival Regression
displayTitle: Survival Regression displayTitle: Survival Regression
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
> This section has been moved into the > This section has been moved into the

15
docs/ml-tuning.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: "ML Tuning" title: "ML Tuning"
displayTitle: "ML Tuning: model selection and hyperparameter tuning" displayTitle: "ML Tuning: model selection and hyperparameter tuning"
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
`\[ `\[

15
docs/mllib-classification-regression.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Classification and Regression - RDD-based API title: Classification and Regression - RDD-based API
displayTitle: Classification and Regression - RDD-based API displayTitle: Classification and Regression - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The `spark.mllib` package supports various methods for The `spark.mllib` package supports various methods for

15
docs/mllib-clustering.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Clustering - RDD-based API title: Clustering - RDD-based API
displayTitle: Clustering - RDD-based API displayTitle: Clustering - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
[Clustering](https://en.wikipedia.org/wiki/Cluster_analysis) is an unsupervised learning problem whereby we aim to group subsets [Clustering](https://en.wikipedia.org/wiki/Cluster_analysis) is an unsupervised learning problem whereby we aim to group subsets

15
docs/mllib-collaborative-filtering.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Collaborative Filtering - RDD-based API title: Collaborative Filtering - RDD-based API
displayTitle: Collaborative Filtering - RDD-based API displayTitle: Collaborative Filtering - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-data-types.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Data Types - RDD-based API title: Data Types - RDD-based API
displayTitle: Data Types - RDD-based API displayTitle: Data Types - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-decision-tree.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Decision Trees - RDD-based API title: Decision Trees - RDD-based API
displayTitle: Decision Trees - RDD-based API displayTitle: Decision Trees - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-dimensionality-reduction.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Dimensionality Reduction - RDD-based API title: Dimensionality Reduction - RDD-based API
displayTitle: Dimensionality Reduction - RDD-based API displayTitle: Dimensionality Reduction - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-ensembles.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Ensembles - RDD-based API title: Ensembles - RDD-based API
displayTitle: Ensembles - RDD-based API displayTitle: Ensembles - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-evaluation-metrics.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Evaluation Metrics - RDD-based API title: Evaluation Metrics - RDD-based API
displayTitle: Evaluation Metrics - RDD-based API displayTitle: Evaluation Metrics - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-feature-extraction.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Feature Extraction and Transformation - RDD-based API title: Feature Extraction and Transformation - RDD-based API
displayTitle: Feature Extraction and Transformation - RDD-based API displayTitle: Feature Extraction and Transformation - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-frequent-pattern-mining.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Frequent Pattern Mining - RDD-based API title: Frequent Pattern Mining - RDD-based API
displayTitle: Frequent Pattern Mining - RDD-based API displayTitle: Frequent Pattern Mining - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Mining frequent items, itemsets, subsequences, or other substructures is usually among the Mining frequent items, itemsets, subsequences, or other substructures is usually among the

15
docs/mllib-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: "MLlib: RDD-based API" title: "MLlib: RDD-based API"
displayTitle: "MLlib: RDD-based API" displayTitle: "MLlib: RDD-based API"
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
This page documents sections of the MLlib guide for the RDD-based API (the `spark.mllib` package). This page documents sections of the MLlib guide for the RDD-based API (the `spark.mllib` package).

15
docs/mllib-isotonic-regression.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Isotonic regression - RDD-based API title: Isotonic regression - RDD-based API
displayTitle: Regression - RDD-based API displayTitle: Regression - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
## Isotonic regression ## Isotonic regression

15
docs/mllib-linear-methods.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Linear Methods - RDD-based API title: Linear Methods - RDD-based API
displayTitle: Linear Methods - RDD-based API displayTitle: Linear Methods - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-migration-guides.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Old Migration Guides - MLlib title: Old Migration Guides - MLlib
displayTitle: Old Migration Guides - MLlib displayTitle: Old Migration Guides - MLlib
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The migration guide for the current Spark version is kept on the [MLlib Guide main page](ml-guide.html#migration-guide). The migration guide for the current Spark version is kept on the [MLlib Guide main page](ml-guide.html#migration-guide).

15
docs/mllib-naive-bayes.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Naive Bayes - RDD-based API title: Naive Bayes - RDD-based API
displayTitle: Naive Bayes - RDD-based API displayTitle: Naive Bayes - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
[Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) is a simple [Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) is a simple

15
docs/mllib-optimization.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Optimization - RDD-based API title: Optimization - RDD-based API
displayTitle: Optimization - RDD-based API displayTitle: Optimization - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-pmml-model-export.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: PMML model export - RDD-based API title: PMML model export - RDD-based API
displayTitle: PMML model export - RDD-based API displayTitle: PMML model export - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/mllib-statistics.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Basic Statistics - RDD-based API title: Basic Statistics - RDD-based API
displayTitle: Basic Statistics - RDD-based API displayTitle: Basic Statistics - RDD-based API
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/monitoring.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Monitoring and Instrumentation title: Monitoring and Instrumentation
description: Monitoring, metrics, and instrumentation guide for Spark SPARK_VERSION_SHORT description: Monitoring, metrics, and instrumentation guide for Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
There are several ways to monitor Spark applications: web UIs, metrics, and external instrumentation. There are several ways to monitor Spark applications: web UIs, metrics, and external instrumentation.

15
docs/programming-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Spark Programming Guide title: Spark Programming Guide
redirect: rdd-programming-guide.html redirect: rdd-programming-guide.html
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
This document has moved [here](rdd-programming-guide.html). This document has moved [here](rdd-programming-guide.html).

15
docs/quick-start.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Quick Start title: Quick Start
description: Quick start tutorial for Spark SPARK_VERSION_SHORT description: Quick start tutorial for Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/rdd-programming-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: RDD Programming Guide title: RDD Programming Guide
description: Spark SPARK_VERSION_SHORT programming guide in Java, Scala and Python description: Spark SPARK_VERSION_SHORT programming guide in Java, Scala and Python
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/running-on-kubernetes.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Running Spark on Kubernetes title: Running Spark on Kubernetes
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/running-on-mesos.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Running Spark on Mesos title: Running Spark on Mesos
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/running-on-yarn.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Running Spark on YARN title: Running Spark on YARN
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/security.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: Spark Security displayTitle: Spark Security
title: Security title: Security
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).
{:toc} {:toc}

15
docs/spark-standalone.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark Standalone Mode title: Spark Standalone Mode
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/sparkr.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: SparkR (R on Spark) displayTitle: SparkR (R on Spark)
title: SparkR (R on Spark) title: SparkR (R on Spark)
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/sql-data-sources-avro.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Apache Avro Data Source Guide title: Apache Avro Data Source Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/sql-data-sources-hive-tables.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Hive Tables title: Hive Tables
displayTitle: Hive Tables displayTitle: Hive Tables
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-data-sources-jdbc.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: JDBC To Other Databases title: JDBC To Other Databases
displayTitle: JDBC To Other Databases displayTitle: JDBC To Other Databases
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-data-sources-json.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: JSON Files title: JSON Files
displayTitle: JSON Files displayTitle: JSON Files
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
<div class="codetabs"> <div class="codetabs">

15
docs/sql-data-sources-load-save-functions.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Generic Load/Save Functions title: Generic Load/Save Functions
displayTitle: Generic Load/Save Functions displayTitle: Generic Load/Save Functions
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-data-sources-orc.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: ORC Files title: ORC Files
displayTitle: ORC Files displayTitle: ORC Files
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Since Spark 2.3, Spark supports a vectorized ORC reader with a new ORC file format for ORC files. Since Spark 2.3, Spark supports a vectorized ORC reader with a new ORC file format for ORC files.

15
docs/sql-data-sources-parquet.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Parquet Files title: Parquet Files
displayTitle: Parquet Files displayTitle: Parquet Files
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-data-sources-troubleshooting.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Troubleshooting title: Troubleshooting
displayTitle: Troubleshooting displayTitle: Troubleshooting
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java's DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs. * The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java's DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs.

15
docs/sql-data-sources.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Data Sources title: Data Sources
displayTitle: Data Sources displayTitle: Data Sources
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---

15
docs/sql-distributed-sql-engine.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Distributed SQL Engine title: Distributed SQL Engine
displayTitle: Distributed SQL Engine displayTitle: Distributed SQL Engine
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-getting-started.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Getting Started title: Getting Started
displayTitle: Getting Started displayTitle: Getting Started
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-keywords.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Spark SQL Keywords title: Spark SQL Keywords
displayTitle: Spark SQL Keywords displayTitle: Spark SQL Keywords
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
When `spark.sql.parser.ansi.enabled` is true, Spark SQL has two kinds of keywords: When `spark.sql.parser.ansi.enabled` is true, Spark SQL has two kinds of keywords:

15
docs/sql-migration-guide-hive-compatibility.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Compatibility with Apache Hive title: Compatibility with Apache Hive
displayTitle: Compatibility with Apache Hive displayTitle: Compatibility with Apache Hive
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-migration-guide-upgrade.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Spark SQL Upgrading Guide title: Spark SQL Upgrading Guide
displayTitle: Spark SQL Upgrading Guide displayTitle: Spark SQL Upgrading Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-migration-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Migration Guide title: Migration Guide
displayTitle: Migration Guide displayTitle: Migration Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* [Spark SQL Upgrading Guide](sql-migration-guide-upgrade.html) * [Spark SQL Upgrading Guide](sql-migration-guide-upgrade.html)

15
docs/sql-performance-tuning.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Performance Tuning title: Performance Tuning
displayTitle: Performance Tuning displayTitle: Performance Tuning
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-programming-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: Spark SQL, DataFrames and Datasets Guide displayTitle: Spark SQL, DataFrames and Datasets Guide
title: Spark SQL and DataFrames title: Spark SQL and DataFrames
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided

15
docs/sql-pyspark-pandas-with-arrow.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: PySpark Usage Guide for Pandas with Apache Arrow title: PySpark Usage Guide for Pandas with Apache Arrow
displayTitle: PySpark Usage Guide for Pandas with Apache Arrow displayTitle: PySpark Usage Guide for Pandas with Apache Arrow
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/sql-reference.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
title: Reference title: Reference
displayTitle: Reference displayTitle: Reference
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* Table of contents * Table of contents

15
docs/storage-openstack-swift.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Accessing OpenStack Swift from Spark title: Accessing OpenStack Swift from Spark
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the Spark's support for Hadoop InputFormat allows it to process data in OpenStack Swift using the

15
docs/streaming-custom-receivers.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark Streaming Custom Receivers title: Spark Streaming Custom Receivers
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Spark Streaming can receive streaming data from any arbitrary data source beyond Spark Streaming can receive streaming data from any arbitrary data source beyond

15
docs/streaming-kafka-0-10-integration.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) title: Spark Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher)
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The Spark Streaming integration for Kafka 0.10 provides simple parallelism, 1:1 correspondence between Kafka The Spark Streaming integration for Kafka 0.10 provides simple parallelism, 1:1 correspondence between Kafka

15
docs/streaming-kafka-integration.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark Streaming + Kafka Integration Guide title: Spark Streaming + Kafka Integration Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
[Apache Kafka](https://kafka.apache.org/) is publish-subscribe messaging rethought as a distributed, partitioned, [Apache Kafka](https://kafka.apache.org/) is publish-subscribe messaging rethought as a distributed, partitioned,

15
docs/streaming-kinesis-integration.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Spark Streaming + Kinesis Integration title: Spark Streaming + Kinesis Integration
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
[Amazon Kinesis](http://aws.amazon.com/kinesis/) is a fully managed service for real-time processing of streaming data at massive scale. [Amazon Kinesis](http://aws.amazon.com/kinesis/) is a fully managed service for real-time processing of streaming data at massive scale.
The Kinesis receiver creates an input DStream using the Kinesis Client Library (KCL) provided by Amazon under the Amazon Software License (ASL). The Kinesis receiver creates an input DStream using the Kinesis Client Library (KCL) provided by Amazon under the Amazon Software License (ASL).

15
docs/streaming-programming-guide.md
View file

@ -3,6 +3,21 @@ layout: global
displayTitle: Spark Streaming Programming Guide displayTitle: Spark Streaming Programming Guide
title: Spark Streaming title: Spark Streaming
description: Spark Streaming programming guide and tutorial for Spark SPARK_VERSION_SHORT description: Spark Streaming programming guide and tutorial for Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/structured-streaming-kafka-integration.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) title: Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher)
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
Structured Streaming integration for Kafka 0.10 to read data from and write data to Kafka. Structured Streaming integration for Kafka 0.10 to read data from and write data to Kafka.

15
docs/structured-streaming-programming-guide.md
View file

@ -2,6 +2,21 @@
layout: global layout: global
displayTitle: Structured Streaming Programming Guide displayTitle: Structured Streaming Programming Guide
title: Structured Streaming Programming Guide title: Structured Streaming Programming Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).

15
docs/submitting-applications.md
View file

@ -1,6 +1,21 @@
--- ---
layout: global layout: global
title: Submitting Applications title: Submitting Applications
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
The `spark-submit` script in Spark's `bin` directory is used to launch applications on a cluster. The `spark-submit` script in Spark's `bin` directory is used to launch applications on a cluster.

15
docs/tuning.md
View file

@ -3,6 +3,21 @@ layout: global
displayTitle: Tuning Spark displayTitle: Tuning Spark
title: Tuning title: Tuning
description: Tuning and performance optimization guide for Spark SPARK_VERSION_SHORT description: Tuning and performance optimization guide for Spark SPARK_VERSION_SHORT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--- ---
* This will become a table of contents (this text will be scraped). * This will become a table of contents (this text will be scraped).