6fb22aa42d
### What changes were proposed in this pull request?
This PR is kind of a followup for SPARK-29641 and SPARK-28234. This PR proposes:
1.. Document the new `pyspark.resource` module introduced at 95aec091e4
, in PySpark API docs.
2.. Move classes into fewer and simpler modules
Before:
```
pyspark
├── resource
│ ├── executorrequests.py
│ │ ├── class ExecutorResourceRequest
│ │ └── class ExecutorResourceRequests
│ ├── taskrequests.py
│ │ ├── class TaskResourceRequest
│ │ └── class TaskResourceRequests
│ ├── resourceprofilebuilder.py
│ │ └── class ResourceProfileBuilder
│ ├── resourceprofile.py
│ │ └── class ResourceProfile
└── resourceinformation
└── class ResourceInformation
```
After:
```
pyspark
└── resource
├── requests.py
│ ├── class ExecutorResourceRequest
│ ├── class ExecutorResourceRequests
│ ├── class TaskResourceRequest
│ └── class TaskResourceRequests
├── profile.py
│ ├── class ResourceProfileBuilder
│ └── class ResourceProfile
└── information.py
└── class ResourceInformation
```
3.. Minor docstring fix e.g.:
```diff
- param name the name of the resource
- param addresses an array of strings describing the addresses of the resource
+ :param name: the name of the resource
+ :param addresses: an array of strings describing the addresses of the resource
+
+ .. versionadded:: 3.0.0
```
### Why are the changes needed?
To document APIs, and move Python modules to fewer and simpler modules.
### Does this PR introduce _any_ user-facing change?
No, the changes are in unreleased branches.
### How was this patch tested?
Manually tested via:
```bash
cd python
./run-tests --python-executables=python3 --modules=pyspark-core
./run-tests --python-executables=python3 --modules=pyspark-resource
```
Closes #28569 from HyukjinKwon/SPARK-28234-SPARK-29641-followup.
Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
54 lines
1.1 KiB
ReStructuredText
54 lines
1.1 KiB
ReStructuredText
.. pyspark documentation master file, created by
|
|
sphinx-quickstart on Thu Aug 28 15:17:47 2014.
|
|
You can adapt this file completely to your liking, but it should at least
|
|
contain the root `toctree` directive.
|
|
|
|
Welcome to Spark Python API Docs!
|
|
===================================
|
|
|
|
Contents:
|
|
|
|
.. toctree::
|
|
:maxdepth: 2
|
|
|
|
pyspark
|
|
pyspark.sql
|
|
pyspark.streaming
|
|
pyspark.ml
|
|
pyspark.mllib
|
|
pyspark.resource
|
|
|
|
|
|
Core classes:
|
|
---------------
|
|
|
|
:class:`pyspark.SparkContext`
|
|
|
|
Main entry point for Spark functionality.
|
|
|
|
:class:`pyspark.RDD`
|
|
|
|
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
|
|
|
|
:class:`pyspark.streaming.StreamingContext`
|
|
|
|
Main entry point for Spark Streaming functionality.
|
|
|
|
:class:`pyspark.streaming.DStream`
|
|
|
|
A Discretized Stream (DStream), the basic abstraction in Spark Streaming.
|
|
|
|
:class:`pyspark.sql.SparkSession`
|
|
|
|
Main entry point for DataFrame and SQL functionality.
|
|
|
|
:class:`pyspark.sql.DataFrame`
|
|
|
|
A distributed collection of data grouped into named columns.
|
|
|
|
|
|
Indices and tables
|
|
==================
|
|
|
|
* :ref:`search`
|