6fb22aa42d
### What changes were proposed in this pull request?
This PR is kind of a followup for SPARK-29641 and SPARK-28234. This PR proposes:
1.. Document the new `pyspark.resource` module introduced at 95aec091e4
, in PySpark API docs.
2.. Move classes into fewer and simpler modules
Before:
```
pyspark
├── resource
│ ├── executorrequests.py
│ │ ├── class ExecutorResourceRequest
│ │ └── class ExecutorResourceRequests
│ ├── taskrequests.py
│ │ ├── class TaskResourceRequest
│ │ └── class TaskResourceRequests
│ ├── resourceprofilebuilder.py
│ │ └── class ResourceProfileBuilder
│ ├── resourceprofile.py
│ │ └── class ResourceProfile
└── resourceinformation
└── class ResourceInformation
```
After:
```
pyspark
└── resource
├── requests.py
│ ├── class ExecutorResourceRequest
│ ├── class ExecutorResourceRequests
│ ├── class TaskResourceRequest
│ └── class TaskResourceRequests
├── profile.py
│ ├── class ResourceProfileBuilder
│ └── class ResourceProfile
└── information.py
└── class ResourceInformation
```
3.. Minor docstring fix e.g.:
```diff
- param name the name of the resource
- param addresses an array of strings describing the addresses of the resource
+ :param name: the name of the resource
+ :param addresses: an array of strings describing the addresses of the resource
+
+ .. versionadded:: 3.0.0
```
### Why are the changes needed?
To document APIs, and move Python modules to fewer and simpler modules.
### Does this PR introduce _any_ user-facing change?
No, the changes are in unreleased branches.
### How was this patch tested?
Manually tested via:
```bash
cd python
./run-tests --python-executables=python3 --modules=pyspark-core
./run-tests --python-executables=python3 --modules=pyspark-resource
```
Closes #28569 from HyukjinKwon/SPARK-28234-SPARK-29641-followup.
Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
38 lines
832 B
ReStructuredText
38 lines
832 B
ReStructuredText
pyspark.sql module
|
|
==================
|
|
|
|
Module Contents
|
|
---------------
|
|
|
|
.. automodule:: pyspark.sql
|
|
:members:
|
|
:undoc-members:
|
|
:inherited-members:
|
|
:exclude-members: builder
|
|
.. We need `exclude-members` to prevent default description generations
|
|
as a workaround for old Sphinx (< 1.6.6).
|
|
|
|
pyspark.sql.types module
|
|
------------------------
|
|
.. automodule:: pyspark.sql.types
|
|
:members:
|
|
:undoc-members:
|
|
|
|
pyspark.sql.functions module
|
|
----------------------------
|
|
.. automodule:: pyspark.sql.functions
|
|
:members:
|
|
:undoc-members:
|
|
|
|
pyspark.sql.avro.functions module
|
|
---------------------------------
|
|
.. automodule:: pyspark.sql.avro.functions
|
|
:members:
|
|
:undoc-members:
|
|
|
|
pyspark.sql.streaming module
|
|
----------------------------
|
|
.. automodule:: pyspark.sql.streaming
|
|
:members:
|
|
:undoc-members:
|