spark-instrumented-optimizer/python/pyspark/resource/__init__.py

31 lines
1.3 KiB
Python
Raw Normal View History

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
APIs to let users manipulate resource requirements.
"""
[SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes ### What changes were proposed in this pull request? This PR is kind of a followup for SPARK-29641 and SPARK-28234. This PR proposes: 1.. Document the new `pyspark.resource` module introduced at https://github.com/apache/spark/commit/95aec091e4d8a45e648ce84d32d912f585eeb151, in PySpark API docs. 2.. Move classes into fewer and simpler modules Before: ``` pyspark ├── resource │   ├── executorrequests.py │   │   ├── class ExecutorResourceRequest │   │   └── class ExecutorResourceRequests │   ├── taskrequests.py │   │   ├── class TaskResourceRequest │   │   └── class TaskResourceRequests │   ├── resourceprofilebuilder.py │   │   └── class ResourceProfileBuilder │   ├── resourceprofile.py │   │   └── class ResourceProfile └── resourceinformation    └── class ResourceInformation ``` After: ``` pyspark └── resource    ├── requests.py    │   ├── class ExecutorResourceRequest    │   ├── class ExecutorResourceRequests    │   ├── class TaskResourceRequest    │   └── class TaskResourceRequests    ├── profile.py    │   ├── class ResourceProfileBuilder    │   └── class ResourceProfile   └── information.py       └── class ResourceInformation ``` 3.. Minor docstring fix e.g.: ```diff - param name the name of the resource - param addresses an array of strings describing the addresses of the resource + :param name: the name of the resource + :param addresses: an array of strings describing the addresses of the resource + + .. versionadded:: 3.0.0 ``` ### Why are the changes needed? To document APIs, and move Python modules to fewer and simpler modules. ### Does this PR introduce _any_ user-facing change? No, the changes are in unreleased branches. ### How was this patch tested? Manually tested via: ```bash cd python ./run-tests --python-executables=python3 --modules=pyspark-core ./run-tests --python-executables=python3 --modules=pyspark-resource ``` Closes #28569 from HyukjinKwon/SPARK-28234-SPARK-29641-followup. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2020-05-19 20:09:37 -04:00
from pyspark.resource.information import ResourceInformation
from pyspark.resource.requests import TaskResourceRequest, TaskResourceRequests, \
ExecutorResourceRequest, ExecutorResourceRequests
from pyspark.resource.profile import ResourceProfile, ResourceProfileBuilder
__all__ = [
"TaskResourceRequest", "TaskResourceRequests", "ExecutorResourceRequest",
[SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes ### What changes were proposed in this pull request? This PR is kind of a followup for SPARK-29641 and SPARK-28234. This PR proposes: 1.. Document the new `pyspark.resource` module introduced at https://github.com/apache/spark/commit/95aec091e4d8a45e648ce84d32d912f585eeb151, in PySpark API docs. 2.. Move classes into fewer and simpler modules Before: ``` pyspark ├── resource │   ├── executorrequests.py │   │   ├── class ExecutorResourceRequest │   │   └── class ExecutorResourceRequests │   ├── taskrequests.py │   │   ├── class TaskResourceRequest │   │   └── class TaskResourceRequests │   ├── resourceprofilebuilder.py │   │   └── class ResourceProfileBuilder │   ├── resourceprofile.py │   │   └── class ResourceProfile └── resourceinformation    └── class ResourceInformation ``` After: ``` pyspark └── resource    ├── requests.py    │   ├── class ExecutorResourceRequest    │   ├── class ExecutorResourceRequests    │   ├── class TaskResourceRequest    │   └── class TaskResourceRequests    ├── profile.py    │   ├── class ResourceProfileBuilder    │   └── class ResourceProfile   └── information.py       └── class ResourceInformation ``` 3.. Minor docstring fix e.g.: ```diff - param name the name of the resource - param addresses an array of strings describing the addresses of the resource + :param name: the name of the resource + :param addresses: an array of strings describing the addresses of the resource + + .. versionadded:: 3.0.0 ``` ### Why are the changes needed? To document APIs, and move Python modules to fewer and simpler modules. ### Does this PR introduce _any_ user-facing change? No, the changes are in unreleased branches. ### How was this patch tested? Manually tested via: ```bash cd python ./run-tests --python-executables=python3 --modules=pyspark-core ./run-tests --python-executables=python3 --modules=pyspark-resource ``` Closes #28569 from HyukjinKwon/SPARK-28234-SPARK-29641-followup. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2020-05-19 20:09:37 -04:00
"ExecutorResourceRequests", "ResourceProfile", "ResourceInformation",
"ResourceProfileBuilder",
]