spark-instrumented-optimizer/python/pyspark/resource/information.py

46 lines
1.5 KiB
Python
Raw Normal View History

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
class ResourceInformation(object):
"""
.. note:: Evolving
Class to hold information about a type of Resource. A resource could be a GPU, FPGA, etc.
The array of addresses are resource specific and its up to the user to interpret the address.
One example is GPUs, where the addresses would be the indices of the GPUs
[SPARK-31748][PYTHON] Document resource module in PySpark doc and rename/move classes ### What changes were proposed in this pull request? This PR is kind of a followup for SPARK-29641 and SPARK-28234. This PR proposes: 1.. Document the new `pyspark.resource` module introduced at https://github.com/apache/spark/commit/95aec091e4d8a45e648ce84d32d912f585eeb151, in PySpark API docs. 2.. Move classes into fewer and simpler modules Before: ``` pyspark ├── resource │   ├── executorrequests.py │   │   ├── class ExecutorResourceRequest │   │   └── class ExecutorResourceRequests │   ├── taskrequests.py │   │   ├── class TaskResourceRequest │   │   └── class TaskResourceRequests │   ├── resourceprofilebuilder.py │   │   └── class ResourceProfileBuilder │   ├── resourceprofile.py │   │   └── class ResourceProfile └── resourceinformation    └── class ResourceInformation ``` After: ``` pyspark └── resource    ├── requests.py    │   ├── class ExecutorResourceRequest    │   ├── class ExecutorResourceRequests    │   ├── class TaskResourceRequest    │   └── class TaskResourceRequests    ├── profile.py    │   ├── class ResourceProfileBuilder    │   └── class ResourceProfile   └── information.py       └── class ResourceInformation ``` 3.. Minor docstring fix e.g.: ```diff - param name the name of the resource - param addresses an array of strings describing the addresses of the resource + :param name: the name of the resource + :param addresses: an array of strings describing the addresses of the resource + + .. versionadded:: 3.0.0 ``` ### Why are the changes needed? To document APIs, and move Python modules to fewer and simpler modules. ### Does this PR introduce _any_ user-facing change? No, the changes are in unreleased branches. ### How was this patch tested? Manually tested via: ```bash cd python ./run-tests --python-executables=python3 --modules=pyspark-core ./run-tests --python-executables=python3 --modules=pyspark-resource ``` Closes #28569 from HyukjinKwon/SPARK-28234-SPARK-29641-followup. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2020-05-19 20:09:37 -04:00
:param name: the name of the resource
:param addresses: an array of strings describing the addresses of the resource
.. versionadded:: 3.0.0
"""
def __init__(self, name, addresses):
self._name = name
self._addresses = addresses
@property
def name(self):
return self._name
@property
def addresses(self):
return self._addresses