cbad616d4c
## What changes were proposed in this pull request? In this PR, we implements a complete process of GPU-aware resources scheduling in Standalone. The whole process looks like: Worker sets up isolated resources when it starts up and registers to master along with its resources. And, Master picks up usable workers according to driver/executor's resource requirements to launch driver/executor on them. Then, Worker launches the driver/executor after preparing resources file, which is created under driver/executor's working directory, with specified resource addresses(told by master). When driver/executor finished, their resources could be recycled to worker. Finally, if a worker stops, it should always release its resources firstly. For the case of Workers and Drivers in **client** mode run on the same host, we introduce a config option named `spark.resources.coordinate.enable`(default true) to indicate whether Spark should coordinate resources for user. If `spark.resources.coordinate.enable=false`, user should be responsible for configuring different resources for Workers and Drivers when use resourcesFile or discovery script. If true, Spark would help user to assign different resources for Workers and Drivers. The solution for Spark to coordinate resources among Workers and Drivers is: Generally, use a shared file named *____allocated_resources____.json* to sync allocated resources info among Workers and Drivers on the same host. After a Worker or Driver found all resources using the configured resourcesFile and/or discovery script during launching, it should filter out available resources by excluding resources already allocated in *____allocated_resources____.json* and acquire resources from available resources according to its own requirement. After that, it should write its allocated resources along with its process id (pid) into *____allocated_resources____.json*. Pid (proposed by tgravescs) here used to check whether the allocated resources are still valid in case of Worker or Driver crashes and doesn't release resources properly. And when a Worker or Driver finished, normally, it would always clean up its own allocated resources in *____allocated_resources____.json*. Note that we'll always get a file lock before any access to file *____allocated_resources____.json* and release the lock finally. Futhermore, we appended resources info in `WorkerSchedulerStateResponse` to work around master change behaviour in HA mode. ## How was this patch tested? Added unit tests in WorkerSuite, MasterSuite, SparkContextSuite. Manually tested with client/cluster mode (e.g. multiple workers) in a single node Standalone. Closes #25047 from Ngone51/SPARK-27371. Authored-by: wuyi <ngone_5451@163.com> Signed-off-by: Thomas Graves <tgraves@apache.org>
102 lines
1.4 KiB
Plaintext
102 lines
1.4 KiB
Plaintext
*#*#
|
|
*.#*
|
|
*.iml
|
|
*.ipr
|
|
*.iws
|
|
*.pyc
|
|
*.pyo
|
|
*.swp
|
|
*~
|
|
.DS_Store
|
|
.cache
|
|
.classpath
|
|
.ensime
|
|
.ensime_cache/
|
|
.ensime_lucene
|
|
.generated-mima*
|
|
.idea/
|
|
.idea_modules/
|
|
.project
|
|
.pydevproject
|
|
.scala_dependencies
|
|
.settings
|
|
/lib/
|
|
R-unit-tests.log
|
|
R/unit-tests.out
|
|
R/cran-check.out
|
|
R/pkg/vignettes/sparkr-vignettes.html
|
|
R/pkg/tests/fulltests/Rplots.pdf
|
|
build/*.jar
|
|
build/apache-maven*
|
|
build/scala*
|
|
build/zinc*
|
|
cache
|
|
checkpoint
|
|
conf/*.cmd
|
|
conf/*.conf
|
|
conf/*.properties
|
|
conf/*.sh
|
|
conf/*.xml
|
|
conf/java-opts
|
|
conf/slaves
|
|
dependency-reduced-pom.xml
|
|
derby.log
|
|
dev/create-release/*final
|
|
dev/create-release/*txt
|
|
dev/pr-deps/
|
|
dist/
|
|
docs/_site
|
|
docs/api
|
|
sql/docs
|
|
sql/site
|
|
lib_managed/
|
|
lint-r-report.log
|
|
log/
|
|
logs/
|
|
out/
|
|
project/boot/
|
|
project/build/target/
|
|
project/plugins/lib_managed/
|
|
project/plugins/project/build.properties
|
|
project/plugins/src_managed/
|
|
project/plugins/target/
|
|
python/lib/pyspark.zip
|
|
python/.eggs/
|
|
python/deps
|
|
python/test_coverage/coverage_data
|
|
python/test_coverage/htmlcov
|
|
python/pyspark/python
|
|
reports/
|
|
scalastyle-on-compile.generated.xml
|
|
scalastyle-output.xml
|
|
scalastyle.txt
|
|
spark-*-bin-*.tgz
|
|
spark-resources/
|
|
spark-tests.log
|
|
src_managed/
|
|
streaming-tests.log
|
|
target/
|
|
unit-tests.log
|
|
work/
|
|
docs/.jekyll-metadata
|
|
|
|
# For Hive
|
|
TempStatsStore/
|
|
metastore/
|
|
metastore_db/
|
|
sql/hive-thriftserver/test_warehouses
|
|
warehouse/
|
|
spark-warehouse/
|
|
|
|
# For R session data
|
|
.RData
|
|
.RHistory
|
|
.Rhistory
|
|
*.Rproj
|
|
*.Rproj.*
|
|
|
|
.Rproj.user
|
|
|
|
# For SBT
|
|
.jvmopts
|