spark-instrumented-optimizer/resource-managers
Thomas Graves 3d2b8d8b13 [SPARK-30638][CORE] Add resources allocated to PluginContext
### What changes were proposed in this pull request?

Add the allocated resources to parameters to the PluginContext so that any plugins in driver or executor could use this information to initialize devices or use this information in a useful manner.

### Why are the changes needed?

To allow users to initialize/track devices once at the executor level before each task runs to use them.

### Does this PR introduce any user-facing change?

Yes to the people using the Executor/Driver plugin interface.

### How was this patch tested?

Unit tests and manually by writing a plugin that initialized GPU's using this interface.

Closes #27367 from tgravescs/pluginWithResources.

Lead-authored-by: Thomas Graves <tgraves@nvidia.com>
Co-authored-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@apache.org>
2020-01-31 08:25:32 -06:00
..
kubernetes [SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env 2020-01-24 12:00:30 -08:00
mesos [SPARK-30638][CORE] Add resources allocated to PluginContext 2020-01-31 08:25:32 -06:00
yarn [SPARK-29306][CORE] Stage Level Sched: Executors need to track what ResourceProfile they are created with 2020-01-17 08:15:25 -06:00