[SPARK-25073][YARN] AM and Executor Memory validation message is not proper while submitting spark yarn application
**## What changes were proposed in this pull request?** When the yarn.nodemanager.resource.memory-mb or yarn.scheduler.maximum-allocation-mb memory assignment is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb even though in message it shows the memory value of yarn.nodemanager.resource.memory-mb parameter,As the error Message is bit misleading to the user we can modify the same, We can keep the error message same as executor memory validation message. Defintion of **yarn.nodemanager.resource.memory-mb:** Amount of physical memory, in MB, that can be allocated for containers. It means the amount of memory YARN can utilize on this node and therefore this property should be lower then the total memory of that machine. **yarn.scheduler.maximum-allocation-mb:** It defines the maximum memory allocation available for a container in MB it means RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more than total allocated memory of the Node. **## How was this patch tested?** Manually tested in hdfs-Yarn clustaer Closes #22199 from sujith71955/maste_am_log. Authored-by: s71955 <sujithchacko.2010@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
parent
ab33028957
commit
c20916a5dc
|
@ -344,7 +344,8 @@ private[spark] class Client(
|
|||
if (amMem > maxMem) {
|
||||
throw new IllegalArgumentException(s"Required AM memory ($amMemory" +
|
||||
s"+$amMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
|
||||
"Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.")
|
||||
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
|
||||
"'yarn.nodemanager.resource.memory-mb'.")
|
||||
}
|
||||
logInfo("Will allocate AM container, with %d MB memory including %d MB overhead".format(
|
||||
amMem,
|
||||
|
|
Loading…
Reference in a new issue