[MINOR][YARN] Add disable yarn.nodemanager.vmem-check-enabled option to memLimitExceededLogMessage
My spark application sometimes will throw `Container killed by YARN for exceeding memory limits`. Even I increased `spark.yarn.executor.memoryOverhead` to 10G, this error still happen. The latest config: <img width="685" alt="memory-config" src="https://user-images.githubusercontent.com/5399861/36975716-f5c548d2-20b5-11e8-95e5-b228d50917b9.png"> And error message: ``` ExecutorLostFailure (executor 121 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 30.7 GB of 30 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. ``` This is because of [Linux glibc >= 2.10 (RHEL 6) malloc may show excessive virtual memory usage](https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en). So disable `yarn.nodemanager.vmem-check-enabled` looks like a good option as [MapR mentioned ](https://mapr.com/blog/best-practices-yarn-resource-management). This PR add disable `yarn.nodemanager.vmem-check-enabled` option to memLimitExceededLogMessage. More details: https://issues.apache.org/jira/browse/YARN-4714 https://stackoverflow.com/a/31450291 https://stackoverflow.com/a/42091255 After this PR: <img width="898" alt="yarn" src="https://user-images.githubusercontent.com/5399861/36975949-c8e7bbbe-20b6-11e8-9513-9f903b868d8d.png"> N/A Author: Yuming Wang <yumwang@ebay.com> Author: Yuming Wang <wgyumg@gmail.com> Closes #20735 from wangyum/YARN-4714. Change-Id: Ie10836e2c07b6384d228c3f9e89f802823bd9f16
This commit is contained in:
parent
4c587eb488
commit
04e71c3160
|
@ -736,7 +736,8 @@ private object YarnAllocator {
|
|||
def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
|
||||
val matcher = pattern.matcher(diagnostics)
|
||||
val diag = if (matcher.find()) " " + matcher.group() + "." else ""
|
||||
("Container killed by YARN for exceeding memory limits." + diag
|
||||
+ " Consider boosting spark.yarn.executor.memoryOverhead.")
|
||||
s"Container killed by YARN for exceeding memory limits. $diag " +
|
||||
"Consider boosting spark.yarn.executor.memoryOverhead or " +
|
||||
"disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714."
|
||||
}
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue