[SPARK-31759][DEPLOY] Support configurable max number of rotate logs for spark daemons
### What changes were proposed in this pull request? in `spark-daemon.sh`, `spark_rotate_log()` accepts `$2` as a custom setting for the number of maximum rotate log files, but this part of code is actually never used. This PR adds `SPARK_LOG_MAX_FILES` environment variable to represent the maximum log files of Spark daemons can rotate to. ### Why are the changes needed? the logs files that all spark daemons are hardcoded as 5, but it supposed to be configurable ### Does this PR introduce _any_ user-facing change? yes, SPARK_LOG_MAX_FILES is added to represent the maximum log files of Spark daemons can rotate to. ### How was this patch tested? verify locally for the added shell script: ```shell kentyaohulk ~ SPARK_LOG_MAX_FILES=1 sh test.sh 1 kentyaohulk ~ SPARK_LOG_MAX_FILES=a sh test.sh Error: SPARK_LOG_MAX_FILES must be a postive number ✘ kentyaohulk ~ SPARK_LOG_MAX_FILES=b sh test.sh Error: SPARK_LOG_MAX_FILES must be a postive number ✘ kentyaohulk ~ SPARK_LOG_MAX_FILES=-1 sh test.sh Error: SPARK_LOG_MAX_FILES must be a postive number ✘ kentyaohulk ~ sh test.sh 5 ✘ kentyaohulk ~ cat test.sh #!/bin/bash if [[ -z ${SPARK_LOG_MAX_FILES} ]] ; then num=5 elif [[ ${SPARK_LOG_MAX_FILES} -gt 0 ]]; then num=${SPARK_LOG_MAX_FILES} else echo "Error: SPARK_LOG_MAX_FILES must be a postive number" exit -1 fi ``` Closes #28580 from yaooqinn/SPARK-31759. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
This commit is contained in:
parent
eaf7a2a4ed
commit
7e2ed40d58
|
@ -62,6 +62,7 @@
|
||||||
# Generic options for the daemons used in the standalone deploy mode
|
# Generic options for the daemons used in the standalone deploy mode
|
||||||
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
|
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
|
||||||
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
|
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
|
||||||
|
# - SPARK_LOG_MAX_FILES Max log files of Spark daemons can rotate to. Default is 5.
|
||||||
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
|
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
|
||||||
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
|
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
|
||||||
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
|
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
|
||||||
|
|
|
@ -23,6 +23,7 @@
|
||||||
#
|
#
|
||||||
# SPARK_CONF_DIR Alternate conf dir. Default is ${SPARK_HOME}/conf.
|
# SPARK_CONF_DIR Alternate conf dir. Default is ${SPARK_HOME}/conf.
|
||||||
# SPARK_LOG_DIR Where log files are stored. ${SPARK_HOME}/logs by default.
|
# SPARK_LOG_DIR Where log files are stored. ${SPARK_HOME}/logs by default.
|
||||||
|
# SPARK_LOG_MAX_FILES Max log files of Spark daemons can rotate to. Default is 5.
|
||||||
# SPARK_MASTER host:path where spark code should be rsync'd from
|
# SPARK_MASTER host:path where spark code should be rsync'd from
|
||||||
# SPARK_PID_DIR The pid files are stored. /tmp by default.
|
# SPARK_PID_DIR The pid files are stored. /tmp by default.
|
||||||
# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default
|
# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default
|
||||||
|
@ -74,10 +75,16 @@ shift
|
||||||
spark_rotate_log ()
|
spark_rotate_log ()
|
||||||
{
|
{
|
||||||
log=$1;
|
log=$1;
|
||||||
num=5;
|
|
||||||
if [ -n "$2" ]; then
|
if [[ -z ${SPARK_LOG_MAX_FILES} ]]; then
|
||||||
num=$2
|
num=5
|
||||||
|
elif [[ ${SPARK_LOG_MAX_FILES} -gt 0 ]]; then
|
||||||
|
num=${SPARK_LOG_MAX_FILES}
|
||||||
|
else
|
||||||
|
echo "Error: SPARK_LOG_MAX_FILES must be a positive number, but got ${SPARK_LOG_MAX_FILES}"
|
||||||
|
exit -1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [ -f "$log" ]; then # rotate logs
|
if [ -f "$log" ]; then # rotate logs
|
||||||
while [ $num -gt 1 ]; do
|
while [ $num -gt 1 ]; do
|
||||||
prev=`expr $num - 1`
|
prev=`expr $num - 1`
|
||||||
|
|
Loading…
Reference in a new issue