spark-instrumented-optimizer/conf
Kent Yao 7e2ed40d58 [SPARK-31759][DEPLOY] Support configurable max number of rotate logs for spark daemons
### What changes were proposed in this pull request?

in `spark-daemon.sh`, `spark_rotate_log()` accepts `$2` as a custom setting for the number of maximum rotate log files, but this part of code is actually never used.

This PR adds `SPARK_LOG_MAX_FILES` environment variable to represent the maximum log files of Spark daemons can rotate to.

### Why are the changes needed?

the logs files that all spark daemons are hardcoded as 5, but it supposed to be configurable

### Does this PR introduce _any_ user-facing change?

yes, SPARK_LOG_MAX_FILES is added to represent the maximum log files of Spark daemons can rotate to.

### How was this patch tested?

verify locally for the added shell script:

```shell
 kentyaohulk  ~  SPARK_LOG_MAX_FILES=1 sh test.sh
1
 kentyaohulk  ~  SPARK_LOG_MAX_FILES=a sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  SPARK_LOG_MAX_FILES=b sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  SPARK_LOG_MAX_FILES=-1 sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  sh test.sh
5
 ✘ kentyaohulk  ~  cat test.sh
#!/bin/bash

if [[ -z ${SPARK_LOG_MAX_FILES} ]] ; then
      num=5
elif [[ ${SPARK_LOG_MAX_FILES} -gt 0 ]]; then
      num=${SPARK_LOG_MAX_FILES}
else
    echo "Error: SPARK_LOG_MAX_FILES must be a postive number"
    exit -1
fi
```

Closes #28580 from yaooqinn/SPARK-31759.

Authored-by: Kent Yao <yaooqinn@hotmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-05-20 19:18:05 +09:00
..
fairscheduler.xml.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
log4j.properties.template [SPARK-27610][YARN] Shade netty native libraries 2019-05-07 10:47:36 -07:00
metrics.properties.template [SPARK-29032][CORE] Add PrometheusServlet to monitor Master/Worker/Driver 2019-09-13 21:31:21 +00:00
slaves.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
spark-defaults.conf.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
spark-env.sh.template [SPARK-31759][DEPLOY] Support configurable max number of rotate logs for spark daemons 2020-05-20 19:18:05 +09:00