spark-instrumented-optimizer/sbin
Kent Yao 7e2ed40d58 [SPARK-31759][DEPLOY] Support configurable max number of rotate logs for spark daemons
### What changes were proposed in this pull request?

in `spark-daemon.sh`, `spark_rotate_log()` accepts `$2` as a custom setting for the number of maximum rotate log files, but this part of code is actually never used.

This PR adds `SPARK_LOG_MAX_FILES` environment variable to represent the maximum log files of Spark daemons can rotate to.

### Why are the changes needed?

the logs files that all spark daemons are hardcoded as 5, but it supposed to be configurable

### Does this PR introduce _any_ user-facing change?

yes, SPARK_LOG_MAX_FILES is added to represent the maximum log files of Spark daemons can rotate to.

### How was this patch tested?

verify locally for the added shell script:

```shell
 kentyaohulk  ~  SPARK_LOG_MAX_FILES=1 sh test.sh
1
 kentyaohulk  ~  SPARK_LOG_MAX_FILES=a sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  SPARK_LOG_MAX_FILES=b sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  SPARK_LOG_MAX_FILES=-1 sh test.sh
Error: SPARK_LOG_MAX_FILES must be a postive number
 ✘ kentyaohulk  ~  sh test.sh
5
 ✘ kentyaohulk  ~  cat test.sh
#!/bin/bash

if [[ -z ${SPARK_LOG_MAX_FILES} ]] ; then
      num=5
elif [[ ${SPARK_LOG_MAX_FILES} -gt 0 ]]; then
      num=${SPARK_LOG_MAX_FILES}
else
    echo "Error: SPARK_LOG_MAX_FILES must be a postive number"
    exit -1
fi
```

Closes #28580 from yaooqinn/SPARK-31759.

Authored-by: Kent Yao <yaooqinn@hotmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-05-20 19:18:05 +09:00
..
decommission-slave.sh [SPARK-20628][CORE][K8S] Start to improve Spark decommissioning & preemption support 2020-02-14 12:36:52 -08:00
slaves.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
spark-config.sh [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 2020-02-20 09:09:30 -08:00
spark-daemon.sh [SPARK-31759][DEPLOY] Support configurable max number of rotate logs for spark daemons 2020-05-20 19:18:05 +09:00
spark-daemons.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
start-all.sh [SPARK-13521][BUILD] Remove reference to Tachyon in cluster & release scripts 2016-02-26 22:35:12 -08:00
start-history-server.sh [SPARK-25711][CORE] Improve start-history-server.sh: show usage User-Friendly and remove deprecated options 2018-10-13 13:34:31 -07:00
start-master.sh [SPARK-25712][CORE][MINOR] Improve usage message of start-master.sh and start-slave.sh 2018-10-12 12:42:34 -05:00
start-mesos-dispatcher.sh [SPARK-17944][DEPLOY] sbin/start-* scripts use of hostname -f fail with Solaris 2016-10-22 09:37:53 +01:00
start-mesos-shuffle-service.sh [SPARK-27056][MESOS] Remove start-shuffle-service.sh 2019-03-08 18:51:38 -06:00
start-slave.sh [SPARK-31018][CORE][DOCS] Deprecate support of multiple workers on the same host in Standalone 2020-04-15 11:29:55 -07:00
start-slaves.sh [SPARK-17944][DEPLOY] sbin/start-* scripts use of hostname -f fail with Solaris 2016-10-22 09:37:53 +01:00
start-thriftserver.sh [SPARK-25735][CORE][MINOR] Improve start-thriftserver.sh: print clean usage and exit with code 1 2018-10-17 09:56:17 -05:00
stop-all.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
stop-history-server.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
stop-master.sh [SPARK-13521][BUILD] Remove reference to Tachyon in cluster & release scripts 2016-02-26 22:35:12 -08:00
stop-mesos-dispatcher.sh [SPARK-13414][MESOS] Allow multiple dispatchers to be launched. 2016-02-20 12:58:47 -08:00
stop-mesos-shuffle-service.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
stop-slave.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00
stop-slaves.sh [SPARK-13521][BUILD] Remove reference to Tachyon in cluster & release scripts 2016-02-26 22:35:12 -08:00
stop-thriftserver.sh [SPARK-2960][DEPLOY] Support executing Spark from symlinks (reopen) 2015-11-04 10:49:34 +00:00