In sbin/spark-config.sh, parameter expansion is used to extract source root as follows.
this="${BASH_SOURCE-$0}"
I think, the parameter expansion should be ":" instead of "".
If we use "-" and BASH_SOURCE="", (empty character is set, not unset),
"" (empty character) is set to $this.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes#2930 from sarutak/SPARK-4076 and squashes the following commits:
32a0370 [Kousuke Saruta] Fixed wrong parameter expansion
https://issues.apache.org/jira/browse/SPARK-3696
We see if SPARK_CONF_DIR is already defined before assignment.
Author: WangTaoTheTonic <barneystinson@aliyun.com>
Closes#2541 from WangTaoTheTonic/confdir and squashes the following commits:
c3f31e0 [WangTaoTheTonic] Do not override the user-difined conf_dir
...
Tested ! TBH, it isn't a great idea to have directory with spaces within. Because emacs doesn't like it then hadoop doesn't like it. and so on...
Author: Prashant Sharma <prashant.s@imaginea.com>
Closes#2229 from ScrapCodes/SPARK-3337/quoting-shell-scripts and squashes the following commits:
d4ad660 [Prashant Sharma] SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within.
Author: Josh Rosen <joshrosen@apache.org>
Closes#1626 from JoshRosen/SPARK-2305 and squashes the following commits:
03fb283 [Josh Rosen] Update Py4J to version 0.8.2.1.
This reopens https://github.com/apache/incubator-spark/pull/640 against the new repo
Author: Sandy Ryza <sandy@cloudera.com>
Closes#30 from sryza/sandy-spark-1004 and squashes the following commits:
89889d4 [Sandy Ryza] Move unzipping py4j to the generate-resources phase so that it gets included in the jar the first time
5165a02 [Sandy Ryza] Fix docs
fd0df79 [Sandy Ryza] PySpark on YARN
