121f9338ce
Launcher is implemented as a Java application and sometimes I'd like to apply Java options. One situation I have met is the time I try to attach debugger to Launcher. Launcher is launched from bin/spark-class but there is no room to apply Java options. ``` build_command() { "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$" printf "%d\0" $? } ``` Considering that it's not so many times to apply Java options to Launcher, one compromise would just modify spark-class by user like as follows. ``` build_command() { "$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$" printf "%d\0" $? } ``` But it doesn't work when any text related to Java options is output to standard output because whole output is used as command-string for spark-shell and spark-submit in current implementation. One example is jdwp. When apply agentlib option to use jdwp for debug, we will get output like as follows. ``` Listening for transport dt_socket at address: 9876 ``` The output shown above is not a command-string so spark-submit and spark-shell will fail. To enable Java options for Launcher, we need treat command-string and others. I changed launcher/Main.java and bin/spark-class to print separator-character and treat it. ## How was this patch tested? Tested manually using Spark Shell with / without LAUNCHER_JAVA_OPTIONS like as follows. ``` SPARK_LAUNCHER_OPTS="-agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:9876,server=y" bin/spark-shell ``` Closes #25265 from sarutak/add-spark-launcher-opts. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> |
||
---|---|---|
.. | ||
beeline | ||
beeline.cmd | ||
docker-image-tool.sh | ||
find-spark-home | ||
find-spark-home.cmd | ||
load-spark-env.cmd | ||
load-spark-env.sh | ||
pyspark | ||
pyspark.cmd | ||
pyspark2.cmd | ||
run-example | ||
run-example.cmd | ||
spark-class | ||
spark-class.cmd | ||
spark-class2.cmd | ||
spark-shell | ||
spark-shell.cmd | ||
spark-shell2.cmd | ||
spark-sql | ||
spark-sql.cmd | ||
spark-sql2.cmd | ||
spark-submit | ||
spark-submit.cmd | ||
spark-submit2.cmd | ||
sparkR | ||
sparkR.cmd | ||
sparkR2.cmd |