Modified typo.
Author: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>
Closes#3560 from tsudukim/feature/SPARK-4701 and squashes the following commits:
ed2a3f1 [Masayoshi TSUZUKI] Another whitespace position error.
1af3a35 [Masayoshi TSUZUKI] [SPARK-4701] Typo in sbt/sbt
In -launch-lib.bash, -Xdebug option is used for debugging. We should use -agentlib option for Java 6+.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes#2904 from sarutak/SPARK-4057 and squashes the following commits:
39b5320 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4057
26b4af8 [Kousuke Saruta] Improved java option for debugging
when building spark by sbt, the function “runAlternateBoot" in sbt/sbt-launch-lib.bash is not used. And this function is not used by spark code. So I think this function is not necessary. And the option of "sbt.boot.properties" can be configured in the command line when building spark, eg:
sbt/sbt assembly -Dsbt.boot.properties=$bootpropsfile.
The file from https://github.com/sbt/sbt-launcher-package is changed. And the function “runAlternateBoot" is deleted in upstream project. I think spark project should delete this function in file sbt/sbt-launch-lib.bash. Thanks.
Author: KaiXinXiaoLei <huleilei1@huawei.com>
Closes#3224 from KaiXinXiaoLei/deleteFunction and squashes the following commits:
e8eac49 [KaiXinXiaoLei] Delete blank lines.
efe36d4 [KaiXinXiaoLei] Delete unnecessary function
sbt-launch-lib.bash includes `die` command but it's not valid command for Linux, MacOS X or Windows.
Closes#2898
Author: Jey Kottalam <jey@kottalam.net>
Closes#3182 from sarutak/SPARK-4312 and squashes the following commits:
24c6677 [Jey Kottalam] bash doesn't have "die"
...
Tested ! TBH, it isn't a great idea to have directory with spaces within. Because emacs doesn't like it then hadoop doesn't like it. and so on...
Author: Prashant Sharma <prashant.s@imaginea.com>
Closes#2229 from ScrapCodes/SPARK-3337/quoting-shell-scripts and squashes the following commits:
d4ad660 [Prashant Sharma] SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within.
In the Jenkins console output, `curl` gives us mountains of `#` symbols as it tries to show its download progress.
I don't think this is useful so I've changed things to suppress these progress bars. If there is actually some use to this, feel free to reject this proposal.
Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Closes#2279 from nchammas/trim-test-output and squashes the following commits:
14a720c [Nicholas Chammas] suppress curl/wget progress bars
NOTE: It is not possible to use both env variable `SBT_MAVEN_PROFILES` and `-P` flag at same time. `-P` if specified takes precedence.
Author: Prashant Sharma <prashant.s@imaginea.com>
Closes#1374 from ScrapCodes/SPARK-2437/rename-MAVEN_PROFILES and squashes the following commits:
8694bde [Prashant Sharma] [SPARK-2437] Rename MAVEN_PROFILES to SBT_MAVEN_PROFILES and add SBT_MAVEN_PROPERTIES
Patch introduces the new way of working also retaining the existing ways of doing things.
For example build instruction for yarn in maven is
`mvn -Pyarn -PHadoop2.2 clean package -DskipTests`
in sbt it can become
`MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly`
Also supports
`sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly`
Author: Prashant Sharma <prashant.s@imaginea.com>
Author: Patrick Wendell <pwendell@gmail.com>
Closes#772 from ScrapCodes/sbt-maven and squashes the following commits:
a8ac951 [Prashant Sharma] Updated sbt version.
62b09bb [Prashant Sharma] Improvements.
fa6221d [Prashant Sharma] Excluding sql from mima
4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default.
72651ca [Prashant Sharma] Addresses code reivew comments.
acab73d [Prashant Sharma] Revert "Small fix to run-examples script."
ac4312c [Prashant Sharma] Revert "minor fix"
6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit.
65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path.
446768e [Prashant Sharma] minor fix
89b9777 [Prashant Sharma] Merge conflicts
d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups.
dccc8ac [Prashant Sharma] updated mima to check against 1.0
a49c61b [Prashant Sharma] Fix for tools jar
a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies.
cf88758 [Prashant Sharma] cleanup
9439ea3 [Prashant Sharma] Small fix to run-examples script.
96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven.
36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins.
4973dbd [Patrick Wendell] Example build using pom reader.
Add golden answer for aforementioned test.
Also, fix golden test generation from sbt/sbt by setting the classpath correctly.
Author: Michael Armbrust <michael@databricks.com>
Closes#244 from marmbrus/partTest and squashes the following commits:
37a33c9 [Michael Armbrust] Un-ignore a test that is now passing, add golden answer for aforementioned test. Fix golden test generation from sbt/sbt.
There was a mistake in sbt build file ( introduced by 012bd5fbc9 ) in which we set the default to 2048 and the immediately reset it to 1024.
Without this, building Spark can run out of permgen space on my machine.
Author: Reynold Xin <rxin@apache.org>
Closes#103 from rxin/sbt and squashes the following commits:
8829c34 [Reynold Xin] Allow sbt to use more than 1G of heap.
Author: Prashant Sharma <prashant.s@imaginea.com>
Author: Patrick Wendell <pwendell@gmail.com>
Closes#17 from ScrapCodes/java8-lambdas and squashes the following commits:
95850e6 [Patrick Wendell] Some doc improvements and build changes to the Java 8 patch.
85a954e [Prashant Sharma] Nit. import orderings.
673f7ac [Prashant Sharma] Added support for -java-home as well
80a13e8 [Prashant Sharma] Used fake class tag syntax
26eb3f6 [Prashant Sharma] Patrick's comments on PR.
35d8d79 [Prashant Sharma] Specified java 8 building in the docs
31d4cd6 [Prashant Sharma] Maven build to support -Pjava8-tests flag.
4ab87d3 [Prashant Sharma] Review feedback on the pr
c33dc2c [Prashant Sharma] SPARK-964, Java 8 API Support.
This allows developers to pass options (such as -D) to sbt. I also modified the SparkBuild to ensure spark specific properties are propagated to forked test JVMs.
Author: Michael Armbrust <michael@databricks.com>
Closes#14 from marmbrus/sbtScripts and squashes the following commits:
c008b18 [Michael Armbrust] Merge the old sbt-launch-lib.bash with the new sbt-launcher jar downloading logic.
Make sbt download an atomic operation
Modifies the `sbt/sbt` script to gracefully recover when a previous invocation died in the middle of downloading the SBT jar.
Author: Jey Kottalam <jey@cs.berkeley.edu>
== Merge branch commits ==
commit 6c600eb434a2f3e7d70b67831aeebde9b5c0f43b
Author: Jey Kottalam <jey@cs.berkeley.edu>
Date: Fri Jan 17 10:43:54 2014 -0800
Make sbt download an atomic operation
This changes our `sbt/sbt` script to not delegate to the user's `sbt`
even if it is present. If users already have sbt installed and they
want to use their own sbt, we'd expect them to just call sbt directly
from within Spark. We no longer set any enironment variables or anything
from this script, so they should just launch sbt directly on their own.
There are a number of hard-to-debug issues which can come from the
current appraoch. One is if the user is unaware of an existing sbt
installation and now without explanation their build breaks because
they haven't configured options correctly (such as permgen size)
within their sbt. Another is if the user has a much older version
of sbt hanging around, in which case some of the older versions
don't acutally work well when newer verisons of sbt are specified
in the build file (reported by @marmbrus). A third is if the user
has done some other modification to their sbt script, such as
setting it to delegate to sbt/sbt in Spark, and this causes
that to break (also reported by @marmbrus).
So to keep things simple let's just avoid this path and
remove it. Any user who already has sbt and wants to build
spark with it should be able to understand easily how to do it.
This allows the spark-shell, spark-class, run-example, make-distribution.sh,
and ./bin/start-* scripts to work under Cygwin. Note that this doesn't
support PySpark under Cygwin, since that requires many additional `cygpath`
calls from within Python and will be non-trivial to implement.
This PR was inspired by, and subsumes, #253 (so close#253 after this is merged).
In some environments, this command
export SPARK_HOME=$(cd "$(dirname $0)/.."; pwd)
echoes two paths, one by the "cd ..", and one by the "pwd". Note the resulting
erroneous -jar paths below:
ctn@ubuntu:~/src/spark$ sbt/sbt
+ EXTRA_ARGS=
+ '[' '' '!=' '' ']'
+++ dirname sbt/sbt
++ cd sbt/..
++ pwd
+ export 'SPARK_HOME=/home/ctn/src/spark
/home/ctn/src/spark'
+ SPARK_HOME='/home/ctn/src/spark
/home/ctn/src/spark'
+ export SPARK_TESTING=1
+ SPARK_TESTING=1
+ java -Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=128m -jar /home/ctn/src/spark /home/ctn/src/spark/sbt/sbt-launch-0.11.3-2.jar
Error: Invalid or corrupt jarfile /home/ctn/src/spark
Committer: ctn <ctn@adatao.com>
On branch master
Changes to be committed:
- Send output of the "cd .." part to /dev/null
modified: sbt/sbt
