Commit graph

10 commits

Author SHA1 Message Date
Matei Zaharia 2ee6a7e32a Print output from spark-daemon only when it fails to launch 2013-08-31 17:31:07 -07:00
Matei Zaharia 89a20b83e9 Delete some code that was added back in a merge and print less info in
spark-daemon
2013-08-31 16:55:25 -07:00
Matei Zaharia 53cd50c069 Change build and run instructions to use assemblies
This commit makes Spark invocation saner by using an assembly JAR to
find all of Spark's dependencies instead of adding all the JARs in
lib_managed. It also packages the examples into an assembly and uses
that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script
with two better-named scripts: "run-examples" for examples, and
"spark-class" for Spark internal classes (e.g. REPL, master, etc). This
is also designed to minimize the confusion people have in trying to use
"run" to run their own classes; it's not meant to do that, but now at
least if they look at it, they can modify run-examples to do a decent
job for them.

As part of this, Bagel's examples are also now properly moved to the
examples package instead of bagel.
2013-08-29 21:19:04 -07:00
Patrick Wendell b4905c383b Log the launch command for Spark daemons
For debugging and analysis purposes, it's nice to have the exact command
used to launch Spark contained within the logs. This adds the necessary
hooks to make that possible.
2013-08-02 16:58:19 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
kalpit aa9134f72a spark instance number must be present in log filename to prevent multiple workers from overriding each other's logs 2013-03-26 17:49:30 -07:00
kalpit f08db010d3 added SPARK_WORKER_INSTANCES : allows spawning multiple worker instances/processes on every slave machine 2013-03-26 17:49:30 -07:00
Matei Zaharia 30362a21e7 Update license info on deploy scripts 2012-09-25 14:43:47 -07:00
Denny 8fb955fd40 Add Apache license to non-trivial scripts taken from Hadoop. 2012-08-04 17:04:33 -07:00
Denny 0ee44c225e Spark standalone mode cluster scripts.
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00