Jey Kottalam
cb4ef19214
yarn support
2013-08-15 16:50:37 -07:00
Joseph E. Gonzalez
499a0d8383
Merged graphx from @rxin into master
2013-08-06 12:28:29 -07:00
Patrick Wendell
b4905c383b
Log the launch command for Spark daemons
...
For debugging and analysis purposes, it's nice to have the exact command
used to launch Spark contained within the logs. This adds the necessary
hooks to make that possible.
2013-08-02 16:58:19 -07:00
Jey Kottalam
1d10192806
Fix setting of SPARK_EXAMPLES_JAR
2013-07-24 14:04:17 -07:00
Josh Rosen
c83680434b
Add JavaAPICompletenessChecker.
...
This is used to find methods in the Scala API that
need to be ported to the Java API. To use it:
./run spark.tools.JavaAPICompletenessChecker
Conflicts:
project/SparkBuild.scala
run
run2.cmd
2013-07-22 16:11:49 -07:00
Ubuntu
88a0823c58
Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable (JIRA Ticket SPARK-817)
2013-07-18 00:51:18 +00:00
Matei Zaharia
4ff494de20
Some missing license headers
2013-07-16 17:26:48 -07:00
Matei Zaharia
af3c9d5042
Add Apache license headers and LICENSE and NOTICE files
2013-07-16 17:21:33 -07:00
Prashant Sharma
a3494d405d
Merge branch 'master' of github.com:mesos/spark into scala-2.10
...
Conflicts:
core/src/main/scala/spark/Utils.scala
core/src/test/scala/spark/ui/UISuite.scala
project/SparkBuild.scala
run
2013-07-15 11:15:55 +05:30
Matei Zaharia
cd28d9c147
Merge remote-tracking branch 'origin/pr/662'
...
Conflicts:
bin/compute-classpath.sh
2013-07-13 19:10:00 -07:00
Prashant Sharma
e86d5dbaad
Merge branch 'master' into master-merge
...
Conflicts:
README.md
core/pom.xml
core/src/main/scala/spark/deploy/JsonProtocol.scala
core/src/main/scala/spark/deploy/LocalSparkCluster.scala
core/src/main/scala/spark/deploy/master/Master.scala
core/src/main/scala/spark/deploy/master/MasterWebUI.scala
core/src/main/scala/spark/deploy/worker/Worker.scala
core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
core/src/main/scala/spark/storage/BlockManagerUI.scala
core/src/main/scala/spark/util/AkkaUtils.scala
pom.xml
project/SparkBuild.scala
streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
2013-07-12 14:49:16 +05:30
Prashant Sharma
69ae7ea227
Removed some unnecessary code and fixed dependencies
2013-07-11 18:30:18 +05:30
Matei Zaharia
43b24635ee
Renamed ML package to MLlib and added it to classpath
2013-07-05 11:38:53 -07:00
Reynold Xin
ce7e270bb4
Added graph package to the classpath.
2013-06-29 21:28:22 -07:00
Evan Chan
1107b4d55b
Merge branch 'master' into 2013-06/assembly-jar-deploy
...
Conflicts:
run
Previous changes that I made to run and set-dev-classpath.sh instead
have been folded into compute-classpath.sh
2013-06-28 17:18:35 -07:00
Matei Zaharia
03906f7f0a
Fixes to compute-classpath on Windows
2013-06-26 17:40:22 -07:00
Matei Zaharia
6c8d1b2ca6
Fix computation of classpath when we launch java directly
...
The previous version assumed that a CLASSPATH environment variable was
set by the "run" script when launching the process that starts the
ExecutorRunner, but unfortunately this is not true in tests. Instead, we
factor the classpath calculation into an extenral script and call that.
NOTE: This includes a Windows version but hasn't yet been tested there.
2013-06-25 18:21:00 -04:00
Evan Chan
4cda8f865a
Add simple usage to start-slave script
2013-06-24 15:14:48 -07:00
Matei Zaharia
dc4073654b
Revert "Fix start-slave not passing instance number to spark-daemon."
...
This reverts commit a674d67c0a
2013-06-11 00:08:02 -04:00
Stephen Haberman
a674d67c0a
Fix start-slave not passing instance number to spark-daemon.
2013-05-28 16:24:19 -05:00
Josh Rosen
cda2b15041
Use ec2-metadata in start-slave.sh.
...
PR #419 applied the same change, but only to start-master.sh,
so some workers were still starting their web UI's using internal
addresses.
This should finally fix SPARK-613.
2013-05-24 13:05:06 -07:00
kalpit
aa9134f72a
spark instance number must be present in log filename to prevent multiple workers from overriding each other's logs
2013-03-26 17:49:30 -07:00
kalpit
f08db010d3
added SPARK_WORKER_INSTANCES : allows spawning multiple worker instances/processes on every slave machine
2013-03-26 17:49:30 -07:00
Shivaram Venkataraman
717b221cca
Detect whether we run on EC2 using ec2-metadata as well
2013-01-26 23:03:11 -08:00
Josh Rosen
1948f46093
Use spark-env.sh to configure standalone master. See SPARK-638.
...
Also fixed a typo in the standalone mode documentation.
2012-12-14 01:20:00 +00:00
Josh Rosen
cdaa0fad51
Use external addresses in standalone WebUI on EC2.
2012-12-01 18:19:13 -08:00
Matei Zaharia
59c0a9ad16
Use hostname instead of IP in deploy scripts to let Akka connect properly
2012-11-27 21:00:04 -08:00
Reynold Xin
f67bcbed07
Use SPARK_MASTER_IP if it is set in start-slaves.sh.
2012-10-19 01:08:23 -07:00
Matei Zaharia
30362a21e7
Update license info on deploy scripts
2012-09-25 14:43:47 -07:00
Denny
8fb955fd40
Add Apache license to non-trivial scripts taken from Hadoop.
2012-08-04 17:04:33 -07:00
Denny
c90c9ec208
Read config variables before to get the master port
2012-08-02 16:12:40 -07:00
Denny
53008c2d8a
Settings variables and bugfix for stop script.
2012-08-02 15:59:39 -07:00
Denny
0ee44c225e
Spark standalone mode cluster scripts.
...
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00
