ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1414 commits 6 branches 181 tags 397 MiB
Commit graph

46 commits

Author SHA1 Message Date
Thomas Dudziak f595bb53d1 Tweaked run file to live more happily with typesafe's debian package 2012-10-22 13:11:05 -07:00
Matei Zaharia 4a3e9cf69c Document how to configure SPARK_MEM & co on a per-job basis 2012-10-13 16:20:25 -07:00
root ce915cadee Made run script add test-classes onto the classpath only if SPARK_TESTING is set; fixes #216 2012-10-07 04:19:16 +00:00
Matei Zaharia c535762deb Don't check for JARs in core/lib anymore 2012-10-04 15:11:43 -07:00
Matei Zaharia 1f539aa473 Update Scala version dependency to 2.9.2 2012-09-24 14:12:48 -07:00
Matei Zaharia 995982b3c9 Added a unit test for local-cluster mode and simplified some of the code involved in that 2012-09-07 17:08:36 -07:00
Matei Zaharia dc8763fcf7 Fixed SPARK_MEM not being passed when runner is java 2012-07-28 19:53:31 -07:00
Matei Zaharia 0a47284003 More work to allow Spark to run on the standalone deploy cluster. 2012-07-08 14:00:04 -07:00
Matei Zaharia 408b5a1332 More work on deploy code (adding Worker class) 2012-06-30 16:45:57 -07:00
Matei Zaharia 08cda89e8a Further fixes to how Mesos is found and used 2012-03-17 13:39:14 -07:00
Ismael Juma 4019305afe Set SCALA_VERSION to 2.9.1 (from 2.9.1.final) to match expectation of SBT 0.11.0 2011-09-26 22:44:41 +01:00
Ismael Juma 483f724d62 Upgrade to Scala 2.9.1. 
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.

A couple of small changes in SparkIMain were also required.

All tests pass and ./spark-shell launches successfully.
 2011-08-31 10:43:05 +01:00
Matei Zaharia 0c965a102e Removed a debugging line 2011-08-29 23:13:33 -07:00
Matei Zaharia 711575391d Merge branch 'scala-2.9' 
Conflicts:
	project/build/SparkProject.scala
 2011-08-01 15:25:26 -07:00
Matei Zaharia 4050d661c5 Updated to newest Mesos API, which includes better memory accounting 
by specifying per-executor memory.
 2011-08-01 13:54:48 -07:00
Ismael Juma f686e3dacb Initial work on converting build to SBT 0.10.1 2011-07-15 03:38:30 +01:00
Matei Zaharia 96daa31a01 Pass quoted arguments properly to run 2011-05-31 23:34:16 -07:00
Matei Zaharia 3854d23dd4 Pass quoted arguments properly to run 2011-05-31 22:17:48 -07:00
Ismael Juma 2c691295bf SCALA_VERSION in run should be 2.9.0-1 2011-05-27 14:59:23 +01:00
Ismael Juma 1d75c6060a Update to Scala 2.9.0-1 and disable repl module for now. 
The repl module requires more complex work.
 2011-05-27 14:59:23 +01:00
Ismael Juma 59f1f42a9a Update run to work with SBT managed dependencies and the newly introduced repl module. 2011-05-27 11:22:59 +01:00
Ankur Dave 45ec9db8af Add Bagel classpath to run script 2011-05-03 15:39:21 -07:00
Matei Zaharia 7408230bfa Updated modified Kryo to use objenesis 2011-03-08 11:58:08 -08:00
Matei Zaharia e7b4b047a6 Added pluggable serializers and Kryo serialization 2011-03-07 18:41:53 -08:00
Matei Zaharia 447debb771 Updated Hadoop to 0.20.2 to include some bug fixes 2011-03-01 10:27:48 -08:00
Matei Zaharia ec28b607fd Merge branch 'master' into sbt 
Conflicts:
	Makefile
	core/src/main/java/spark/compress/lzf/LZF.java
	core/src/main/java/spark/compress/lzf/LZFInputStream.java
	core/src/main/java/spark/compress/lzf/LZFOutputStream.java
	core/src/main/native/spark_compress_lzf_LZF.c
	run
 2011-02-02 00:25:54 -08:00
Matei Zaharia e5c4cd8a5e Made examples and core subprojects 2011-02-01 15:11:08 -08:00
Joshua Hartman 2fb849502f Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense) 2010-12-05 21:20:15 -08:00
Matei Zaharia e86b620f9e Fixed some more stuff (Eclispe target and native build) 2010-11-13 22:46:00 -08:00
Matei Zaharia 89fcd96702 Initial work to get Spark compiling with SBT 0.7.5 RC0 2010-11-13 22:07:08 -08:00
Matei Zaharia b84769a107 Modified project structure to work with buildr 2010-11-13 17:18:05 -08:00
Matei Zaharia c0b856a056 Set absolute path for SPARK_HOME 2010-10-16 12:18:02 -07:00
Matei Zaharia 6c1dee2e42 Added code so that Spark jobs can be launched from outside the Spark 
directory by setting SPARK_HOME and locating the executor relative to
that. Entries on SPARK_CLASSPATH and SPARK_LIBRARY_PATH are also passed
along to worker nodes.
 2010-10-15 19:42:26 -07:00
Matei Zaharia 57a778426c Updated guava to version r07 2010-10-15 15:55:58 -07:00
Matei Zaharia 7090dea44b Changed printlns to log statements and fixed a bug in run that was causing it to fail on a Mesos cluster 2010-09-28 23:54:29 -07:00
Matei Zaharia 516248aa66 Added log4j.properties 2010-09-28 23:22:39 -07:00
Matei Zaharia c7d233b911 Added log4j jars and paths 2010-09-28 23:08:01 -07:00
Matei Zaharia e068f21e01 More work on HTTP class loading 2010-09-28 22:32:38 -07:00
root a9481c3514 Update to work with latest Mesos API changes 2010-08-13 07:39:36 +00:00
Matei Zaharia b56ed67553 Updated code to work with Nexus->Mesos name change 2010-07-25 23:53:46 -04:00
Matei Zaharia e240e38ee9 Updated a bunch of libraries, and increased the default memory in run so 
that unit tests can run successfully.
 2010-07-25 21:10:03 -04:00
Matei Zaharia 0435de9e87 Made it possible to set various Spark options and environment variables 
in general through a conf/spark-env.sh script.
 2010-07-19 18:00:30 -07:00
Matei Zaharia 7d0eae17e3 Merge branch 'dev' 
Conflicts:
	src/scala/spark/HdfsFile.scala
	src/scala/spark/NexusScheduler.scala
	src/test/spark/repl/ReplSuite.scala
 2010-06-27 15:21:54 -07:00
Matei Zaharia 1473987fb7 Fixed classpath for tests 2010-06-10 22:36:45 -07:00
Matei Zaharia 06aac8a889 Imported changes from old repository (mostly Mosharaf's work, 
plus some fault tolerance code).
 2010-04-03 23:44:55 -07:00
Matei Zaharia df29d0ea4c Initial commit 2010-03-29 16:17:55 -07:00