ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
682 commits 6 branches 181 tags 397 MiB
Commit graph

682 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ismael Juma 483f724d62 Upgrade to Scala 2.9.1. 
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.

A couple of small changes in SparkIMain were also required.

All tests pass and ./spark-shell launches successfully.
 2011-08-31 10:43:05 +01:00
Matei Zaharia 1069740264 Added a jarOfObject method to get the JAR of the class that an object 
belongs to, which seems like a more common case.
 2011-08-29 23:27:10 -07:00
Matei Zaharia 0c965a102e Removed a debugging line 2011-08-29 23:13:33 -07:00
Matei Zaharia 0aa23bf17e Added a convenience method for getting the JAR file that loaded a class 
(useful for jobs to pass their own JAR files to SparkContext).
 2011-08-29 22:59:44 -07:00
Matei Zaharia 9b7215d74a Only include JAR files in lib in the unmanaged classpath. 2011-08-29 22:59:18 -07:00
Matei Zaharia 4cc5e51e28 Use the new SBT 0.10 assembly plugin instead of the DepJar code we had. 2011-08-29 16:13:26 -07:00
Matei Zaharia a161f00610 Made a log message slightly less ugly 2011-08-27 16:58:54 -07:00
Matei Zaharia 3759bcd061 New mesos.jar 2011-08-10 14:03:48 -07:00
Matei Zaharia c22043f150 Minor fix: can use >= when checking memory 2011-08-02 19:11:17 -07:00
Matei Zaharia f6c8c2a394 Merge pull request #77 from ijuma/issue69 
Enable -optimize in the build
 2011-08-02 15:47:58 -07:00
Matei Zaharia cfc6000ede Merge pull request #76 from ijuma/deprecations-scala-2.9 
Fix code not to use code deprecated in Scala 2.9
 2011-08-02 15:47:42 -07:00
Ismael Juma 2b6fd3198d Fix issue #69: Enable -optimize in the build 2011-08-02 10:34:41 +01:00
Ismael Juma 59c7131dff Add note about why we can't enable -deprecation switch. 2011-08-02 10:26:17 +01:00
Ismael Juma 065043a14f Use process instead of main as the latter is deprecated. 2011-08-02 10:26:03 +01:00
Ismael Juma 6ff57f5594 Use scala.math instead of Math as the latter is deprecated. 2011-08-02 10:25:47 +01:00
Ismael Juma 620de2dd1d Change currentThread to Thread.currentThread as the former is deprecated. 2011-08-02 10:25:16 +01:00
Ismael Juma 0fba22b3d2 Fix issue #65: Change @serializable to extends Serializable in 2.9 branch 
Note that we use scala.Serializable introduced in Scala 2.9 instead of
java.io.Serializable. Also, case classes inherit from scala.Serializable by
default.
 2011-08-02 10:16:33 +01:00
Matei Zaharia 2e57338896 Merge branch 'scala-2.9' 2011-08-01 15:27:08 -07:00
Matei Zaharia a0b7865037 Merge pull request #75 from ijuma/scala-2.9 
Workaround for scalac bug and publishTo configuration
 2011-08-01 15:26:31 -07:00
Matei Zaharia 711575391d Merge branch 'scala-2.9' 
Conflicts:
	project/build/SparkProject.scala
 2011-08-01 15:25:26 -07:00
Matei Zaharia 4050d661c5 Updated to newest Mesos API, which includes better memory accounting 
by specifying per-executor memory.
 2011-08-01 13:54:48 -07:00
Ismael Juma 3f5d7b5d11 Add publishTo configuration that publishes to target directory. 
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
 2011-07-31 20:17:58 +01:00
Ismael Juma 7565cc2e33 Add type annotation to result of depJarSettings to workaround scalac bug. 2011-07-31 20:17:58 +01:00
Matei Zaharia d12122502b Various improvements to Kryo serializer: 
- Replaced modified Kryo version with the standard one augmented with
  the kryo-serializers package, which includes support for classes with
  no-arg constructors (that was why we had a modified Kryo before)
- The kryo-serializers version also fixes issue #72.
- Added a bunch of tests.
- Serialize maps and a few other common types properly by default.
 2011-07-21 22:09:33 -07:00
Matei Zaharia baa72e2747 Removed a debug statement that slipped in as a println 2011-07-21 16:09:33 -07:00
Matei Zaharia 2bfd7931e8 Merge branch 'new-rdds-protobuf' 
Conflicts:
	core/src/main/scala/spark/Executor.scala
	core/src/main/scala/spark/RDD.scala
 2011-07-21 16:08:39 -07:00
Ismael Juma ecb8b69fa0 sbt 0.10.1 calls update automatically so remove it from the documentation. 2011-07-21 01:04:55 +01:00
Ismael Juma fc0a2c8db8 Add and configure junit_xml_listener as a replacement for XmlTestReport. 2011-07-21 01:04:29 +01:00
Ismael Juma 51673ca62e Introduce DepJarPlugin based on AssemblyPlugin and use it in SparkBuild. 2011-07-18 10:34:51 +01:00
Ismael Juma c71fc27c74 Update sbt-idea to non-snapshot version and uncomment sbt-assembly dependency. 2011-07-18 00:17:21 +01:00
Ismael Juma 8531c2a079 Update test dependencies. 2011-07-18 00:16:08 +01:00
Ismael Juma 635f501492 Fix copy & paste error in version. 2011-07-18 00:13:37 +01:00
Ismael Juma f686e3dacb Initial work on converting build to SBT 0.10.1 2011-07-15 03:38:30 +01:00
Matei Zaharia cf8f5de61b Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
	repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
 2011-07-14 17:48:56 -04:00
Matei Zaharia 02678724a4 Update version number to 0.4-SNAPSHOT 2011-07-14 17:47:39 -04:00
Matei Zaharia 7c77b2fa6a Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
 2011-07-14 17:39:34 -04:00
Matei Zaharia c86af80022 Change version to 0.3 2011-07-14 17:38:43 -04:00
Matei Zaharia 1450fd74d9 Merge branch 'master' into scala-2.9 2011-07-14 17:37:24 -04:00
Matei Zaharia ccf48388cd Lowered default number of splits for files 2011-07-14 17:37:04 -04:00
Matei Zaharia 146a18c2a4 Merge branch 'master' into scala-2.9 2011-07-14 17:29:17 -04:00
Matei Zaharia c8eb8b2b90 Set class loader for remote actors to fix a bug that happens in 2.9 2011-07-14 17:29:11 -04:00
Matei Zaharia 8ea67307b9 Merge branch 'master' into scala-2.9 2011-07-14 14:47:12 -04:00
Matei Zaharia e4c3402d2d Renamed ParallelArray to ParallelCollection 2011-07-14 14:47:01 -04:00
Matei Zaharia 9ac461d85d Remove RDD.toString because it looked confusing 2011-07-14 14:39:32 -04:00
Matei Zaharia 797b4547c3 Fix tracking of updates in accumulators to solve an issue that would manifest in the 2.9 interpreter 2011-07-14 14:08:34 -04:00
Matei Zaharia 3efd9e94d8 Merge branch 'master' into scala-2.9 2011-07-14 12:42:57 -04:00
Matei Zaharia 0ccfe20755 Forgot to add a file 2011-07-14 12:42:50 -04:00
Matei Zaharia 38f38dda5b Merge branch 'master' into scala-2.9 2011-07-14 12:42:02 -04:00
Matei Zaharia 969644df8e Cleaned up a few issues to do with default parallelism levels. Also 
renamed HadoopFileWriter to HadoopWriter (since it's not only for files)
and fixed a bug for lookup().
 2011-07-14 12:40:56 -04:00
Matei Zaharia 2fb906e8e5 Merge branch 'master' into scala-2.9 2011-07-14 00:20:14 -04:00