ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
760 commits 6 branches 181 tags 397 MiB
Commit graph

760 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 7fd05cbb8f Update to SBT 0.11.1 2011-11-07 20:28:08 -08:00
Matei Zaharia 63da22c025 Update REPL code to use our own version of JLineReader, which fixes #89. 
I'm not entirely sure why this broke in the jump from Scala 2.9.0.1 to
2.9.1 -- maybe something about name resolution changed?
 2011-11-07 20:16:25 -08:00
Matei Zaharia 3fad5e580f Fix Scala version requirement in README 2011-11-03 22:42:36 -07:00
Matei Zaharia c2b7fd6899 Make parallelize() work efficiently for ranges of Long, Double, etc 
(splitting them into sub-ranges). Fixes #87.
 2011-11-02 15:16:02 -07:00
Matei Zaharia d4c8e69dc7 K-means example 2011-11-01 19:25:58 -07:00
Matei Zaharia 157279e9eb Update Spark to work with the latest Mesos API 2011-10-30 14:10:56 -07:00
root 3a0e6c4363 Miscellaneous fixes: 
- Executor should initialize logging properly
- groupByKey should allow custom partitioner
 2011-10-17 18:07:35 +00:00
root 49505a0b0b Switched Jetty to version 7.5 because 8.0 was causing a conflict with the log4j and Jetty libraries in Hadoop. 2011-10-17 18:06:41 +00:00
root 62aa820084 Merge branch 'ankur-master' 2011-10-14 02:14:07 +00:00
Ankur Dave ab3889f627 Implement standalone WikipediaPageRank with custom serializer 2011-10-09 16:53:10 -07:00
Ankur Dave cbdc01eecd Update WikipediaPageRank to reflect Bagel API changes 2011-10-09 16:19:34 -07:00
Ankur Dave 6d707f6b63 Remove ShortestPath for now 2011-10-09 16:19:34 -07:00
Ankur Dave 0028caf3a4 Simplify and genericize type parameters in Bagel 2011-10-09 15:58:39 -07:00
Ankur Dave 2d7057bf5d Implement PairRDDFunctions.partitionBy 2011-10-09 15:52:09 -07:00
Ankur Dave 06637cb69e Fix PairRDDFunctions.groupWith partitioning 
This commit fixes a bug in groupWith that was causing it to destroy
partitioning information. It replaces a call to map with a call to
mapValues, which preserves partitioning.
 2011-10-09 15:48:46 -07:00
Ankur Dave 2911a783d6 Add custom partitioner support to PairRDDFunctions.combineByKey 2011-10-09 15:47:20 -07:00
Ankur Dave 6c6e47e3cd Use BufferedOutputStream in ShuffleMapTask 2011-10-09 15:43:31 -07:00
Matei Zaharia 6483b41377 Merge pull request #83 from ijuma/sbt-0.11 
Upgrade to SBT 0.11.0
 2011-09-30 21:52:18 -07:00
Ismael Juma d76c0fc781 Upgrade to sbt-idea 0.11.0 final. 2011-09-27 23:13:38 +01:00
Ismael Juma 7e92ef9d19 Add workaround for bug in SBT (issue #206). 2011-09-27 00:04:59 +01:00
Ismael Juma 4019305afe Set SCALA_VERSION to 2.9.1 (from 2.9.1.final) to match expectation of SBT 0.11.0 2011-09-26 22:44:41 +01:00
Ismael Juma 3562db6374 Include "spark-" prefix in project name (used when artifact is published). 2011-09-26 22:41:07 +01:00
Ismael Juma 28b5d5a2af Upgrade compress-lzf to 0.8.4. 2011-09-26 22:32:05 +01:00
Ismael Juma 315e55fde3 Upgrade Jetty to 8.0.1. 2011-09-26 22:32:05 +01:00
Ismael Juma ee980439e2 Use scalatest and scalacheck compiled against Scala 2.9.1. 2011-09-26 22:32:05 +01:00
Ismael Juma bd774eb274 Use new layout for plugins definitions (recommended for SBT 0.11) 2011-09-26 22:32:05 +01:00
Ismael Juma e39edcce60 Upgrade to SBT 0.11.0. 2011-09-26 22:24:29 +01:00
Matei Zaharia a362985f03 Merge pull request #82 from ijuma/scala-2.9.1 
Upgrade to Scala 2.9.1.
 2011-08-31 11:27:16 -07:00
Ismael Juma 483f724d62 Upgrade to Scala 2.9.1. 
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.

A couple of small changes in SparkIMain were also required.

All tests pass and ./spark-shell launches successfully.
 2011-08-31 10:43:05 +01:00
Matei Zaharia 1069740264 Added a jarOfObject method to get the JAR of the class that an object 
belongs to, which seems like a more common case.
 2011-08-29 23:27:10 -07:00
Matei Zaharia 0c965a102e Removed a debugging line 2011-08-29 23:13:33 -07:00
Matei Zaharia 0aa23bf17e Added a convenience method for getting the JAR file that loaded a class 
(useful for jobs to pass their own JAR files to SparkContext).
 2011-08-29 22:59:44 -07:00
Matei Zaharia 9b7215d74a Only include JAR files in lib in the unmanaged classpath. 2011-08-29 22:59:18 -07:00
Matei Zaharia 4cc5e51e28 Use the new SBT 0.10 assembly plugin instead of the DepJar code we had. 2011-08-29 16:13:26 -07:00
Matei Zaharia a161f00610 Made a log message slightly less ugly 2011-08-27 16:58:54 -07:00
Matei Zaharia 3759bcd061 New mesos.jar 2011-08-10 14:03:48 -07:00
Matei Zaharia c22043f150 Minor fix: can use >= when checking memory 2011-08-02 19:11:17 -07:00
Matei Zaharia f6c8c2a394 Merge pull request #77 from ijuma/issue69 
Enable -optimize in the build
 2011-08-02 15:47:58 -07:00
Matei Zaharia cfc6000ede Merge pull request #76 from ijuma/deprecations-scala-2.9 
Fix code not to use code deprecated in Scala 2.9
 2011-08-02 15:47:42 -07:00
Ismael Juma 2b6fd3198d Fix issue #69: Enable -optimize in the build 2011-08-02 10:34:41 +01:00
Ismael Juma 59c7131dff Add note about why we can't enable -deprecation switch. 2011-08-02 10:26:17 +01:00
Ismael Juma 065043a14f Use process instead of main as the latter is deprecated. 2011-08-02 10:26:03 +01:00
Ismael Juma 6ff57f5594 Use scala.math instead of Math as the latter is deprecated. 2011-08-02 10:25:47 +01:00
Ismael Juma 620de2dd1d Change currentThread to Thread.currentThread as the former is deprecated. 2011-08-02 10:25:16 +01:00
Ismael Juma 0fba22b3d2 Fix issue #65: Change @serializable to extends Serializable in 2.9 branch 
Note that we use scala.Serializable introduced in Scala 2.9 instead of
java.io.Serializable. Also, case classes inherit from scala.Serializable by
default.
 2011-08-02 10:16:33 +01:00
Matei Zaharia 2e57338896 Merge branch 'scala-2.9' 2011-08-01 15:27:08 -07:00
Matei Zaharia a0b7865037 Merge pull request #75 from ijuma/scala-2.9 
Workaround for scalac bug and publishTo configuration
 2011-08-01 15:26:31 -07:00
Matei Zaharia 711575391d Merge branch 'scala-2.9' 
Conflicts:
	project/build/SparkProject.scala
 2011-08-01 15:25:26 -07:00
Matei Zaharia 4050d661c5 Updated to newest Mesos API, which includes better memory accounting 
by specifying per-executor memory.
 2011-08-01 13:54:48 -07:00
Ismael Juma 3f5d7b5d11 Add publishTo configuration that publishes to target directory. 
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
 2011-07-31 20:17:58 +01:00