Ankur Dave
cbdc01eecd
Update WikipediaPageRank to reflect Bagel API changes
2011-10-09 16:19:34 -07:00
Ankur Dave
6d707f6b63
Remove ShortestPath for now
2011-10-09 16:19:34 -07:00
Ankur Dave
0028caf3a4
Simplify and genericize type parameters in Bagel
2011-10-09 15:58:39 -07:00
Ankur Dave
2d7057bf5d
Implement PairRDDFunctions.partitionBy
2011-10-09 15:52:09 -07:00
Ankur Dave
06637cb69e
Fix PairRDDFunctions.groupWith partitioning
...
This commit fixes a bug in groupWith that was causing it to destroy
partitioning information. It replaces a call to map with a call to
mapValues, which preserves partitioning.
2011-10-09 15:48:46 -07:00
Ankur Dave
2911a783d6
Add custom partitioner support to PairRDDFunctions.combineByKey
2011-10-09 15:47:20 -07:00
Ankur Dave
6c6e47e3cd
Use BufferedOutputStream in ShuffleMapTask
2011-10-09 15:43:31 -07:00
Matei Zaharia
6483b41377
Merge pull request #83 from ijuma/sbt-0.11
...
Upgrade to SBT 0.11.0
2011-09-30 21:52:18 -07:00
Ismael Juma
d76c0fc781
Upgrade to sbt-idea 0.11.0 final.
2011-09-27 23:13:38 +01:00
Ismael Juma
7e92ef9d19
Add workaround for bug in SBT (issue #206 ).
2011-09-27 00:04:59 +01:00
Ismael Juma
4019305afe
Set SCALA_VERSION to 2.9.1 (from 2.9.1.final) to match expectation of SBT 0.11.0
2011-09-26 22:44:41 +01:00
Ismael Juma
3562db6374
Include "spark-" prefix in project name (used when artifact is published).
2011-09-26 22:41:07 +01:00
Ismael Juma
28b5d5a2af
Upgrade compress-lzf to 0.8.4.
2011-09-26 22:32:05 +01:00
Ismael Juma
315e55fde3
Upgrade Jetty to 8.0.1.
2011-09-26 22:32:05 +01:00
Ismael Juma
ee980439e2
Use scalatest and scalacheck compiled against Scala 2.9.1.
2011-09-26 22:32:05 +01:00
Ismael Juma
bd774eb274
Use new layout for plugins definitions (recommended for SBT 0.11)
2011-09-26 22:32:05 +01:00
Ismael Juma
e39edcce60
Upgrade to SBT 0.11.0.
2011-09-26 22:24:29 +01:00
Matei Zaharia
a362985f03
Merge pull request #82 from ijuma/scala-2.9.1
...
Upgrade to Scala 2.9.1.
2011-08-31 11:27:16 -07:00
Ismael Juma
483f724d62
Upgrade to Scala 2.9.1.
...
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.
A couple of small changes in SparkIMain were also required.
All tests pass and ./spark-shell launches successfully.
2011-08-31 10:43:05 +01:00
Matei Zaharia
1069740264
Added a jarOfObject method to get the JAR of the class that an object
...
belongs to, which seems like a more common case.
2011-08-29 23:27:10 -07:00
Matei Zaharia
0c965a102e
Removed a debugging line
2011-08-29 23:13:33 -07:00
Matei Zaharia
0aa23bf17e
Added a convenience method for getting the JAR file that loaded a class
...
(useful for jobs to pass their own JAR files to SparkContext).
2011-08-29 22:59:44 -07:00
Matei Zaharia
9b7215d74a
Only include JAR files in lib in the unmanaged classpath.
2011-08-29 22:59:18 -07:00
Matei Zaharia
4cc5e51e28
Use the new SBT 0.10 assembly plugin instead of the DepJar code we had.
2011-08-29 16:13:26 -07:00
Matei Zaharia
a161f00610
Made a log message slightly less ugly
2011-08-27 16:58:54 -07:00
Matei Zaharia
3759bcd061
New mesos.jar
2011-08-10 14:03:48 -07:00
Matei Zaharia
c22043f150
Minor fix: can use >= when checking memory
2011-08-02 19:11:17 -07:00
Matei Zaharia
f6c8c2a394
Merge pull request #77 from ijuma/issue69
...
Enable -optimize in the build
2011-08-02 15:47:58 -07:00
Matei Zaharia
cfc6000ede
Merge pull request #76 from ijuma/deprecations-scala-2.9
...
Fix code not to use code deprecated in Scala 2.9
2011-08-02 15:47:42 -07:00
Ismael Juma
2b6fd3198d
Fix issue #69 : Enable -optimize in the build
2011-08-02 10:34:41 +01:00
Ismael Juma
59c7131dff
Add note about why we can't enable -deprecation switch.
2011-08-02 10:26:17 +01:00
Ismael Juma
065043a14f
Use process instead of main as the latter is deprecated.
2011-08-02 10:26:03 +01:00
Ismael Juma
6ff57f5594
Use scala.math instead of Math as the latter is deprecated.
2011-08-02 10:25:47 +01:00
Ismael Juma
620de2dd1d
Change currentThread to Thread.currentThread as the former is deprecated.
2011-08-02 10:25:16 +01:00
Ismael Juma
0fba22b3d2
Fix issue #65 : Change @serializable to extends Serializable in 2.9 branch
...
Note that we use scala.Serializable introduced in Scala 2.9 instead of
java.io.Serializable. Also, case classes inherit from scala.Serializable by
default.
2011-08-02 10:16:33 +01:00
Matei Zaharia
2e57338896
Merge branch 'scala-2.9'
2011-08-01 15:27:08 -07:00
Matei Zaharia
a0b7865037
Merge pull request #75 from ijuma/scala-2.9
...
Workaround for scalac bug and publishTo configuration
2011-08-01 15:26:31 -07:00
Matei Zaharia
711575391d
Merge branch 'scala-2.9'
...
Conflicts:
project/build/SparkProject.scala
2011-08-01 15:25:26 -07:00
Matei Zaharia
4050d661c5
Updated to newest Mesos API, which includes better memory accounting
...
by specifying per-executor memory.
2011-08-01 13:54:48 -07:00
Ismael Juma
3f5d7b5d11
Add publishTo configuration that publishes to target directory.
...
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
2011-07-31 20:17:58 +01:00
Ismael Juma
7565cc2e33
Add type annotation to result of depJarSettings to workaround scalac bug.
2011-07-31 20:17:58 +01:00
Matei Zaharia
d12122502b
Various improvements to Kryo serializer:
...
- Replaced modified Kryo version with the standard one augmented with
the kryo-serializers package, which includes support for classes with
no-arg constructors (that was why we had a modified Kryo before)
- The kryo-serializers version also fixes issue #72 .
- Added a bunch of tests.
- Serialize maps and a few other common types properly by default.
2011-07-21 22:09:33 -07:00
Matei Zaharia
baa72e2747
Removed a debug statement that slipped in as a println
2011-07-21 16:09:33 -07:00
Matei Zaharia
2bfd7931e8
Merge branch 'new-rdds-protobuf'
...
Conflicts:
core/src/main/scala/spark/Executor.scala
core/src/main/scala/spark/RDD.scala
2011-07-21 16:08:39 -07:00
Ismael Juma
ecb8b69fa0
sbt 0.10.1 calls update
automatically so remove it from the documentation.
2011-07-21 01:04:55 +01:00
Ismael Juma
fc0a2c8db8
Add and configure junit_xml_listener as a replacement for XmlTestReport.
2011-07-21 01:04:29 +01:00
Ismael Juma
51673ca62e
Introduce DepJarPlugin based on AssemblyPlugin and use it in SparkBuild.
2011-07-18 10:34:51 +01:00
Ismael Juma
c71fc27c74
Update sbt-idea to non-snapshot version and uncomment sbt-assembly dependency.
2011-07-18 00:17:21 +01:00
Ismael Juma
8531c2a079
Update test dependencies.
2011-07-18 00:16:08 +01:00
Ismael Juma
635f501492
Fix copy & paste error in version.
2011-07-18 00:13:37 +01:00