ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
21980 commits 6 branches 181 tags 397 MiB
Commit graph

623 commits

Author SHA1 Message Date
Reynold Xin eac566a7f4 Merge branch 'master' of github.com:mesos/spark into dev 
Conflicts:
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/PairRDDFunctions.scala
	core/src/main/scala/spark/ParallelCollection.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/CartesianRDD.scala
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/rdd/CoalescedRDD.scala
	core/src/main/scala/spark/rdd/FilteredRDD.scala
	core/src/main/scala/spark/rdd/FlatMappedRDD.scala
	core/src/main/scala/spark/rdd/GlommedRDD.scala
	core/src/main/scala/spark/rdd/HadoopRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala
	core/src/main/scala/spark/rdd/MappedRDD.scala
	core/src/main/scala/spark/rdd/PipedRDD.scala
	core/src/main/scala/spark/rdd/SampledRDD.scala
	core/src/main/scala/spark/rdd/ShuffledRDD.scala
	core/src/main/scala/spark/rdd/UnionRDD.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerId.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/MetadataCleaner.scala
	core/src/main/scala/spark/util/TimeStampedHashMap.scala
	core/src/test/scala/spark/storage/BlockManagerSuite.scala
	run
 2012-12-20 14:53:40 -08:00
Reynold Xin 9397c5014e Let the slave notify the master block removal. 2012-12-20 01:37:09 -08:00
Patrick Wendell 3ff9710265 Adding Flume InputDStream 2012-12-07 16:42:39 -08:00
Denny 556c38ed91 Added kafka JAR 2012-12-05 11:54:42 -08:00
Denny 0c1de43fc7 Working on kafka. 2012-11-06 09:41:42 -08:00
Matei Zaharia 863a55ae42 Merge remote-tracking branch 'public/master' into dev 
Conflicts:
	core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
	core/src/main/scala/spark/KryoSerializer.scala
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/executor/Executor.scala
	core/src/main/scala/spark/network/Connection.scala
	core/src/main/scala/spark/network/ConnectionManagerTest.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/NewHadoopRDD.scala
	core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockMessage.scala
	core/src/main/scala/spark/storage/BlockStore.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	project/SparkBuild.scala
	run
 2012-10-24 23:21:00 -07:00
Matei Zaharia 0967e71a00 Bump up version to 0.7.0-SNAPSHOT for master branch 2012-10-22 11:49:42 -07:00
Matei Zaharia 902a608187 Update version to 0.6.1-SNAPSHOT to show this is in development 2012-10-22 11:43:57 -07:00
Thomas Dudziak d9c2a89c57 Support for Hadoop 2 distributions such as cdh4 2012-10-18 16:08:54 -07:00
Reynold Xin 4a3fb06ac2 Updated Kryo to 2.20. 2012-10-16 01:10:01 -07:00
Patrick Wendell 629dd2691e Removing credentials line in build. 2012-10-14 19:33:39 -07:00
Matei Zaharia f8768da418 Comment out PGP stuff for publish-local to work 2012-10-14 17:37:21 -07:00
Matei Zaharia 64b52166ee Changed default Hadoop version back to 0.20.205 2012-10-14 09:51:34 -07:00
Matei Zaharia ce6b5a3ee5 Uncomment Maven publishing stuff and set version to 0.6.0 2012-10-13 15:55:39 -07:00
Patrick Wendell 6d328f54d0 Changing tabs to spaces 2012-10-10 18:54:22 -07:00
Patrick Wendell 3ed172ea59 Adding code for publishing to Sonatype. 
By default - I'm leaving this commented out. This is because
there is a bug in the PGP signing plugin which causes it to active
even duing a publish-local. So we'll just uncomment when we decide
to publish.
 2012-10-10 17:25:29 -07:00
Andy Konwinski 5897567679 Removes the included mesos-0.9.0.jar and adds a libraryDependency to 
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
 2012-10-03 08:58:05 -07:00
Matei Zaharia 6112b1a83c Don't build an assembly for the REPL 2012-10-02 17:08:16 -07:00
Matei Zaharia a925754675 Place Spray in front of Cloudera in Maven search path 2012-10-02 12:02:00 -07:00
Matei Zaharia 22684653a5 Revert "Place Spray repo ahead of Cloudera in Maven search path" 
This reverts commit 42e0a68082.
 2012-10-02 12:01:32 -07:00
Matei Zaharia 42e0a68082 Place Spray repo ahead of Cloudera in Maven search path 2012-10-02 11:37:19 -07:00
Patrick Wendell 6fee76d6d5 publish-local should go to maven + ivy by default 2012-10-01 15:34:47 -07:00
Reynold Xin 5783236ae6 Added a new command "pl" in sbt to publish to both Maven and Ivy. 2012-10-01 00:17:13 -07:00
Matei Zaharia 35cc9f13e9 Update Akka to 2.0.3 2012-09-24 14:17:10 -07:00
Matei Zaharia 1f539aa473 Update Scala version dependency to 2.9.2 2012-09-24 14:12:48 -07:00
Tathagata Das 7419d2c7ea Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project. 2012-09-02 02:35:17 -07:00
Matei Zaharia 5a8015d2db Merge remote-tracking branch 'public/dev' into dev 2012-08-24 16:11:44 -07:00
Denny 0008994044 merged dev branch 2012-08-02 16:00:33 -07:00
Matei Zaharia 71a958b0b7 Merge branch 'dev' of github.com:mesos/spark into dev 
Conflicts:
	project/SparkBuild.scala
 2012-08-02 17:23:13 -04:00
Denny ba7e30fb5e Mostly stlyistic changes. 2012-08-02 13:55:09 -07:00
Josh Rosen 039b41cb54 Use sbt mergeStrategy for reference.conf files. 
Cleans up #158 / 509b721.
 2012-08-02 10:21:50 -07:00
Denny 0ee44c225e Spark standalone mode cluster scripts. 
Heavily inspired by Hadoop cluster scripts ;-)
 2012-08-01 20:38:52 -07:00
Denny 1b29e90a79 merge dev branch 2012-08-01 14:06:09 -07:00
Denny 7a295fee96 Spark WebUI Implementation. 2012-08-01 11:01:09 -07:00
Josh Rosen 509b721d12 Fix Akka configuration in assembly jar. 
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.

This solution is from the Akka Team Blog:

http://letitcrash.com/post/21025950392/
 2012-07-30 18:05:13 -07:00
Matei Zaharia 47b7ebad12 Added the Spark Streaing code, ported to Akka 2 2012-07-28 20:03:26 -07:00
Josh Rosen 01dce3f569 Add Java API 
Add distinct() method to RDD.

Fix bug in DoubleRDDFunctions.
 2012-07-18 17:34:29 -07:00
Matei Zaharia 2fb6e7d71e Initial framework to get a master and web UI up. 2012-06-30 14:45:55 -07:00
rrmckinley 697b0bee2c Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24. Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too. 2012-06-29 16:42:05 -07:00
Matei Zaharia 3920189932 Upgraded to Akka 2 and fixed test execution (which was still parallel 
across projects).
 2012-06-28 23:51:28 -07:00
Tathagata Das ede615d719 Fixed issues duplicate class issues in sbt assembly. 2012-06-22 15:03:09 -07:00
Matei Zaharia 08579ffa11 Update version number for dev branch 2012-06-15 23:55:43 -07:00
Matei Zaharia f58da6164e Merge branch 'master' into dev 2012-06-15 23:47:11 -07:00
Matei Zaharia 4449eb9783 Changed version in master branch to 0.5.1-SNAPSHOT for further 
development.
 2012-06-13 22:26:14 -04:00
Matei Zaharia 0472cf8e0d Update version in SBT 2012-06-12 14:30:49 -04:00
Matei Zaharia 63051dd2bc Merge in engine improvements from the Spark Streaming project, developed 
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
 2012-06-07 12:45:38 -07:00
Matei Zaharia 95fb1a16b8 Use Mesos 0.9 RC3 JAR and protobuf 2.4.1 2012-03-30 11:38:49 -04:00
Matei Zaharia 4d52cc6738 Merge branch 'master' into mesos-0.9 2012-03-29 21:29:39 -04:00
Matei Zaharia ca5c19c1ba Remove dependency on Akka 2012-03-29 01:03:34 -04:00
Reynold Xin 90418b70ff Added sbt-assembly for spark-repl project so we can generate an 
assembled jar for Shark.
 2012-03-22 18:46:31 -07:00
Matei Zaharia a099a63a8a Initial work to make Spark compile with Mesos 0.9 and Hadoop 1.0 2012-03-17 12:31:34 -07:00
Matei Zaharia d6ec664b48 Add dependency on fastutil and update Guava 2012-02-06 15:37:27 -08:00
Matei Zaharia 5fd101d79e Add dependency on Akka and Netty 2011-12-15 13:21:14 +01:00
Matei Zaharia c7d6f1a65c Really upgrade to SBT 0.11.1 (through build.properties and plugin changes) 2011-11-08 21:45:29 -08:00
root 49505a0b0b Switched Jetty to version 7.5 because 8.0 was causing a conflict with the log4j and Jetty libraries in Hadoop. 2011-10-17 18:06:41 +00:00
Ismael Juma 7e92ef9d19 Add workaround for bug in SBT (issue #206). 2011-09-27 00:04:59 +01:00
Ismael Juma 3562db6374 Include "spark-" prefix in project name (used when artifact is published). 2011-09-26 22:41:07 +01:00
Ismael Juma 28b5d5a2af Upgrade compress-lzf to 0.8.4. 2011-09-26 22:32:05 +01:00
Ismael Juma 315e55fde3 Upgrade Jetty to 8.0.1. 2011-09-26 22:32:05 +01:00
Ismael Juma ee980439e2 Use scalatest and scalacheck compiled against Scala 2.9.1. 2011-09-26 22:32:05 +01:00
Ismael Juma 483f724d62 Upgrade to Scala 2.9.1. 
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.

A couple of small changes in SparkIMain were also required.

All tests pass and ./spark-shell launches successfully.
 2011-08-31 10:43:05 +01:00
Matei Zaharia 9b7215d74a Only include JAR files in lib in the unmanaged classpath. 2011-08-29 22:59:18 -07:00
Matei Zaharia 4cc5e51e28 Use the new SBT 0.10 assembly plugin instead of the DepJar code we had. 2011-08-29 16:13:26 -07:00
Ismael Juma 2b6fd3198d Fix issue #69: Enable -optimize in the build 2011-08-02 10:34:41 +01:00
Ismael Juma 59c7131dff Add note about why we can't enable -deprecation switch. 2011-08-02 10:26:17 +01:00
Matei Zaharia 2e57338896 Merge branch 'scala-2.9' 2011-08-01 15:27:08 -07:00
Matei Zaharia 711575391d Merge branch 'scala-2.9' 
Conflicts:
	project/build/SparkProject.scala
 2011-08-01 15:25:26 -07:00
Ismael Juma 3f5d7b5d11 Add publishTo configuration that publishes to target directory. 
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
 2011-07-31 20:17:58 +01:00
Ismael Juma fc0a2c8db8 Add and configure junit_xml_listener as a replacement for XmlTestReport. 2011-07-21 01:04:29 +01:00
Ismael Juma 51673ca62e Introduce DepJarPlugin based on AssemblyPlugin and use it in SparkBuild. 2011-07-18 10:34:51 +01:00
Ismael Juma 8531c2a079 Update test dependencies. 2011-07-18 00:16:08 +01:00
Ismael Juma 635f501492 Fix copy & paste error in version. 2011-07-18 00:13:37 +01:00
Ismael Juma f686e3dacb Initial work on converting build to SBT 0.10.1 2011-07-15 03:38:30 +01:00