Matei Zaharia
6e3754bf47
Add Maven build file for streaming, and fix some issues in SBT file
...
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html )
2013-01-20 19:22:24 -08:00
Tathagata Das
cd1521cfdb
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/FilteredRDD.scala
docs/_layouts/global.html
docs/index.md
run
2013-01-15 12:08:51 -08:00
Matei Zaharia
6d1c230281
Merge pull request #357 from tysonjh/master
...
JSON support added to WebUI
2013-01-10 19:06:07 -08:00
Tyson
549ee388a1
Removed io.spray spray-json dependency as it is not needed.
2013-01-09 15:12:23 -05:00
Tyson
6e8c8f61c4
Added the spray implicit marshaller library
...
Added the io.spray JSON library
2013-01-09 10:40:33 -05:00
Stephen Haberman
c3f1675f9c
Retrieve jars to a flat directory so * can be used for the classpath.
2013-01-08 14:44:33 -06:00
Tathagata Das
64dceec293
Merge branch 'streaming-merge' into dev-merge
2013-01-07 16:54:35 -08:00
Shivaram Venkataraman
aed368a970
Update Hadoop dependency to 1.0.3 as 0.20 has Sun specific dependencies. Also
...
fix SequenceFileRDDFunctions to pick the right type conversion across Hadoop
versions
2013-01-07 15:57:33 -08:00
Tathagata Das
af8738dfb5
Moved Spark Streaming examples to examples sub-project.
2013-01-06 19:31:54 -08:00
Patrick Wendell
518111573f
Merge pull request #8 from radlab/twitter-example
...
Adding a Twitter InputDStream with an example
2012-12-29 14:23:01 -08:00
Tathagata Das
7c33f76291
Merge branch 'mesos' into dev-merge
2012-12-26 19:19:07 -08:00
Patrick Wendell
9ac4cb1c5f
Adding a Twitter InputDStream with an example
2012-12-21 17:18:19 -08:00
Matei Zaharia
3334b7c6b5
Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677
...
Kryo2 update against Spark master
2012-12-21 15:31:23 -08:00
Reynold Xin
eac566a7f4
Merge branch 'master' of github.com:mesos/spark into dev
...
Conflicts:
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/PairRDDFunctions.scala
core/src/main/scala/spark/ParallelCollection.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/CartesianRDD.scala
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/CoalescedRDD.scala
core/src/main/scala/spark/rdd/FilteredRDD.scala
core/src/main/scala/spark/rdd/FlatMappedRDD.scala
core/src/main/scala/spark/rdd/GlommedRDD.scala
core/src/main/scala/spark/rdd/HadoopRDD.scala
core/src/main/scala/spark/rdd/MapPartitionsRDD.scala
core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala
core/src/main/scala/spark/rdd/MappedRDD.scala
core/src/main/scala/spark/rdd/PipedRDD.scala
core/src/main/scala/spark/rdd/SampledRDD.scala
core/src/main/scala/spark/rdd/ShuffledRDD.scala
core/src/main/scala/spark/rdd/UnionRDD.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockManagerId.scala
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/MetadataCleaner.scala
core/src/main/scala/spark/util/TimeStampedHashMap.scala
core/src/test/scala/spark/storage/BlockManagerSuite.scala
run
2012-12-20 14:53:40 -08:00
Reynold Xin
9397c5014e
Let the slave notify the master block removal.
2012-12-20 01:37:09 -08:00
Patrick Wendell
3ff9710265
Adding Flume InputDStream
2012-12-07 16:42:39 -08:00
Denny
556c38ed91
Added kafka JAR
2012-12-05 11:54:42 -08:00
Denny
0c1de43fc7
Working on kafka.
2012-11-06 09:41:42 -08:00
Matei Zaharia
863a55ae42
Merge remote-tracking branch 'public/master' into dev
...
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
core/src/main/scala/spark/KryoSerializer.scala
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/executor/Executor.scala
core/src/main/scala/spark/network/Connection.scala
core/src/main/scala/spark/network/ConnectionManagerTest.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/NewHadoopRDD.scala
core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockMessage.scala
core/src/main/scala/spark/storage/BlockStore.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/AkkaUtils.scala
project/SparkBuild.scala
run
2012-10-24 23:21:00 -07:00
Matei Zaharia
0967e71a00
Bump up version to 0.7.0-SNAPSHOT for master branch
2012-10-22 11:49:42 -07:00
Matei Zaharia
902a608187
Update version to 0.6.1-SNAPSHOT to show this is in development
2012-10-22 11:43:57 -07:00
Thomas Dudziak
d9c2a89c57
Support for Hadoop 2 distributions such as cdh4
2012-10-18 16:08:54 -07:00
Reynold Xin
4a3fb06ac2
Updated Kryo to 2.20.
2012-10-16 01:10:01 -07:00
Patrick Wendell
629dd2691e
Removing credentials line in build.
2012-10-14 19:33:39 -07:00
Matei Zaharia
f8768da418
Comment out PGP stuff for publish-local to work
2012-10-14 17:37:21 -07:00
Matei Zaharia
64b52166ee
Changed default Hadoop version back to 0.20.205
2012-10-14 09:51:34 -07:00
Matei Zaharia
ce6b5a3ee5
Uncomment Maven publishing stuff and set version to 0.6.0
2012-10-13 15:55:39 -07:00
Patrick Wendell
6d328f54d0
Changing tabs to spaces
2012-10-10 18:54:22 -07:00
Patrick Wendell
3ed172ea59
Adding code for publishing to Sonatype.
...
By default - I'm leaving this commented out. This is because
there is a bug in the PGP signing plugin which causes it to active
even duing a publish-local. So we'll just uncomment when we decide
to publish.
2012-10-10 17:25:29 -07:00
Andy Konwinski
5897567679
Removes the included mesos-0.9.0.jar and adds a libraryDependency to
...
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
2012-10-03 08:58:05 -07:00
Matei Zaharia
6112b1a83c
Don't build an assembly for the REPL
2012-10-02 17:08:16 -07:00
Matei Zaharia
a925754675
Place Spray in front of Cloudera in Maven search path
2012-10-02 12:02:00 -07:00
Matei Zaharia
22684653a5
Revert "Place Spray repo ahead of Cloudera in Maven search path"
...
This reverts commit 42e0a68082
.
2012-10-02 12:01:32 -07:00
Matei Zaharia
42e0a68082
Place Spray repo ahead of Cloudera in Maven search path
2012-10-02 11:37:19 -07:00
Patrick Wendell
6fee76d6d5
publish-local should go to maven + ivy by default
2012-10-01 15:34:47 -07:00
Reynold Xin
5783236ae6
Added a new command "pl" in sbt to publish to both Maven and Ivy.
2012-10-01 00:17:13 -07:00
Matei Zaharia
35cc9f13e9
Update Akka to 2.0.3
2012-09-24 14:17:10 -07:00
Matei Zaharia
1f539aa473
Update Scala version dependency to 2.9.2
2012-09-24 14:12:48 -07:00
Tathagata Das
7419d2c7ea
Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project.
2012-09-02 02:35:17 -07:00
Matei Zaharia
5a8015d2db
Merge remote-tracking branch 'public/dev' into dev
2012-08-24 16:11:44 -07:00
Denny
0008994044
merged dev branch
2012-08-02 16:00:33 -07:00
Matei Zaharia
71a958b0b7
Merge branch 'dev' of github.com:mesos/spark into dev
...
Conflicts:
project/SparkBuild.scala
2012-08-02 17:23:13 -04:00
Denny
ba7e30fb5e
Mostly stlyistic changes.
2012-08-02 13:55:09 -07:00
Josh Rosen
039b41cb54
Use sbt mergeStrategy for reference.conf files.
...
Cleans up #158 / 509b721
.
2012-08-02 10:21:50 -07:00
Denny
0ee44c225e
Spark standalone mode cluster scripts.
...
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00
Denny
1b29e90a79
merge dev branch
2012-08-01 14:06:09 -07:00
Denny
7a295fee96
Spark WebUI Implementation.
2012-08-01 11:01:09 -07:00
Josh Rosen
509b721d12
Fix Akka configuration in assembly jar.
...
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.
This solution is from the Akka Team Blog:
http://letitcrash.com/post/21025950392/
2012-07-30 18:05:13 -07:00
Matei Zaharia
47b7ebad12
Added the Spark Streaing code, ported to Akka 2
2012-07-28 20:03:26 -07:00
Josh Rosen
01dce3f569
Add Java API
...
Add distinct() method to RDD.
Fix bug in DoubleRDDFunctions.
2012-07-18 17:34:29 -07:00