ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
5271 commits 6 branches 181 tags 397 MiB
Commit graph

372 commits

Author SHA1 Message Date
Matei Zaharia 668b0dc6a7 Merge branch 'master' of github.com:mesos/spark 2013-07-13 19:10:46 -07:00
Matei Zaharia cd28d9c147 Merge remote-tracking branch 'origin/pr/662' 
Conflicts:
	bin/compute-classpath.sh
 2013-07-13 19:10:00 -07:00
seanm c4d5b01e44 changing com.google.code.findbugs maven coordinates 2013-07-13 14:56:23 -06:00
Prashant Sharma e86d5dbaad Merge branch 'master' into master-merge 
Conflicts:
	README.md
	core/pom.xml
	core/src/main/scala/spark/deploy/JsonProtocol.scala
	core/src/main/scala/spark/deploy/LocalSparkCluster.scala
	core/src/main/scala/spark/deploy/master/Master.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
	project/SparkBuild.scala
	streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
 2013-07-12 14:49:16 +05:30
Prashant Sharma 69ae7ea227 Removed some unnecessary code and fixed dependencies 2013-07-11 18:30:18 +05:30
Matei Zaharia 3cc6818f13 Merge pull request #668 from shimingfei/guava-14.0.1 
update guava version from 11.0.1 to 14.0.1
 2013-07-06 19:51:20 -07:00
Matei Zaharia 1ffadb2d9e Merge remote-tracking branch 'pwendell/ui-updates' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
 2013-07-06 15:51:41 -07:00
Matei Zaharia 43b24635ee Renamed ML package to MLlib and added it to classpath 2013-07-05 11:38:53 -07:00
Matei Zaharia 05be233ce2 Removed dependency on Apache Commons Math 2013-07-05 11:13:46 -07:00
Reynold Xin 6a9a9a364c Minor clean up of the RidgeRegression code. I am not even sure why I did 
this :s.
 2013-07-05 11:13:45 -07:00
Matei Zaharia 729e463f64 Import RidgeRegression example 
Conflicts:
	run
 2013-07-05 11:13:41 -07:00
Gavin Li 94238aae57 fix dependencies 2013-07-03 18:08:38 +00:00
Mingfei 04567a1771 update guava version from 11.0.1 to 14.0.1 2013-07-03 17:43:37 +08:00
Prashant Sharma a5f1f6a907 Merge branch 'master' into master-merge 
Conflicts:
	core/pom.xml
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/RDDCheckpointData.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/Utils.scala
	core/src/main/scala/spark/api/python/PythonRDD.scala
	core/src/main/scala/spark/deploy/client/Client.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/ZippedRDD.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/BlockManagerMasterActor.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	core/src/test/scala/spark/SizeEstimatorSuite.scala
	pom.xml
	project/SparkBuild.scala
	repl/src/main/scala/spark/repl/SparkILoop.scala
	repl/src/test/scala/spark/repl/ReplSuite.scala
	streaming/src/main/scala/spark/streaming/StreamingContext.scala
	streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala
	streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
	streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
 2013-07-03 11:43:26 +05:30
Matei Zaharia 5cfcd3c336 Remove Twitter4J specific repo since it's in Maven central 2013-06-29 15:37:27 -07:00
Evan Chan 1107b4d55b Merge branch 'master' into 2013-06/assembly-jar-deploy 
Conflicts:
	run

Previous changes that I made to run and set-dev-classpath.sh instead
have been folded into compute-classpath.sh
 2013-06-28 17:18:35 -07:00
Matei Zaharia 32370da4e4 Don't use forward slash in exclusion for JAR signature files 2013-06-25 22:08:19 -04:00
Evan Chan d2f46ac680 Merge branch 'master' into 2013-06/assembly-jar-deploy 
Conflicts:
	run
 2013-06-25 14:50:16 -07:00
Tathagata Das c89af0a7f9 Merge branch 'master' into streaming 
Conflicts:
	.gitignore
 2013-06-24 23:57:47 -07:00
Patrick Wendell 91ec5a1a04 Changing JSON protocol and removing spray code 2013-06-22 10:31:36 -07:00
Matei Zaharia b350f34703 Increase memory for tests to prevent a crash on JDK 7 2013-06-22 07:48:20 -07:00
Evan Chan 071ff7efa1 Enable building a fat jar for the Spark REPL 2013-06-20 17:53:23 -07:00
Matei Zaharia ae7a5da6b3 Fix some dependency issues in SBT build (same will be needed for Maven): 
- Exclude a version of ASM 3.x that comes from HBase
- Don't use a special ASF repo for HBase
- Update SLF4J version
- Add sbt-dependency-graph plugin so we can easily find dependency trees
 2013-06-20 18:44:46 +02:00
Matei Zaharia 7902baddc7 Update ASM to version 4.0 2013-06-19 13:34:30 +02:00
Matei Zaharia dbfab49d2a Merge remote-tracking branch 'milliondreams/casdemo' 
Conflicts:
	project/SparkBuild.scala
 2013-06-18 14:55:31 +02:00
Matei Zaharia 73f4c7d2d1 Merge pull request #605 from esjewett/SPARK-699 
Add hBase example (retry of pull request #596)
 2013-06-18 04:21:17 -07:00
Matei Zaharia 2ab311f4ce Removed second version of junit test plugin from plugins.sbt 2013-06-18 00:40:25 +02:00
Christopher Nguyen 479442a9b9 Add zeroLengthPartitions() test to make sure, e.g., StatCounter.scala can handle empty partitions without incorrectly returning NaN 2013-06-15 17:35:55 -07:00
Matei Zaharia 5b5b5aedbf Fixed a few test issues due to Akka 2.1, as well as SBT memory. 
Unfortunately, in Akka 2.1, ActorSystem.awaitTermination hangs for
remote actors, and Akka also leaves a non-daemon Netty thread even when
run in daemon mode. Thus I had to comment out some of the calls to
awaitTermination, and we still have one failing test.
 2013-06-08 01:09:24 -07:00
Rohit Rai 6d8423fd1b Adding deps to examples/pom.xml 
Fixing exclusion in examples deps in SparkBuild.scala
 2013-06-02 13:03:45 +05:30
Rohit Rai 3be7bdcefd Adding example to make Spark RDD from Cassandra 2013-06-01 19:32:17 +05:30
Reynold Xin f742435f18 Removed the duplicated netty dependency in SBT build file. 2013-05-16 14:31:03 -07:00
Reynold Xin f3491cb89b Merge branch 'master' of github.com:mesos/spark into shufflemerge 
Conflicts:
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/test/scala/spark/DistributedSuite.scala
	project/SparkBuild.scala
 2013-05-15 00:31:52 -07:00
Reynold Xin 81ad2fa331 Merge branch 'jdbc' of github.com:koeninger/spark 
Conflicts:
	project/SparkBuild.scala
 2013-05-14 23:12:00 -07:00
Cody Koeninger b16c4896f6 add test for JdbcRDD using embedded derby, per rxin suggestion 2013-05-14 23:44:04 -05:00
Ethan Jewett ee6f6aa6cd Add hBase example 2013-05-09 18:33:38 -05:00
Reynold Xin 012c9e5ab0 Revert "Merge pull request #596 from esjewett/master" because the 
dependency on hbase introduces netty-3.2.2 which conflicts with
netty-3.5.3 already in Spark. This caused multiple test failures.

This reverts commit 0f1b7a06e1, reversing
changes made to aacca1b8a8.
 2013-05-09 14:20:01 -07:00
Reynold Xin 90577ada69 Merge branch 'shuffle-performance-fix-0.7' of github.com:shane-huang/spark into shufflemerge 
Conflicts:
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/DiskStore.scala
	project/SparkBuild.scala
 2013-05-07 15:56:19 -07:00
Ethan Jewett 02e8cfa617 HBase example 2013-05-04 12:31:30 -05:00
Jey Kottalam 207afe4088 Remove spark-repl's extraneous dependency on spark-streaming 2013-05-01 16:57:31 -07:00
Prashant Sharma 4041a2689e Updated to latest stable scala 2.10.1 and akka 2.1.2 2013-05-01 11:35:35 +05:30
Matei Zaharia f1f92c88eb Build against Hadoop 1 by default 2013-04-29 17:08:45 -07:00
Prashant Sharma 4b4a36ea7d Fixed pom.xml with updated dependencies. 2013-04-29 12:55:43 +05:30
Matei Zaharia 1b169f190c Exclude old versions of Netty, which had a different Maven organization 2013-04-25 19:52:12 -07:00
Matei Zaharia eef9ea1993 Update unit test memory to 2 GB 2013-04-25 00:42:29 -07:00
Matei Zaharia 01d9ba5038 Add back line removed during YARN merge 2013-04-25 00:11:27 -07:00
Prashant Sharma ad88f083a6 scala 2.10 and master merge 2013-04-24 18:08:26 +05:30
Mridul Muralidharan 3b594a4e3b Do not add signature files - results in validation errors when using assembled file 2013-04-24 10:18:25 +05:30
Mridul Muralidharan dd515ca3ee Attempt at fixing merge conflict 2013-04-24 09:24:17 +05:30
Mridul Muralidharan adcda84f96 Pull latest SparkBuild.scala from master and merge conflicts 2013-04-24 08:57:25 +05:30
Mridul Muralidharan 5b85c715c8 Revert back to 2.0.2-alpha : 0.23.7 has protocol changes which break against cloudera 2013-04-24 02:57:51 +05:30
Mridul Muralidharan 8faf5c51c3 Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch 2013-04-24 02:31:57 +05:30
Prashant Sharma 185bb9525a Manually merged scala-2.10 and master 2013-04-22 14:14:03 +05:30
Mridul Muralidharan 7acab3ab45 Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo 2013-04-22 08:01:13 +05:30
Matei Zaharia 17e076de80 Turn on forking in test JVMs to reduce the pressure on perm gen and code 
cache sizes due to having 2 instances of the Scala compiler and a bunch
of classloaders.
 2013-04-18 22:25:57 -07:00
Mridul Muralidharan 5d891534fd Move back to 2.0.2-alpha, since 2.0.3-alpha is not available in cloudera yet. Also, add netty dependency explicitly to prevent resolving to older 2.3x version. Additionally, comment out retrievePattern to ensure correct netty is picked up 2013-04-17 05:54:43 +05:30
Matei Zaharia ec5e553b41 Merge pull request #558 from ash211/patch-jackson-conflict 
Don't pull in old versions of Jackson via hadoop-core
 2013-04-14 08:20:13 -07:00
Matei Zaharia ed336e0d44 Fix tests from different projects running in parallel in SBT 0.12 2013-04-11 22:29:37 -04:00
Prashant Sharma 9f26318bbd Fixed previously removed dependencies 2013-04-10 14:46:42 +05:30
Andrew Ash 18bd41d1a3 Don't pull in old versions of Jackson via hadoop-core 2013-04-09 14:44:47 -04:00
Matei Zaharia 65caa8f711 Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0' 
Conflicts:
	docs/_config.yml
	project/SparkBuild.scala
 2013-04-08 12:43:17 -04:00
Matei Zaharia 1cb3eb9762 Merge remote-tracking branch 'kalpit/master' 
Conflicts:
	project/SparkBuild.scala
 2013-04-07 20:54:18 -04:00
Matei Zaharia b362df39ea Merge pull request #552 from MLnick/master 
Bumping version for Twitter Algebird to latest
 2013-04-07 17:17:52 -07:00
Mridul Muralidharan 6798a09df8 Add support for building against hadoop2-yarn : adding new maven profile for it 2013-04-07 17:47:38 +05:30
shane-huang df47b40b76 Shuffle Performance fix: Use netty embeded OIO file server instead of ConnectionManager 
Shuffle Performance Optimization: do not send 0-byte block requests to reduce network messages
change reference from io.Source to scala.io.Source to avoid looking into io.netty package

Signed-off-by: shane-huang <shengsheng.huang@intel.com>
 2013-04-07 14:37:12 +08:00
Andy Konwinski 5555811bd5 Update build to Scala 2.9.3 2013-04-04 13:26:45 -07:00
Nick Pentreath 0f54344fd8 Bumping Algebird version in examples now that it supports JDK 1.6 2013-04-03 13:15:34 +02:00
Jey Kottalam bc8ba222ff Bump development version to 0.8.0 2013-03-28 15:42:01 -07:00
kalpit f0164e5047 upgraded sbt version, sbt plugins and some library dependencies to latest stable version 2013-03-26 17:49:29 -07:00
Holden Karau 8456d673e2 Re-enable deprecation warnings since there are only two 2013-03-24 17:30:23 -07:00
Holden Karau e104a76016 Makes the syntax highlighting on the build file not broken in emacs. 2013-03-24 16:16:05 -07:00
Prashant Sharma 15530c2b23 porting of repl to scala-2.10 2013-03-17 10:47:17 +05:30
seanm 42822cf95d changing streaming resolver for akka 2013-03-13 11:40:42 -06:00
seanm 4aa1205202 adding typesafe repo to streaming resolvers so that akka-zeromq is found 2013-03-11 12:37:29 -06:00
Hiral Patel 664e5fd24b Fix reference bug in Kryo serializer, add test, update version 2013-03-07 22:16:11 -08:00
Matei Zaharia db9b90fdbd Change version to 0.7.1-SNAPSHOT for development branch 2013-02-27 09:15:26 -08:00
Matei Zaharia 7e67c626ee Change version number to 0.7.0 2013-02-25 20:30:47 -08:00
Matei Zaharia 6494cab19d Update Hadoop dependency to 1.0.4 2013-02-25 15:38:21 -08:00
Prashant Sharma 254acb1666 Moving akka dependency resolver to shared. 2013-02-25 13:37:07 +05:30
Tathagata Das 5ab37be983 Fixed class paths and dependencies based on Matei's comments. 2013-02-24 16:24:52 -08:00
Tathagata Das f282bc4960 Changed Algebird from 0.1.9 to 0.1.8 2013-02-24 12:44:12 -08:00
Tathagata Das 24c0cd6168 Fixed resolver for akka-zeromq 2013-02-22 18:23:29 -08:00
Tathagata Das cfa65ebff1 Merge pull request #480 from MLnick/streaming-eg-algebird 
[Streaming] Examples using Twitter's Algebird library
 2013-02-22 12:29:04 -08:00
Nick Pentreath 718474b9c6 Bumping Algebird to 0.1.9 2013-02-21 12:11:31 +02:00
Prashant Sharma 4e5b09664c fixes corresponding to review feedback at pull request #479 2013-02-20 19:14:52 +05:30
Prashant Sharma f7d3e309cb ZeroMQ stream as receiver 2013-02-19 19:32:52 +05:30
Nick Pentreath 315ea069e8 Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird 
Conflicts:
	project/SparkBuild.scala
 2013-02-19 13:58:05 +02:00
Nick Pentreath 015893f0e8 Adding streaming HyperLogLog example using Algebird 2013-02-19 13:21:33 +02:00
Tathagata Das def8126d77 Added TwitterInputDStream from example to StreamingContext. Renamed example TwitterBasic to TwitterPopularTags. 2013-02-14 17:49:43 -08:00
Charles Reiss 0f81025eca Add easymock to SBT configuration. 2013-01-29 18:55:42 -08:00
Matei Zaharia 6e3754bf47 Add Maven build file for streaming, and fix some issues in SBT file 
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
 2013-01-20 19:22:24 -08:00
Tathagata Das cd1521cfdb Merge branch 'master' into streaming 
Conflicts:
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/rdd/FilteredRDD.scala
	docs/_layouts/global.html
	docs/index.md
	run
 2013-01-15 12:08:51 -08:00
folone 25c0739bad Moved to scala 2.10.0. Notable changes are: 
- akka  2.0.3  → 2.1.0
- spray 1.0-M1 → 1.1-M7
For now the repl subproject is commented out, as scala reflection api changed very much since the introduction of macros.
 2013-01-14 09:52:11 +01:00
Matei Zaharia 6d1c230281 Merge pull request #357 from tysonjh/master 
JSON support added to WebUI
 2013-01-10 19:06:07 -08:00
Tyson 549ee388a1 Removed io.spray spray-json dependency as it is not needed. 2013-01-09 15:12:23 -05:00
Tyson 6e8c8f61c4 Added the spray implicit marshaller library 
Added the io.spray JSON library
 2013-01-09 10:40:33 -05:00
Stephen Haberman c3f1675f9c Retrieve jars to a flat directory so * can be used for the classpath. 2013-01-08 14:44:33 -06:00
Tathagata Das 64dceec293 Merge branch 'streaming-merge' into dev-merge 2013-01-07 16:54:35 -08:00
Shivaram Venkataraman aed368a970 Update Hadoop dependency to 1.0.3 as 0.20 has Sun specific dependencies. Also 
fix SequenceFileRDDFunctions to pick the right type conversion across Hadoop
versions
 2013-01-07 15:57:33 -08:00
Tathagata Das af8738dfb5 Moved Spark Streaming examples to examples sub-project. 2013-01-06 19:31:54 -08:00
Patrick Wendell 518111573f Merge pull request #8 from radlab/twitter-example 
Adding a Twitter InputDStream with an example
 2012-12-29 14:23:01 -08:00
Tathagata Das 7c33f76291 Merge branch 'mesos' into dev-merge 2012-12-26 19:19:07 -08:00
Patrick Wendell 9ac4cb1c5f Adding a Twitter InputDStream with an example 2012-12-21 17:18:19 -08:00
Matei Zaharia 3334b7c6b5 Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677 
Kryo2 update against Spark master
 2012-12-21 15:31:23 -08:00
Reynold Xin eac566a7f4 Merge branch 'master' of github.com:mesos/spark into dev 
Conflicts:
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/PairRDDFunctions.scala
	core/src/main/scala/spark/ParallelCollection.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/CartesianRDD.scala
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/rdd/CoalescedRDD.scala
	core/src/main/scala/spark/rdd/FilteredRDD.scala
	core/src/main/scala/spark/rdd/FlatMappedRDD.scala
	core/src/main/scala/spark/rdd/GlommedRDD.scala
	core/src/main/scala/spark/rdd/HadoopRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsRDD.scala
	core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala
	core/src/main/scala/spark/rdd/MappedRDD.scala
	core/src/main/scala/spark/rdd/PipedRDD.scala
	core/src/main/scala/spark/rdd/SampledRDD.scala
	core/src/main/scala/spark/rdd/ShuffledRDD.scala
	core/src/main/scala/spark/rdd/UnionRDD.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerId.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/MetadataCleaner.scala
	core/src/main/scala/spark/util/TimeStampedHashMap.scala
	core/src/test/scala/spark/storage/BlockManagerSuite.scala
	run
 2012-12-20 14:53:40 -08:00
Reynold Xin 9397c5014e Let the slave notify the master block removal. 2012-12-20 01:37:09 -08:00
Patrick Wendell 3ff9710265 Adding Flume InputDStream 2012-12-07 16:42:39 -08:00
Denny 556c38ed91 Added kafka JAR 2012-12-05 11:54:42 -08:00
Denny 0c1de43fc7 Working on kafka. 2012-11-06 09:41:42 -08:00
Matei Zaharia 863a55ae42 Merge remote-tracking branch 'public/master' into dev 
Conflicts:
	core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
	core/src/main/scala/spark/KryoSerializer.scala
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/executor/Executor.scala
	core/src/main/scala/spark/network/Connection.scala
	core/src/main/scala/spark/network/ConnectionManagerTest.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/NewHadoopRDD.scala
	core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockMessage.scala
	core/src/main/scala/spark/storage/BlockStore.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	project/SparkBuild.scala
	run
 2012-10-24 23:21:00 -07:00
Matei Zaharia 0967e71a00 Bump up version to 0.7.0-SNAPSHOT for master branch 2012-10-22 11:49:42 -07:00
Matei Zaharia 902a608187 Update version to 0.6.1-SNAPSHOT to show this is in development 2012-10-22 11:43:57 -07:00
Thomas Dudziak d9c2a89c57 Support for Hadoop 2 distributions such as cdh4 2012-10-18 16:08:54 -07:00
Reynold Xin 4a3fb06ac2 Updated Kryo to 2.20. 2012-10-16 01:10:01 -07:00
Patrick Wendell 629dd2691e Removing credentials line in build. 2012-10-14 19:33:39 -07:00
Matei Zaharia f8768da418 Comment out PGP stuff for publish-local to work 2012-10-14 17:37:21 -07:00
Matei Zaharia 64b52166ee Changed default Hadoop version back to 0.20.205 2012-10-14 09:51:34 -07:00
Matei Zaharia ce6b5a3ee5 Uncomment Maven publishing stuff and set version to 0.6.0 2012-10-13 15:55:39 -07:00
Patrick Wendell 6d328f54d0 Changing tabs to spaces 2012-10-10 18:54:22 -07:00
Patrick Wendell 3ed172ea59 Adding code for publishing to Sonatype. 
By default - I'm leaving this commented out. This is because
there is a bug in the PGP signing plugin which causes it to active
even duing a publish-local. So we'll just uncomment when we decide
to publish.
 2012-10-10 17:25:29 -07:00
Andy Konwinski 5897567679 Removes the included mesos-0.9.0.jar and adds a libraryDependency to 
the build file so that mesos-0.9.0-incubating.jar (which contains the
same class files, but has a silightly different name) will be pulled
down from Maven Central instead.
 2012-10-03 08:58:05 -07:00
Matei Zaharia 6112b1a83c Don't build an assembly for the REPL 2012-10-02 17:08:16 -07:00
Matei Zaharia a925754675 Place Spray in front of Cloudera in Maven search path 2012-10-02 12:02:00 -07:00
Matei Zaharia 22684653a5 Revert "Place Spray repo ahead of Cloudera in Maven search path" 
This reverts commit 42e0a68082.
 2012-10-02 12:01:32 -07:00
Matei Zaharia 42e0a68082 Place Spray repo ahead of Cloudera in Maven search path 2012-10-02 11:37:19 -07:00
Patrick Wendell 6fee76d6d5 publish-local should go to maven + ivy by default 2012-10-01 15:34:47 -07:00
Reynold Xin 5783236ae6 Added a new command "pl" in sbt to publish to both Maven and Ivy. 2012-10-01 00:17:13 -07:00
Matei Zaharia 35cc9f13e9 Update Akka to 2.0.3 2012-09-24 14:17:10 -07:00
Matei Zaharia 1f539aa473 Update Scala version dependency to 2.9.2 2012-09-24 14:12:48 -07:00
Tathagata Das 7419d2c7ea Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project. 2012-09-02 02:35:17 -07:00
Matei Zaharia 5a8015d2db Merge remote-tracking branch 'public/dev' into dev 2012-08-24 16:11:44 -07:00
Denny 0008994044 merged dev branch 2012-08-02 16:00:33 -07:00
Matei Zaharia 71a958b0b7 Merge branch 'dev' of github.com:mesos/spark into dev 
Conflicts:
	project/SparkBuild.scala
 2012-08-02 17:23:13 -04:00
Denny ba7e30fb5e Mostly stlyistic changes. 2012-08-02 13:55:09 -07:00
Josh Rosen 039b41cb54 Use sbt mergeStrategy for reference.conf files. 
Cleans up #158 / 509b721.
 2012-08-02 10:21:50 -07:00
Denny 0ee44c225e Spark standalone mode cluster scripts. 
Heavily inspired by Hadoop cluster scripts ;-)
 2012-08-01 20:38:52 -07:00
Denny 1b29e90a79 merge dev branch 2012-08-01 14:06:09 -07:00
Denny 7a295fee96 Spark WebUI Implementation. 2012-08-01 11:01:09 -07:00
Josh Rosen 509b721d12 Fix Akka configuration in assembly jar. 
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.

This solution is from the Akka Team Blog:

http://letitcrash.com/post/21025950392/
 2012-07-30 18:05:13 -07:00
Matei Zaharia 47b7ebad12 Added the Spark Streaing code, ported to Akka 2 2012-07-28 20:03:26 -07:00
Josh Rosen 01dce3f569 Add Java API 
Add distinct() method to RDD.

Fix bug in DoubleRDDFunctions.
 2012-07-18 17:34:29 -07:00
Matei Zaharia 2fb6e7d71e Initial framework to get a master and web UI up. 2012-06-30 14:45:55 -07:00
Matei Zaharia c53670b9bf Various code style fixes, mostly from IntelliJ IDEA 2012-06-29 18:47:12 -07:00
rrmckinley 697b0bee2c Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24. Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too. 2012-06-29 16:42:05 -07:00
Matei Zaharia 3920189932 Upgraded to Akka 2 and fixed test execution (which was still parallel 
across projects).
 2012-06-28 23:51:28 -07:00
Tathagata Das ede615d719 Fixed issues duplicate class issues in sbt assembly. 2012-06-22 15:03:09 -07:00
Matei Zaharia 3e0396c953 Update SBT and SBT-Eclipse version 2012-06-17 14:37:18 -07:00
Matei Zaharia 08579ffa11 Update version number for dev branch 2012-06-15 23:55:43 -07:00
Matei Zaharia f58da6164e Merge branch 'master' into dev 2012-06-15 23:47:11 -07:00
Matei Zaharia 4449eb9783 Changed version in master branch to 0.5.1-SNAPSHOT for further 
development.
 2012-06-13 22:26:14 -04:00
Matei Zaharia 0472cf8e0d Update version in SBT 2012-06-12 14:30:49 -04:00
Matei Zaharia 63051dd2bc Merge in engine improvements from the Spark Streaming project, developed 
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
 2012-06-07 12:45:38 -07:00
Matei Zaharia 95fb1a16b8 Use Mesos 0.9 RC3 JAR and protobuf 2.4.1 2012-03-30 11:38:49 -04:00
Matei Zaharia 4d52cc6738 Merge branch 'master' into mesos-0.9 2012-03-29 21:29:39 -04:00
Matei Zaharia ca5c19c1ba Remove dependency on Akka 2012-03-29 01:03:34 -04:00
Reynold Xin 90418b70ff Added sbt-assembly for spark-repl project so we can generate an 
assembled jar for Shark.
 2012-03-22 18:46:31 -07:00
Matei Zaharia a099a63a8a Initial work to make Spark compile with Mesos 0.9 and Hadoop 1.0 2012-03-17 12:31:34 -07:00
Matei Zaharia d6ec664b48 Add dependency on fastutil and update Guava 2012-02-06 15:37:27 -08:00
Matei Zaharia 5fd101d79e Add dependency on Akka and Netty 2011-12-15 13:21:14 +01:00
Matei Zaharia c7d6f1a65c Really upgrade to SBT 0.11.1 (through build.properties and plugin changes) 2011-11-08 21:45:29 -08:00
root 49505a0b0b Switched Jetty to version 7.5 because 8.0 was causing a conflict with the log4j and Jetty libraries in Hadoop. 2011-10-17 18:06:41 +00:00
Ismael Juma d76c0fc781 Upgrade to sbt-idea 0.11.0 final. 2011-09-27 23:13:38 +01:00
Ismael Juma 7e92ef9d19 Add workaround for bug in SBT (issue #206). 2011-09-27 00:04:59 +01:00
Ismael Juma 3562db6374 Include "spark-" prefix in project name (used when artifact is published). 2011-09-26 22:41:07 +01:00
Ismael Juma 28b5d5a2af Upgrade compress-lzf to 0.8.4. 2011-09-26 22:32:05 +01:00
Ismael Juma 315e55fde3 Upgrade Jetty to 8.0.1. 2011-09-26 22:32:05 +01:00
Ismael Juma ee980439e2 Use scalatest and scalacheck compiled against Scala 2.9.1. 2011-09-26 22:32:05 +01:00
Ismael Juma bd774eb274 Use new layout for plugins definitions (recommended for SBT 0.11) 2011-09-26 22:32:05 +01:00
Ismael Juma e39edcce60 Upgrade to SBT 0.11.0. 2011-09-26 22:24:29 +01:00
Ismael Juma 483f724d62 Upgrade to Scala 2.9.1. 
Interestingly, the version in Maven is 2.9.1, but SBT outputs file to the 2.9.1.final
directory inside target.

A couple of small changes in SparkIMain were also required.

All tests pass and ./spark-shell launches successfully.
 2011-08-31 10:43:05 +01:00
Matei Zaharia 9b7215d74a Only include JAR files in lib in the unmanaged classpath. 2011-08-29 22:59:18 -07:00
Matei Zaharia 4cc5e51e28 Use the new SBT 0.10 assembly plugin instead of the DepJar code we had. 2011-08-29 16:13:26 -07:00
Ismael Juma 2b6fd3198d Fix issue #69: Enable -optimize in the build 2011-08-02 10:34:41 +01:00
Ismael Juma 59c7131dff Add note about why we can't enable -deprecation switch. 2011-08-02 10:26:17 +01:00
Matei Zaharia 2e57338896 Merge branch 'scala-2.9' 2011-08-01 15:27:08 -07:00
Matei Zaharia 711575391d Merge branch 'scala-2.9' 
Conflicts:
	project/build/SparkProject.scala
 2011-08-01 15:25:26 -07:00
Ismael Juma 3f5d7b5d11 Add publishTo configuration that publishes to target directory. 
Until Spark is available in a Maven repository, this makes it easy to deploy to
e.g. GitHub pages by copying target/maven to the repository.
 2011-07-31 20:17:58 +01:00
Ismael Juma 7565cc2e33 Add type annotation to result of depJarSettings to workaround scalac bug. 2011-07-31 20:17:58 +01:00
Matei Zaharia d12122502b Various improvements to Kryo serializer: 
- Replaced modified Kryo version with the standard one augmented with
  the kryo-serializers package, which includes support for classes with
  no-arg constructors (that was why we had a modified Kryo before)
- The kryo-serializers version also fixes issue #72.
- Added a bunch of tests.
- Serialize maps and a few other common types properly by default.
 2011-07-21 22:09:33 -07:00
Matei Zaharia 2bfd7931e8 Merge branch 'new-rdds-protobuf' 
Conflicts:
	core/src/main/scala/spark/Executor.scala
	core/src/main/scala/spark/RDD.scala
 2011-07-21 16:08:39 -07:00
Ismael Juma fc0a2c8db8 Add and configure junit_xml_listener as a replacement for XmlTestReport. 2011-07-21 01:04:29 +01:00
Ismael Juma 51673ca62e Introduce DepJarPlugin based on AssemblyPlugin and use it in SparkBuild. 2011-07-18 10:34:51 +01:00
Ismael Juma c71fc27c74 Update sbt-idea to non-snapshot version and uncomment sbt-assembly dependency. 2011-07-18 00:17:21 +01:00
Ismael Juma 8531c2a079 Update test dependencies. 2011-07-18 00:16:08 +01:00
Ismael Juma 635f501492 Fix copy & paste error in version. 2011-07-18 00:13:37 +01:00
Ismael Juma f686e3dacb Initial work on converting build to SBT 0.10.1 2011-07-15 03:38:30 +01:00
Matei Zaharia cf8f5de61b Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
	repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
 2011-07-14 17:48:56 -04:00
Matei Zaharia 02678724a4 Update version number to 0.4-SNAPSHOT 2011-07-14 17:47:39 -04:00
Matei Zaharia 7c77b2fa6a Merge branch 'master' into scala-2.9 
Conflicts:
	project/build.properties
 2011-07-14 17:39:34 -04:00
Matei Zaharia c86af80022 Change version to 0.3 2011-07-14 17:38:43 -04:00
Matei Zaharia d05fea24f3 Simplified parallel shuffle fetcher to use URLConnection 2011-07-11 22:12:36 -04:00
Matei Zaharia aea5cb4413 Added parallel shuffle fetcher 2011-07-09 17:25:56 -04:00
Matei Zaharia c62bb4091b Merge remote-tracking branch 'origin/master' into scala-2.9 2011-06-07 00:42:23 -07:00
Ismael Juma 1ad4dcd3de Move managedStyle to SparkProject. 
I had added it to DepJar by mistake.
 2011-06-02 14:06:54 +01:00
Ismael Juma 3def9fdb96 Upgrade to scalacheck 1.9. 2011-05-31 22:11:33 -07:00
Matei Zaharia beb9c117f0 Merge branch 'master' into scala-2.9 
Conflicts:
	project/build/SparkProject.scala
 2011-05-31 19:23:07 -07:00
Matei Zaharia bcce6e8d01 Various work to use the 2.9 interpreter 2011-05-31 17:31:51 -07:00
Matei Zaharia 4096c2287e Various fixes 2011-05-29 18:46:01 -07:00
Ismael Juma 0c62ee4321 Depend on jetty-server in compile scope and upgrade to 7.4.2. 
As Matei described: "We're using Jetty to run an HTTP server, not to embed Spark
in a webapp"
 2011-05-29 20:12:50 +01:00
Ismael Juma 1d75c6060a Update to Scala 2.9.0-1 and disable repl module for now. 
The repl module requires more complex work.
 2011-05-27 14:59:23 +01:00