ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2354 commits 6 branches 181 tags 397 MiB
Commit graph

2354 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 1ef58dadcc Add a config property for Akka lifecycle event logging 2013-02-25 14:01:24 -08:00
Matei Zaharia 6b87ef7c86 Fix compile error 2013-02-25 14:01:16 -08:00
Matei Zaharia 01bd136ba5 Use public method sparkContext instead of protected sc in streaming examples 2013-02-25 13:27:11 -08:00
Matei Zaharia 848321f910 Change doc color scheme slightly for Spark 0.7 (to differ from 0.6) 2013-02-25 13:15:30 -08:00
Matei Zaharia 3c7dcb61ab Use a single setting for disabling API doc build 2013-02-25 13:15:12 -08:00
Matei Zaharia 0cb8ee93cd Merge branch 'master' of github.com:mesos/spark 2013-02-25 13:10:35 -08:00
Matei Zaharia ceaec4a675 Merge pull request #498 from pwendell/shutup-akka 
Disable remote lifecycle logging from Akka.
 2013-02-25 12:31:24 -08:00
Patrick Wendell 85a85646d9 Disable remote lifecycle logging from Akka. 
This changes the default setting to `off` for remote lifecycle events. When this is on, it is very chatty at the INFO level. It also prints out several ERROR messages sometimes when sc.stop() is called.
 2013-02-25 12:25:43 -08:00
Matei Zaharia 25f737804a Change tabs to spaces 2013-02-25 11:53:55 -08:00
Matei Zaharia 6ae9a22c3e Get spark.default.paralellism on each call to defaultPartitioner, 
instead of only once, in case the user changes it across Spark uses
 2013-02-25 10:28:08 -08:00
Matei Zaharia d6e6abece3 Merge pull request #459 from stephenh/bettersplits 
Change defaultPartitioner to use upstream split size.
 2013-02-25 09:22:04 -08:00
Matei Zaharia fb76250598 Merge pull request #497 from ScrapCodes/dep-resolution-fix 
Moving akka dependency resolver to shared.
 2013-02-25 00:16:52 -08:00
Prashant Sharma 254acb1666 Moving akka dependency resolver to shared. 2013-02-25 13:37:07 +05:30
Stephen Haberman c44ccf2862 Use default parallelism if its set. 2013-02-24 23:54:03 -06:00
Stephen Haberman 44032bc476 Merge branch 'master' into bettersplits 
Conflicts:
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/test/scala/spark/ShuffleSuite.scala
 2013-02-24 22:08:14 -06:00
Matei Zaharia 22f68b7e6b Merge pull request #494 from mesos/streaming 
Final set of Spark Streaming changes
 2013-02-24 17:48:17 -08:00
Tathagata Das f9881b10c6 Merge pull request #495 from tdas/streaming 
Changed class paths and dependencies based on Matei's comments
 2013-02-24 16:39:04 -08:00
Tathagata Das abb5471865 Removing duplicate doc. 2013-02-24 16:32:44 -08:00
Tathagata Das 5ab37be983 Fixed class paths and dependencies based on Matei's comments. 2013-02-24 16:24:52 -08:00
Tathagata Das 32f1a2d540 Merge pull request #493 from tdas/streaming 
Streaming bug fixes
 2013-02-24 14:46:19 -08:00
Matei Zaharia 066ce0ffa4 Merge pull request #492 from JoshRosen/pyspark-split-rename 
Change numSplits to numPartitions in PySpark
 2013-02-24 14:11:07 -08:00
Josh Rosen 2c966c98fb Change numSplits to numPartitions in PySpark. 2013-02-24 13:25:09 -08:00
Tathagata Das 28f8b721f6 Added back the initial spark job before starting streaming receivers 2013-02-24 13:01:54 -08:00
Tathagata Das f282bc4960 Changed Algebird from 0.1.9 to 0.1.8 2013-02-24 12:44:12 -08:00
Tathagata Das dff53d1b94 Merge branch 'mesos-master' into streaming 2013-02-24 12:17:22 -08:00
Tathagata Das 68c7934b1a Fixed missing dependencies in streaming/pom.xml 2013-02-24 11:51:45 -08:00
Tathagata Das c1a040db3a Fixed bugs in examples. 2013-02-24 11:00:30 -08:00
Tathagata Das b4eb24de96 Updated streaming programming guide with Java API info, and comments from Patrick. 2013-02-23 23:59:45 -08:00
Matei Zaharia 3b9f929467 Merge pull request #468 from haitaoyao/master 
support customized java options for Master, Worker, Executor, and Repl
 2013-02-23 23:38:15 -08:00
Matei Zaharia beb79dbbfd Merge pull request #473 from stephenh/subtract 
Add RDD.subtract.
 2013-02-23 23:25:27 -08:00
Stephen Haberman 37c7a71f9c Add subtract to JavaRDD, JavaDoubleRDD, and JavaPairRDD. 2013-02-24 00:27:53 -06:00
Stephen Haberman f442e7d83c Update for split->partition rename. 2013-02-24 00:27:14 -06:00
Stephen Haberman cec87a0653 Merge branch 'master' into subtract 2013-02-23 23:27:55 -06:00
Tathagata Das d853aa9658 Change spark.cleaner.delay to spark.cleaner.ttl. Updated docs. 2013-02-23 17:42:26 -08:00
Tathagata Das 41285eaae3 Fixed differences in APIs of StreamingContext and JavaStreamingContext. Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite. 2013-02-23 16:25:07 -08:00
Matei Zaharia 8a0ca6f05c Merge pull request #491 from pwendell/java-vector 
Adding Java-accessible methods to Vector.scala
 2013-02-23 15:55:06 -08:00
Patrick Wendell 931f439be9 Responding to code review 2013-02-23 15:40:41 -08:00
Patrick Wendell f51b0f93f2 Adding Java-accessible methods to Vector.scala 
This is needed for the Strata machine learning tutorial (and
also is generally helpful).
 2013-02-23 13:26:59 -08:00
Matei Zaharia d942d39072 Handle exceptions in RecordReader.close() better (suggested by Jim 
Donahue)
 2013-02-23 11:19:07 -08:00
Matei Zaharia c89824046a Merge pull request #490 from woggling/conn-death 
Detect when SendingConnections disconnect even if we aren't sending to them
 2013-02-22 22:58:19 -08:00
Tathagata Das d8cee52d52 Merge branch 'mesos-streaming' into streaming 2013-02-22 18:25:34 -08:00
Tathagata Das 24c0cd6168 Fixed resolver for akka-zeromq 2013-02-22 18:23:29 -08:00
Charles Reiss 50cf8c8b79 Add fault tolerance test that uses replicated RDDs. 2013-02-22 16:11:53 -08:00
Charles Reiss c8a7886921 Detect when SendingConnections drop by trying to read them. 
Comment fix
 2013-02-22 16:11:52 -08:00
Matei Zaharia d4d7993bf5 Several fixes to the work to log when no resources can be used by a job. 
Fixed some of the messages as well as code style.
 2013-02-22 15:51:37 -08:00
Matei Zaharia f33662c133 Merge remote-tracking branch 'pwendell/starvation-check' 
Also fixed a bug where master was offering executors on dead workers

Conflicts:
	core/src/main/scala/spark/deploy/master/Master.scala
 2013-02-22 15:27:41 -08:00
Matei Zaharia 7341de0d48 Merge pull request #475 from JoshRosen/spark-668 
Remove hack workaround for SPARK-668
 2013-02-22 14:56:18 -08:00
Matei Zaharia fcf7fa8448 Merge pull request #489 from pwendell/SPARK-702 
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
 2013-02-22 14:55:46 -08:00
Patrick Wendell f8c3a03d55 SPARK-702: Replace Function --> JFunction in JavaAPI Suite. 
In a few places the Scala (rather than Java) function class is used.
 2013-02-22 12:54:15 -08:00
Tathagata Das cfa65ebff1 Merge pull request #480 from MLnick/streaming-eg-algebird 
[Streaming] Examples using Twitter's Algebird library
 2013-02-22 12:29:04 -08:00