ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
11336 commits 6 branches 181 tags 397 MiB
Commit graph

11336 commits

This branch
This branch
All branches
Author SHA1 Message Date
Stephen Haberman cbf72bffa5 Include name, if set, in RDD.toString(). 2013-01-29 00:20:36 -06:00
Stephen Haberman 3cda14af3f Add number of splits. 2013-01-29 00:12:31 -06:00
Matei Zaharia a1ecec8d79 Merge branch 'master' of github.com:mesos/spark 2013-01-28 22:08:44 -08:00
Stephen Haberman 951cfd9ba2 Add JavaRDDLike.toDebugString(). 2013-01-29 00:02:17 -06:00
Matei Zaharia f6eb1f0825 Merge pull request #413 from pwendell/stage-logging 
SPARK-658: Adding logging of stage duration
 2013-01-28 22:01:52 -08:00
Stephen Haberman b45857c965 Add RDD.toDebugString. 
Original idea by Nathan Kronenfeld.
 2013-01-28 23:56:56 -06:00
Patrick Wendell 7ee824e42e Units from ms -> s 2013-01-28 21:48:32 -08:00
Stephen Haberman 13368818af Merge branch 'master' into driver 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/SparkEnv.scala
	core/src/main/scala/spark/deploy/LocalSparkCluster.scala
	core/src/main/scala/spark/executor/StandaloneExecutorBackend.scala
	core/src/main/scala/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneClusterMessage.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/ThreadingTest.scala
	core/src/test/scala/spark/MapOutputTrackerSuite.scala
 2013-01-28 23:30:24 -06:00
Matei Zaharia dda2ce017c Merge pull request #424 from pwendell/logging-cleanup 
Some DEBUG-level log cleanup.
 2013-01-28 21:18:54 -08:00
Matei Zaharia 8160f03ac4 Merge pull request #423 from squito/long_float_accums 
add long and float accumulatorparams
 2013-01-28 21:18:01 -08:00
Patrick Wendell 1f9b486a8b Some DEBUG-level log cleanup. 
A few changes to make the DEBUG-level logs less
noisy and more readable.

- Moved a few very frequent messages to Trace
- Changed some BlockManger log messages to make them
  more understandable

SPARK-666 #resolve
 2013-01-28 20:29:35 -08:00
Imran Rashid efff7bfb33 add long and float accumulatorparams 2013-01-28 20:23:11 -08:00
Imran Rashid cec9c768c2 convenient name available in StageInfo 2013-01-28 20:09:41 -08:00
Imran Rashid 01d77f329f expose stageInfo in SparkContext 2013-01-28 20:09:40 -08:00
Imran Rashid 38b83bc66b can get task runtime summary from task info 2013-01-28 20:09:40 -08:00
Imran Rashid b88daee916 simple util to summarize distributions 2013-01-28 20:09:40 -08:00
Imran Rashid b14841455c track task completion in DAGScheduler, and send a stageCompleted event with taskInfo to SparkListeners 2013-01-28 20:09:40 -08:00
Imran Rashid 0f22c4207f better formatting for RDDInfo 2013-01-28 20:07:53 -08:00
Imran Rashid a423ee546c expose RDD & storage info directly via SparkContext 2013-01-28 20:07:53 -08:00
haitao.yao 4670c99a21 Merge branch 'mesos' 2013-01-29 09:57:57 +08:00
Shivaram Venkataraman da44a391fb Add an option to use the old scripts 2013-01-28 11:16:14 -08:00
Patrick Wendell 501433f1d5 Making submission time a field 2013-01-28 10:45:57 -08:00
Patrick Wendell c423be7d8e Renaming stage finished function 2013-01-28 10:45:57 -08:00
Patrick Wendell 07f568e1bf SPARK-658: Adding logging of stage duration 2013-01-28 10:45:57 -08:00
Matei Zaharia 286f8f876f Change time unit in MetadataCleaner to seconds 2013-01-28 01:29:27 -08:00
Matei Zaharia f03d9760fd Clean up BlockManagerUI a little (make it not be an object, merge with 
Directives, and bind to a random port)
 2013-01-27 23:56:14 -08:00
Matei Zaharia 909850729e Rename more things from slave to executor 2013-01-27 23:17:20 -08:00
Matei Zaharia 44b4a0f88f Track workers by executor ID instead of hostname to allow multiple 
executors per machine and remove the need for multiple IP addresses in
unit tests.
 2013-01-27 19:23:49 -08:00
Matei Zaharia b9e2d9efec Merge pull request #419 from shivaram/ec2-ip-change 
Detect whether we run on EC2 using ec2-metadata as well
 2013-01-27 18:41:11 -08:00
haitao.yao aeef6f6b08 Merge branch 'mesos' 2013-01-28 10:13:55 +08:00
Matei Zaharia 6ad8540b40 Merge pull request #401 from squito/blockmanager_ui 
Blockmanager ui
 2013-01-27 15:51:08 -08:00
Shivaram Venkataraman dc9d3ab6ed Add option to start ganglia. Also enable Hadoop ports even if cluster type is 
not mesos
 2013-01-27 00:26:00 -08:00
Shivaram Venkataraman 0243b081ce Fix swap variable name 2013-01-27 00:25:29 -08:00
Shivaram Venkataraman 717b221cca Detect whether we run on EC2 using ec2-metadata as well 2013-01-26 23:03:11 -08:00
Shivaram Venkataraman d012cfa0e8 Update spark_ec2.py to use new spark-ec2 scripts 2013-01-26 22:48:39 -08:00
Matei Zaharia 49f6472c0f Merge pull request #418 from woggling/reregister-deadlock 
Fix BlockManager reregistration deadlock; do BlockManager reregistration more asynchronously
 2013-01-26 18:59:02 -08:00
Charles Reiss 58fc6b2bed Handle duplicate registrations better. 2013-01-26 18:30:44 -08:00
Charles Reiss ad4232b4da Fix deadlock in BlockManager reregistration triggered by failed updates. 2013-01-26 18:30:38 -08:00
Matei Zaharia ec2dadb521 Merge pull request #417 from JoshRosen/spark-668 
Fix JavaRDDLike.flatMap(PairFlatMapFunction) (SPARK-668).
 2013-01-26 16:20:57 -08:00
Josh Rosen d49cf0e587 Fix JavaRDDLike.flatMap(PairFlatMapFunction) (SPARK-668). 
This workaround is easier than rewriting JavaRDDLike in Java.
 2013-01-26 16:13:18 -08:00
Imran Rashid 49c05608f5 add metadatacleaner for persisentRdd map 2013-01-25 17:04:16 -08:00
Matei Zaharia 2435b7b5b7 Merge pull request #416 from stephenh/morefinally 
Call executeOnCompleteCallbacks in more finally blocks.
 2013-01-25 15:33:26 -08:00
Stephen Haberman 8efbda0b17 Call executeOnCompleteCallbacks in more finally blocks. 2013-01-25 14:55:33 -06:00
Imran Rashid a1d9d1767d fixup 1cadaa1, changed api of map 2013-01-25 10:05:26 -08:00
Imran Rashid 1cadaa164e switch to TimeStampedHashMap for storing persistent Rdds 2013-01-25 09:30:21 -08:00
Imran Rashid 539491bbc3 code reformatting 2013-01-25 09:29:59 -08:00
Stephen Haberman 7dfb82a992 Replace old 'master' term with 'driver'. 2013-01-25 11:03:00 -06:00
Prashant Sharma 4496bf197b Improved document comment in example 2013-01-25 14:34:38 +05:30
Matei Zaharia 04bfee2d08 Merge pull request #411 from stephenh/localsparkcontext 
Add LocalSparkContext to manage common sc variable.
 2013-01-24 23:00:50 -08:00
Stephen Haberman ec43a51b38 Merge branch 'master' into localsparkcontext 
Conflicts:
	core/src/test/scala/spark/FileServerSuite.scala
	core/src/test/scala/spark/RDDSuite.scala
 2013-01-24 21:17:30 -06:00