ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3917 commits 6 branches 181 tags 397 MiB
Commit graph

3917 commits

This branch
This branch
All branches
Author SHA1 Message Date
Jey Kottalam 54e9379de2 Revert "Allow build configuration to be set in conf/spark-env.sh" 
This reverts commit 66e7a38a32.
 2013-08-21 21:13:34 -07:00
Matei Zaharia e6d66c8abd Merge pull request #853 from AndreSchumacher/double_rdd 
Implementing SPARK-838: Add DoubleRDDFunctions methods to PySpark
 2013-08-21 17:44:31 -07:00
Jey Kottalam f9cc1fbf27 Remove references to unsupported Hadoop versions 2013-08-21 17:14:36 -07:00
Andre Schumacher 76077bf9f4 Implementing SPARK-838: Add DoubleRDDFunctions methods to PySpark 2013-08-21 17:05:58 -07:00
Patrick Wendell c02585ea13 Make initial connection failure message less daunting. 
Right now it seems like something has gone wrong when this message is printed out.
Instead, this is a normal condition. So I changed the message a bit.
 2013-08-21 15:45:45 -07:00
Patrick Wendell 6be6b71c8c Merge branch 'master' into ec2-updates 
Conflicts:
	ec2/spark_ec2.py
 2013-08-21 15:34:31 -07:00
Jey Kottalam 4d737b6d32 Example should make sense 2013-08-21 15:03:37 -07:00
Jey Kottalam 6585f49841 Update build docs 2013-08-21 14:51:56 -07:00
Jey Kottalam 66e7a38a32 Allow build configuration to be set in conf/spark-env.sh 2013-08-21 14:25:53 -07:00
Mark Hamstra ff6f1b0500 Synced sbt and maven builds 2013-08-21 13:50:24 -07:00
Jey Kottalam 31644a011c Use "hadoop.version" property when specifying Hadoop YARN version too 2013-08-21 13:24:28 -07:00
Jey Kottalam 9c6f8df30f Update jekyll plugin to match docs/README.md 2013-08-21 12:57:56 -07:00
Matei Zaharia 111b2741fd Change default SPARK_HADOOP_VERSION in make-distribution.sh too 2013-08-21 11:54:10 -07:00
Reynold Xin 8e3ea4c7db Merge branch 'master' of github.com:mesos/spark 2013-08-21 11:38:51 -07:00
Reynold Xin af602ba9d3 Downgraded default build hadoop version to 1.0.4. 2013-08-21 11:38:24 -07:00
Matei Zaharia 53b1c30607 Update docs for Spark UI port 2013-08-20 22:57:11 -07:00
Patrick Wendell 51a1a0c602 Bump spark version 2013-08-20 22:14:52 -07:00
Reynold Xin 2905611c13 Merge pull request #851 from markhamstra/MutablePairTE 
Removed meaningless types
 2013-08-20 17:36:14 -07:00
Mark Hamstra 5eea613ec0 Removed meaningless types 2013-08-20 16:49:18 -07:00
Ali Ghodsi f20ed14e87 Merged in from upstream to use TaskLocation instead of strings 2013-08-20 16:21:43 -07:00
Ali Ghodsi 5cd21c4195 added curly braces to make the code more consistent 2013-08-20 16:16:05 -07:00
Ali Ghodsi db4bc55bef indent 2013-08-20 16:16:05 -07:00
Ali Ghodsi c0942a710f Bug in test fixed 2013-08-20 16:16:05 -07:00
Ali Ghodsi 5db41919b5 Added a test to make sure no locality preferences are ignored 2013-08-20 16:16:05 -07:00
Ali Ghodsi 7b123b3126 Simpler code 2013-08-20 16:16:05 -07:00
Ali Ghodsi 9192c358e4 simpler code 2013-08-20 16:16:05 -07:00
Ali Ghodsi a75a64eade Fixed almost all of Matei's feedback 2013-08-20 16:16:05 -07:00
Ali Ghodsi f1c853d76d fixed Matei's comments 2013-08-20 16:16:04 -07:00
Ali Ghodsi 890ea6ba79 making CoalescedRDDPartition public 2013-08-20 16:16:04 -07:00
Ali Ghodsi d6b6c680be comment in the test to make it more understandable 2013-08-20 16:16:04 -07:00
Ali Ghodsi b69e7166ba Coalescer now uses current preferred locations for derived RDDs. Made run() in DAGScheduler thread safe and added a method to be able to ask it for preferred locations. Added a similar method that wraps the former inside SparkContext. 2013-08-20 16:16:04 -07:00
Ali Ghodsi 3b5bb8a4ae added one test that will test a future functionality 2013-08-20 16:13:37 -07:00
Ali Ghodsi 33a0f59354 Added error messages to the tests to make failed tests less cryptic 2013-08-20 16:13:37 -07:00
Ali Ghodsi abcefb3858 fixed matei's comments 2013-08-20 16:13:37 -07:00
Ali Ghodsi 35537e6341 Made a function object that returns the coalesced groups 2013-08-20 16:13:37 -07:00
Ali Ghodsi 339598c080 several of Reynold's suggestions implemented 2013-08-20 16:13:37 -07:00
Ali Ghodsi 02d6464f2f space removed 2013-08-20 16:13:37 -07:00
Ali Ghodsi 4f99be1ffd use count rather than foreach 2013-08-20 16:13:37 -07:00
Ali Ghodsi f67753cdfc made preferredLocation a val of the surrounding case class 2013-08-20 16:13:37 -07:00
Ali Ghodsi f24861b60a Fix bug in tests 2013-08-20 16:13:36 -07:00
Ali Ghodsi f6e47e8b51 Renamed split to partition 2013-08-20 16:13:36 -07:00
Ali Ghodsi 937f72feb8 word wrap before 100 chars per line 2013-08-20 16:13:36 -07:00
Ali Ghodsi c4d59910b1 added goals inline as comment 2013-08-20 16:13:36 -07:00
Ali Ghodsi 7a2a33e32d Large scale load and locality tests for the coalesced partitions added 2013-08-20 16:13:36 -07:00
Ali Ghodsi 66edf854aa Bug, should compute slack wrt parent partition size, not number of bins 2013-08-20 16:13:36 -07:00
Ali Ghodsi 1ede102ba5 load balancing coalescer 2013-08-20 16:13:36 -07:00
Patrick Wendell 07e5c8b695 Set default Hadoop version to 1 2013-08-20 15:49:52 -07:00
Matei Zaharia aa2b89d98d Merge remote-tracking branch 'jey/hadoop-agnostic' 
Conflicts:
	core/src/main/scala/spark/PairRDDFunctions.scala
 2013-08-20 10:14:15 -07:00
Matei Zaharia d61337f640 Merge pull request #844 from markhamstra/priorityRename 
Renamed 'priority' to 'jobId' and assorted minor changes
 2013-08-20 10:06:06 -07:00
Mark Hamstra 1630fbf838 changeGeneration --> changeEpoch renaming 2013-08-20 00:17:16 -07:00