Matei Zaharia
e6d66c8abd
Merge pull request #853 from AndreSchumacher/double_rdd
...
Implementing SPARK-838: Add DoubleRDDFunctions methods to PySpark
2013-08-21 17:44:31 -07:00
Jey Kottalam
f9cc1fbf27
Remove references to unsupported Hadoop versions
2013-08-21 17:14:36 -07:00
Andre Schumacher
76077bf9f4
Implementing SPARK-838: Add DoubleRDDFunctions methods to PySpark
2013-08-21 17:05:58 -07:00
Patrick Wendell
c02585ea13
Make initial connection failure message less daunting.
...
Right now it seems like something has gone wrong when this message is printed out.
Instead, this is a normal condition. So I changed the message a bit.
2013-08-21 15:45:45 -07:00
Patrick Wendell
6be6b71c8c
Merge branch 'master' into ec2-updates
...
Conflicts:
ec2/spark_ec2.py
2013-08-21 15:34:31 -07:00
Jey Kottalam
4d737b6d32
Example should make sense
2013-08-21 15:03:37 -07:00
Jey Kottalam
6585f49841
Update build docs
2013-08-21 14:51:56 -07:00
Jey Kottalam
66e7a38a32
Allow build configuration to be set in conf/spark-env.sh
2013-08-21 14:25:53 -07:00
Mark Hamstra
ff6f1b0500
Synced sbt and maven builds
2013-08-21 13:50:24 -07:00
Jey Kottalam
31644a011c
Use "hadoop.version" property when specifying Hadoop YARN version too
2013-08-21 13:24:28 -07:00
Jey Kottalam
9c6f8df30f
Update jekyll plugin to match docs/README.md
2013-08-21 12:57:56 -07:00
Matei Zaharia
111b2741fd
Change default SPARK_HADOOP_VERSION in make-distribution.sh too
2013-08-21 11:54:10 -07:00
Reynold Xin
8e3ea4c7db
Merge branch 'master' of github.com:mesos/spark
2013-08-21 11:38:51 -07:00
Reynold Xin
af602ba9d3
Downgraded default build hadoop version to 1.0.4.
2013-08-21 11:38:24 -07:00
Matei Zaharia
53b1c30607
Update docs for Spark UI port
2013-08-20 22:57:11 -07:00
Patrick Wendell
51a1a0c602
Bump spark version
2013-08-20 22:14:52 -07:00
Reynold Xin
2905611c13
Merge pull request #851 from markhamstra/MutablePairTE
...
Removed meaningless types
2013-08-20 17:36:14 -07:00
Mark Hamstra
5eea613ec0
Removed meaningless types
2013-08-20 16:49:18 -07:00
Ali Ghodsi
f20ed14e87
Merged in from upstream to use TaskLocation instead of strings
2013-08-20 16:21:43 -07:00
Ali Ghodsi
5cd21c4195
added curly braces to make the code more consistent
2013-08-20 16:16:05 -07:00
Ali Ghodsi
db4bc55bef
indent
2013-08-20 16:16:05 -07:00
Ali Ghodsi
c0942a710f
Bug in test fixed
2013-08-20 16:16:05 -07:00
Ali Ghodsi
5db41919b5
Added a test to make sure no locality preferences are ignored
2013-08-20 16:16:05 -07:00
Ali Ghodsi
7b123b3126
Simpler code
2013-08-20 16:16:05 -07:00
Ali Ghodsi
9192c358e4
simpler code
2013-08-20 16:16:05 -07:00
Ali Ghodsi
a75a64eade
Fixed almost all of Matei's feedback
2013-08-20 16:16:05 -07:00
Ali Ghodsi
f1c853d76d
fixed Matei's comments
2013-08-20 16:16:04 -07:00
Ali Ghodsi
890ea6ba79
making CoalescedRDDPartition public
2013-08-20 16:16:04 -07:00
Ali Ghodsi
d6b6c680be
comment in the test to make it more understandable
2013-08-20 16:16:04 -07:00
Ali Ghodsi
b69e7166ba
Coalescer now uses current preferred locations for derived RDDs. Made run() in DAGScheduler thread safe and added a method to be able to ask it for preferred locations. Added a similar method that wraps the former inside SparkContext.
2013-08-20 16:16:04 -07:00
Ali Ghodsi
3b5bb8a4ae
added one test that will test a future functionality
2013-08-20 16:13:37 -07:00
Ali Ghodsi
33a0f59354
Added error messages to the tests to make failed tests less cryptic
2013-08-20 16:13:37 -07:00
Ali Ghodsi
abcefb3858
fixed matei's comments
2013-08-20 16:13:37 -07:00
Ali Ghodsi
35537e6341
Made a function object that returns the coalesced groups
2013-08-20 16:13:37 -07:00
Ali Ghodsi
339598c080
several of Reynold's suggestions implemented
2013-08-20 16:13:37 -07:00
Ali Ghodsi
02d6464f2f
space removed
2013-08-20 16:13:37 -07:00
Ali Ghodsi
4f99be1ffd
use count rather than foreach
2013-08-20 16:13:37 -07:00
Ali Ghodsi
f67753cdfc
made preferredLocation a val of the surrounding case class
2013-08-20 16:13:37 -07:00
Ali Ghodsi
f24861b60a
Fix bug in tests
2013-08-20 16:13:36 -07:00
Ali Ghodsi
f6e47e8b51
Renamed split to partition
2013-08-20 16:13:36 -07:00
Ali Ghodsi
937f72feb8
word wrap before 100 chars per line
2013-08-20 16:13:36 -07:00
Ali Ghodsi
c4d59910b1
added goals inline as comment
2013-08-20 16:13:36 -07:00
Ali Ghodsi
7a2a33e32d
Large scale load and locality tests for the coalesced partitions added
2013-08-20 16:13:36 -07:00
Ali Ghodsi
66edf854aa
Bug, should compute slack wrt parent partition size, not number of bins
2013-08-20 16:13:36 -07:00
Ali Ghodsi
1ede102ba5
load balancing coalescer
2013-08-20 16:13:36 -07:00
Patrick Wendell
07e5c8b695
Set default Hadoop version to 1
2013-08-20 15:49:52 -07:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Matei Zaharia
d61337f640
Merge pull request #844 from markhamstra/priorityRename
...
Renamed 'priority' to 'jobId' and assorted minor changes
2013-08-20 10:06:06 -07:00
Mark Hamstra
1630fbf838
changeGeneration --> changeEpoch renaming
2013-08-20 00:17:16 -07:00
Mark Hamstra
ad18410427
Renamed 'priority' to 'jobId' and assorted minor changes
2013-08-20 00:07:04 -07:00