Jey Kottalam
6585f49841
Update build docs
2013-08-21 14:51:56 -07:00
Jey Kottalam
66e7a38a32
Allow build configuration to be set in conf/spark-env.sh
2013-08-21 14:25:53 -07:00
Mark Hamstra
ff6f1b0500
Synced sbt and maven builds
2013-08-21 13:50:24 -07:00
Jey Kottalam
31644a011c
Use "hadoop.version" property when specifying Hadoop YARN version too
2013-08-21 13:24:28 -07:00
Jey Kottalam
9c6f8df30f
Update jekyll plugin to match docs/README.md
2013-08-21 12:57:56 -07:00
Matei Zaharia
111b2741fd
Change default SPARK_HADOOP_VERSION in make-distribution.sh too
2013-08-21 11:54:10 -07:00
Reynold Xin
8e3ea4c7db
Merge branch 'master' of github.com:mesos/spark
2013-08-21 11:38:51 -07:00
Reynold Xin
af602ba9d3
Downgraded default build hadoop version to 1.0.4.
2013-08-21 11:38:24 -07:00
Matei Zaharia
53b1c30607
Update docs for Spark UI port
2013-08-20 22:57:11 -07:00
Patrick Wendell
51a1a0c602
Bump spark version
2013-08-20 22:14:52 -07:00
Reynold Xin
2905611c13
Merge pull request #851 from markhamstra/MutablePairTE
...
Removed meaningless types
2013-08-20 17:36:14 -07:00
Mark Hamstra
5eea613ec0
Removed meaningless types
2013-08-20 16:49:18 -07:00
Ali Ghodsi
f20ed14e87
Merged in from upstream to use TaskLocation instead of strings
2013-08-20 16:21:43 -07:00
Ali Ghodsi
5cd21c4195
added curly braces to make the code more consistent
2013-08-20 16:16:05 -07:00
Ali Ghodsi
db4bc55bef
indent
2013-08-20 16:16:05 -07:00
Ali Ghodsi
c0942a710f
Bug in test fixed
2013-08-20 16:16:05 -07:00
Ali Ghodsi
5db41919b5
Added a test to make sure no locality preferences are ignored
2013-08-20 16:16:05 -07:00
Ali Ghodsi
7b123b3126
Simpler code
2013-08-20 16:16:05 -07:00
Ali Ghodsi
9192c358e4
simpler code
2013-08-20 16:16:05 -07:00
Ali Ghodsi
a75a64eade
Fixed almost all of Matei's feedback
2013-08-20 16:16:05 -07:00
Ali Ghodsi
f1c853d76d
fixed Matei's comments
2013-08-20 16:16:04 -07:00
Ali Ghodsi
890ea6ba79
making CoalescedRDDPartition public
2013-08-20 16:16:04 -07:00
Ali Ghodsi
d6b6c680be
comment in the test to make it more understandable
2013-08-20 16:16:04 -07:00
Ali Ghodsi
b69e7166ba
Coalescer now uses current preferred locations for derived RDDs. Made run() in DAGScheduler thread safe and added a method to be able to ask it for preferred locations. Added a similar method that wraps the former inside SparkContext.
2013-08-20 16:16:04 -07:00
Ali Ghodsi
3b5bb8a4ae
added one test that will test a future functionality
2013-08-20 16:13:37 -07:00
Ali Ghodsi
33a0f59354
Added error messages to the tests to make failed tests less cryptic
2013-08-20 16:13:37 -07:00
Ali Ghodsi
abcefb3858
fixed matei's comments
2013-08-20 16:13:37 -07:00
Ali Ghodsi
35537e6341
Made a function object that returns the coalesced groups
2013-08-20 16:13:37 -07:00
Ali Ghodsi
339598c080
several of Reynold's suggestions implemented
2013-08-20 16:13:37 -07:00
Ali Ghodsi
02d6464f2f
space removed
2013-08-20 16:13:37 -07:00
Ali Ghodsi
4f99be1ffd
use count rather than foreach
2013-08-20 16:13:37 -07:00
Ali Ghodsi
f67753cdfc
made preferredLocation a val of the surrounding case class
2013-08-20 16:13:37 -07:00
Ali Ghodsi
f24861b60a
Fix bug in tests
2013-08-20 16:13:36 -07:00
Ali Ghodsi
f6e47e8b51
Renamed split to partition
2013-08-20 16:13:36 -07:00
Ali Ghodsi
937f72feb8
word wrap before 100 chars per line
2013-08-20 16:13:36 -07:00
Ali Ghodsi
c4d59910b1
added goals inline as comment
2013-08-20 16:13:36 -07:00
Ali Ghodsi
7a2a33e32d
Large scale load and locality tests for the coalesced partitions added
2013-08-20 16:13:36 -07:00
Ali Ghodsi
66edf854aa
Bug, should compute slack wrt parent partition size, not number of bins
2013-08-20 16:13:36 -07:00
Ali Ghodsi
1ede102ba5
load balancing coalescer
2013-08-20 16:13:36 -07:00
Patrick Wendell
07e5c8b695
Set default Hadoop version to 1
2013-08-20 15:49:52 -07:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Matei Zaharia
d61337f640
Merge pull request #844 from markhamstra/priorityRename
...
Renamed 'priority' to 'jobId' and assorted minor changes
2013-08-20 10:06:06 -07:00
Mark Hamstra
1630fbf838
changeGeneration --> changeEpoch renaming
2013-08-20 00:17:16 -07:00
Mark Hamstra
ad18410427
Renamed 'priority' to 'jobId' and assorted minor changes
2013-08-20 00:07:04 -07:00
Matei Zaharia
8cae72e94e
Merge pull request #828 from mateiz/sched-improvements
...
Scheduler fixes and improvements
2013-08-19 23:40:04 -07:00
Matei Zaharia
efeb142981
Merge pull request #849 from mateiz/web-fixes
...
Small fixes to web UI
2013-08-19 19:23:50 -07:00
Matei Zaharia
793a722f8e
Allow some wiggle room in UISuite port test and in EC2 ports
2013-08-19 18:51:00 -07:00
Matei Zaharia
abdc1f8bbb
Merge pull request #847 from rxin/rdd
...
Allow subclasses of Product2 in all key-value related classes
2013-08-19 18:30:56 -07:00
Matei Zaharia
498a26189b
Small fixes to web UI:
...
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
2013-08-19 18:17:49 -07:00
Reynold Xin
5054abd41b
Code review feedback. (added tests for cogroup and substract; added more documentation on MutablePair)
2013-08-19 12:58:02 -07:00