Jey Kottalam
bdd861c6c3
Fix Maven build with Hadoop 0.23.9
2013-08-18 18:28:57 -07:00
Jey Kottalam
47a7c4338a
Don't assume spark-examples JAR always exists
2013-08-18 16:59:02 -07:00
Jey Kottalam
44000b10ff
Make YARN POM file valid
2013-08-18 16:23:22 -07:00
Jey Kottalam
67b593607c
Rename YARN build flag to SPARK_WITH_YARN
2013-08-16 14:00:05 -07:00
Jey Kottalam
b1d99744a8
Fix SBT build under Hadoop 0.23.x
2013-08-16 13:50:12 -07:00
Jey Kottalam
c1e547bb7f
Updates to repl and example POMs to match SBT build
2013-08-16 13:50:12 -07:00
Jey Kottalam
ad580b94d5
Maven build now also works with YARN
2013-08-16 13:50:12 -07:00
Jey Kottalam
741ecd56fe
Forgot to remove a few references to ${classifier}
2013-08-16 13:50:12 -07:00
Jey Kottalam
9dd15fe700
Don't mark hadoop-client as 'provided'
2013-08-16 13:50:12 -07:00
Jey Kottalam
11b42a84db
Maven build now works with CDH hadoop-2.0.0-mr1
2013-08-16 13:50:12 -07:00
Jey Kottalam
353fab2440
Initial changes to make Maven build agnostic of hadoop version
2013-08-16 13:50:12 -07:00
Jey Kottalam
8add2d7a59
Fix repl/assembly when YARN enabled
2013-08-16 13:50:12 -07:00
Jey Kottalam
3f98eff63a
Allow make-distribution.sh to specify Hadoop version used
2013-08-16 13:50:09 -07:00
Jey Kottalam
a0f0848463
Update default version of Hadoop to 1.2.1
2013-08-15 16:50:37 -07:00
Jey Kottalam
a06a9d5c5f
Rename HadoopWriter to SparkHadoopWriter since it's outside of our package
2013-08-15 16:50:37 -07:00
Jey Kottalam
8f979edef5
Fix newTaskAttemptID to work under YARN
2013-08-15 16:50:37 -07:00
Jey Kottalam
14b6bcdf93
update YARN docs
2013-08-15 16:50:37 -07:00
Jey Kottalam
8bb0bd11ce
YARN ApplicationMaster shouldn't wait forever
2013-08-15 16:50:37 -07:00
Jey Kottalam
e2d7656ca3
re-enable YARN support
2013-08-15 16:50:37 -07:00
Jey Kottalam
bd0bab47c9
SparkEnv isn't available this early, and not needed anyway
2013-08-15 16:50:37 -07:00
Jey Kottalam
4f43fd791a
make SparkHadoopUtil a member of SparkEnv
2013-08-15 16:50:37 -07:00
Jey Kottalam
43ebcb8484
rename HadoopMapRedUtil => SparkHadoopMapRedUtil, HadoopMapReduceUtil => SparkHadoopMapReduceUtil
2013-08-15 16:50:37 -07:00
Jey Kottalam
cb4ef19214
yarn support
2013-08-15 16:50:37 -07:00
Jey Kottalam
8b1c1520fc
add comment
2013-08-15 16:50:37 -07:00
Jey Kottalam
5d0785b4e5
remove hadoop-yarn's org/apache/...
2013-08-15 16:50:37 -07:00
Jey Kottalam
273b499b9a
yarn sbt
2013-08-15 16:50:37 -07:00
Jey Kottalam
69c3bbf688
dynamically detect hadoop version
2013-08-15 16:50:37 -07:00
Jey Kottalam
f67b94ad4f
remove core/src/hadoop{1,2} dirs
2013-08-15 16:50:36 -07:00
Jey Kottalam
b877e20a33
move yarn to its own directory
2013-08-15 16:50:36 -07:00
Matei Zaharia
28369ff773
Merge pull request #829 from JoshRosen/pyspark-unit-tests-python-2.6
...
Fix PySpark unit tests on Python 2.6
2013-08-15 16:44:02 -07:00
Reynold Xin
1a13460cb0
Merge pull request #833 from rxin/ui
...
Various UI improvements.
2013-08-15 15:50:44 -07:00
Reynold Xin
1a51deae8a
More minor UI changes including code review feedback.
2013-08-15 14:34:07 -07:00
Reynold Xin
2d2a556bdf
Various UI improvements.
2013-08-14 23:23:09 -07:00
Reynold Xin
044a088c0d
Merge pull request #831 from rxin/scheduler
...
A few small scheduler / job description changes.
2013-08-14 20:43:49 -07:00
Reynold Xin
290e3e6e65
Renamed setCurrentJobDescription to setJobDescription.
2013-08-14 18:40:53 -07:00
Reynold Xin
3886b54933
A few small scheduler / job description changes.
...
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.
2. Renamed SparkContext.setDescription to setCurrentJobDescription.
3. Throw an exception if the fair scheduler allocation file is invalid.
2013-08-14 17:19:42 -07:00
Matei Zaharia
839f2d4f3f
Merge pull request #822 from pwendell/ui-features
...
Adding GC Stats to TaskMetrics (and three small fixes)
2013-08-14 16:17:23 -07:00
Josh Rosen
7a9abb9ddc
Fix PySpark unit tests on Python 2.6.
2013-08-14 15:12:12 -07:00
Patrick Wendell
04ad78b09d
Style cleanup based on Matei feedback
2013-08-14 14:57:21 -07:00
Reynold Xin
63446f9208
Merge pull request #826 from kayousterhout/ui_fix
...
Fixed 2 bugs in executor UI (incl. SPARK-877)
2013-08-14 00:17:07 -07:00
Kay Ousterhout
a88aa5e6ed
Fixed 2 bugs in executor UI.
...
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
2013-08-13 23:44:58 -07:00
Matei Zaharia
3f14cbab05
Merge pull request #825 from shivaram/maven-repl-fix
...
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:09:51 -07:00
Shivaram Venkataraman
a1227708e9
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:06:47 -07:00
Matei Zaharia
596adc63be
Merge pull request #824 from mateiz/mesos-0.12.1
...
Update to Mesos 0.12.1
2013-08-13 19:41:34 -07:00
Matei Zaharia
d9588183fa
Update to Mesos 0.12.1
2013-08-13 18:51:35 -07:00
Patrick Wendell
c223176388
Small style clean-up
2013-08-13 16:56:37 -07:00
Patrick Wendell
fab5cee111
Correcting terminology in RDD page
2013-08-13 16:25:55 -07:00
Patrick Wendell
024e5c5ce1
Correct sorting order for stages
2013-08-13 16:25:55 -07:00
Patrick Wendell
4e9f0c2df6
Capturing GC detials in TaskMetrics
2013-08-13 16:25:55 -07:00
Patrick Wendell
f0382007dc
Bug fix for display of shuffle read/write metrics.
...
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
2013-08-13 16:25:55 -07:00
