Jey Kottalam
bd0bab47c9
SparkEnv isn't available this early, and not needed anyway
2013-08-15 16:50:37 -07:00
Jey Kottalam
4f43fd791a
make SparkHadoopUtil a member of SparkEnv
2013-08-15 16:50:37 -07:00
Jey Kottalam
43ebcb8484
rename HadoopMapRedUtil => SparkHadoopMapRedUtil, HadoopMapReduceUtil => SparkHadoopMapReduceUtil
2013-08-15 16:50:37 -07:00
Jey Kottalam
cb4ef19214
yarn support
2013-08-15 16:50:37 -07:00
Jey Kottalam
8b1c1520fc
add comment
2013-08-15 16:50:37 -07:00
Jey Kottalam
5d0785b4e5
remove hadoop-yarn's org/apache/...
2013-08-15 16:50:37 -07:00
Jey Kottalam
273b499b9a
yarn sbt
2013-08-15 16:50:37 -07:00
Jey Kottalam
69c3bbf688
dynamically detect hadoop version
2013-08-15 16:50:37 -07:00
Jey Kottalam
f67b94ad4f
remove core/src/hadoop{1,2} dirs
2013-08-15 16:50:36 -07:00
Jey Kottalam
b877e20a33
move yarn to its own directory
2013-08-15 16:50:36 -07:00
Matei Zaharia
28369ff773
Merge pull request #829 from JoshRosen/pyspark-unit-tests-python-2.6
...
Fix PySpark unit tests on Python 2.6
2013-08-15 16:44:02 -07:00
Joseph E. Gonzalez
327a4db9f7
changing caching behavior on indexedrdds
2013-08-15 16:36:26 -07:00
Patrick Wendell
4c6ade1ad5
Rename memoryBytesToString
and memoryMegabytesToString
...
These are used all over the place now and they are not specific to memory at all.
memoryBytesToString --> bytesToString
memoryMegabytesToString --> megabytesToString
2013-08-15 15:58:07 -07:00
Reynold Xin
1a13460cb0
Merge pull request #833 from rxin/ui
...
Various UI improvements.
2013-08-15 15:50:44 -07:00
Reynold Xin
1a51deae8a
More minor UI changes including code review feedback.
2013-08-15 14:34:07 -07:00
Joseph E. Gonzalez
3bb6e019d4
adding better error handling when indexing an RDD
2013-08-15 14:29:48 -07:00
Joseph E. Gonzalez
61281756f2
IndexedRDD passes all PairRDD Function tests
2013-08-15 14:20:59 -07:00
Daemoen
ad2e8b5126
Updated json output to allow for display of worker state
...
Ops teams need to ensure that the cluster is functional and performant. Having to scrape the html source for worker state won't work reliably, and will be slow. By exposing the state in the json output, ops teams are able to ensure a fully functional environment by querying for the json output and parsing for dead nodes.
2013-08-15 12:19:14 -07:00
Reynold Xin
2d2a556bdf
Various UI improvements.
2013-08-14 23:23:09 -07:00
Reynold Xin
044a088c0d
Merge pull request #831 from rxin/scheduler
...
A few small scheduler / job description changes.
2013-08-14 20:43:49 -07:00
Reynold Xin
290e3e6e65
Renamed setCurrentJobDescription to setJobDescription.
2013-08-14 18:40:53 -07:00
Reynold Xin
3886b54933
A few small scheduler / job description changes.
...
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.
2. Renamed SparkContext.setDescription to setCurrentJobDescription.
3. Throw an exception if the fair scheduler allocation file is invalid.
2013-08-14 17:19:42 -07:00
Joseph E. Gonzalez
54b54903c3
Adding testing code for indexedrdd
2013-08-14 16:35:20 -07:00
Matei Zaharia
839f2d4f3f
Merge pull request #822 from pwendell/ui-features
...
Adding GC Stats to TaskMetrics (and three small fixes)
2013-08-14 16:17:23 -07:00
Joseph E. Gonzalez
b71d4febbc
Finished early prototype of IndexedRDD
2013-08-14 15:25:56 -07:00
Josh Rosen
7a9abb9ddc
Fix PySpark unit tests on Python 2.6.
2013-08-14 15:12:12 -07:00
Patrick Wendell
04ad78b09d
Style cleanup based on Matei feedback
2013-08-14 14:57:21 -07:00
Reynold Xin
63446f9208
Merge pull request #826 from kayousterhout/ui_fix
...
Fixed 2 bugs in executor UI (incl. SPARK-877)
2013-08-14 00:17:07 -07:00
Kay Ousterhout
a88aa5e6ed
Fixed 2 bugs in executor UI.
...
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
2013-08-13 23:44:58 -07:00
Matei Zaharia
3f14cbab05
Merge pull request #825 from shivaram/maven-repl-fix
...
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:09:51 -07:00
Shivaram Venkataraman
a1227708e9
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:06:47 -07:00
Matei Zaharia
596adc63be
Merge pull request #824 from mateiz/mesos-0.12.1
...
Update to Mesos 0.12.1
2013-08-13 19:41:34 -07:00
Matei Zaharia
d9588183fa
Update to Mesos 0.12.1
2013-08-13 18:51:35 -07:00
Patrick Wendell
c223176388
Small style clean-up
2013-08-13 16:56:37 -07:00
Shivaram Venkataraman
c874625354
Specify label format in LogisticRegression.
2013-08-13 16:55:53 -07:00
Patrick Wendell
fab5cee111
Correcting terminology in RDD page
2013-08-13 16:25:55 -07:00
Patrick Wendell
024e5c5ce1
Correct sorting order for stages
2013-08-13 16:25:55 -07:00
Patrick Wendell
4e9f0c2df6
Capturing GC detials in TaskMetrics
2013-08-13 16:25:55 -07:00
Patrick Wendell
f0382007dc
Bug fix for display of shuffle read/write metrics.
...
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
2013-08-13 16:25:55 -07:00
Matei Zaharia
d316af9c84
Merge pull request #821 from pwendell/print-launch-command
...
Print run command to stderr rather than stdout
2013-08-13 15:31:01 -07:00
Matei Zaharia
1f79d21f33
Merge pull request #818 from kayousterhout/killed_fix
...
Properly account for killed tasks.
2013-08-13 15:23:54 -07:00
Patrick Wendell
a7feb69ae8
Print run command to stderr rather than stdout
2013-08-13 15:07:03 -07:00
Kay Ousterhout
1beb843a6f
Reuse the set of failed states rather than creating a new object each time
2013-08-13 14:27:40 -07:00
Joseph E. Gonzalez
f2b8dd3929
second indexedrdd design
2013-08-13 14:21:49 -07:00
Shivaram Venkataraman
0ab6ff4c32
Fix SVM model and unit test to work with {0,1}.
...
Also rename validateFuncs to validators.
2013-08-13 13:57:06 -07:00
Kay Ousterhout
c92dd627ca
Properly account for killed tasks.
...
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
2013-08-13 12:40:15 -07:00
Shivaram Venkataraman
654087194d
Change SVM to use {0,1} labels.
...
Also add a data validation check to make sure classification labels
are always 0 or 1 and add an appropriate test case.
2013-08-13 11:44:47 -07:00
Patrick Wendell
622f83ce1c
Merge pull request #817 from pwendell/pr_784
...
Minor clean-up in metrics servlet code
2013-08-13 09:58:52 -07:00
Patrick Wendell
ed6a1646e6
Slight change to pr-784
2013-08-13 09:29:40 -07:00
Patrick Wendell
a0133bfbad
Merge pull request #784 from jerryshao/dev-metrics-servlet
...
Add MetricsServlet for Spark metrics system
2013-08-13 09:28:18 -07:00