Reynold Xin
044a088c0d
Merge pull request #831 from rxin/scheduler
...
A few small scheduler / job description changes.
2013-08-14 20:43:49 -07:00
Reynold Xin
290e3e6e65
Renamed setCurrentJobDescription to setJobDescription.
2013-08-14 18:40:53 -07:00
Reynold Xin
3886b54933
A few small scheduler / job description changes.
...
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.
2. Renamed SparkContext.setDescription to setCurrentJobDescription.
3. Throw an exception if the fair scheduler allocation file is invalid.
2013-08-14 17:19:42 -07:00
Joseph E. Gonzalez
54b54903c3
Adding testing code for indexedrdd
2013-08-14 16:35:20 -07:00
Matei Zaharia
839f2d4f3f
Merge pull request #822 from pwendell/ui-features
...
Adding GC Stats to TaskMetrics (and three small fixes)
2013-08-14 16:17:23 -07:00
Joseph E. Gonzalez
b71d4febbc
Finished early prototype of IndexedRDD
2013-08-14 15:25:56 -07:00
Josh Rosen
7a9abb9ddc
Fix PySpark unit tests on Python 2.6.
2013-08-14 15:12:12 -07:00
Patrick Wendell
04ad78b09d
Style cleanup based on Matei feedback
2013-08-14 14:57:21 -07:00
Reynold Xin
63446f9208
Merge pull request #826 from kayousterhout/ui_fix
...
Fixed 2 bugs in executor UI (incl. SPARK-877)
2013-08-14 00:17:07 -07:00
Kay Ousterhout
a88aa5e6ed
Fixed 2 bugs in executor UI.
...
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
2013-08-13 23:44:58 -07:00
Matei Zaharia
3f14cbab05
Merge pull request #825 from shivaram/maven-repl-fix
...
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:09:51 -07:00
Shivaram Venkataraman
a1227708e9
Set SPARK_CLASSPATH for maven repl tests
2013-08-13 20:06:47 -07:00
Matei Zaharia
596adc63be
Merge pull request #824 from mateiz/mesos-0.12.1
...
Update to Mesos 0.12.1
2013-08-13 19:41:34 -07:00
Matei Zaharia
d9588183fa
Update to Mesos 0.12.1
2013-08-13 18:51:35 -07:00
Patrick Wendell
c223176388
Small style clean-up
2013-08-13 16:56:37 -07:00
Shivaram Venkataraman
c874625354
Specify label format in LogisticRegression.
2013-08-13 16:55:53 -07:00
Patrick Wendell
fab5cee111
Correcting terminology in RDD page
2013-08-13 16:25:55 -07:00
Patrick Wendell
024e5c5ce1
Correct sorting order for stages
2013-08-13 16:25:55 -07:00
Patrick Wendell
4e9f0c2df6
Capturing GC detials in TaskMetrics
2013-08-13 16:25:55 -07:00
Patrick Wendell
f0382007dc
Bug fix for display of shuffle read/write metrics.
...
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
2013-08-13 16:25:55 -07:00
Matei Zaharia
d316af9c84
Merge pull request #821 from pwendell/print-launch-command
...
Print run command to stderr rather than stdout
2013-08-13 15:31:01 -07:00
Matei Zaharia
1f79d21f33
Merge pull request #818 from kayousterhout/killed_fix
...
Properly account for killed tasks.
2013-08-13 15:23:54 -07:00
Patrick Wendell
a7feb69ae8
Print run command to stderr rather than stdout
2013-08-13 15:07:03 -07:00
Kay Ousterhout
1beb843a6f
Reuse the set of failed states rather than creating a new object each time
2013-08-13 14:27:40 -07:00
Joseph E. Gonzalez
f2b8dd3929
second indexedrdd design
2013-08-13 14:21:49 -07:00
Shivaram Venkataraman
0ab6ff4c32
Fix SVM model and unit test to work with {0,1}.
...
Also rename validateFuncs to validators.
2013-08-13 13:57:06 -07:00
Kay Ousterhout
c92dd627ca
Properly account for killed tasks.
...
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
2013-08-13 12:40:15 -07:00
Shivaram Venkataraman
654087194d
Change SVM to use {0,1} labels.
...
Also add a data validation check to make sure classification labels
are always 0 or 1 and add an appropriate test case.
2013-08-13 11:44:47 -07:00
Patrick Wendell
622f83ce1c
Merge pull request #817 from pwendell/pr_784
...
Minor clean-up in metrics servlet code
2013-08-13 09:58:52 -07:00
Patrick Wendell
ed6a1646e6
Slight change to pr-784
2013-08-13 09:29:40 -07:00
Patrick Wendell
a0133bfbad
Merge pull request #784 from jerryshao/dev-metrics-servlet
...
Add MetricsServlet for Spark metrics system
2013-08-13 09:28:18 -07:00
Holden Karau
d145da818e
Code review feedback :)
2013-08-12 22:13:08 -07:00
Holden Karau
705c9ace2a
Use less instances of the random class during ALS setup
2013-08-12 22:08:36 -07:00
Matei Zaharia
e2fdac60da
Merge pull request #802 from stayhf/SPARK-760-Python
...
Simple PageRank algorithm implementation in Python for SPARK-760
2013-08-12 21:26:59 -07:00
Matei Zaharia
d3525babee
Merge pull request #813 from AndreSchumacher/add_files_pyspark
...
Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
2013-08-12 21:02:39 -07:00
Andre Schumacher
8fd5c7bc00
Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
...
Now ADD_FILES uses a comma as file name separator.
2013-08-12 20:22:52 -07:00
Matei Zaharia
9e02da2763
Merge pull request #812 from shivaram/maven-mllib-tests
...
Create SparkContext in beforeAll for MLLib tests
2013-08-12 20:22:27 -07:00
Matei Zaharia
65d0d91fba
Merge pull request #807 from JoshRosen/guava-optional
...
Change scala.Option to Guava Optional in Java APIs
2013-08-12 19:00:57 -07:00
Josh Rosen
cf08bb7a3e
Fix import organization.
2013-08-12 18:55:02 -07:00
Evan Sparks
4346f0a1e9
Merge pull request #809 from shivaram/sgd-cleanup
...
Clean up scaladoc in ML Lib.
2013-08-12 12:12:12 -07:00
Matei Zaharia
ea1b4baabd
Merge pull request #806 from apivovarov/yarn-205
...
Changed yarn.version to 2.0.5 in pom.xml
2013-08-12 08:09:58 -07:00
Shivaram Venkataraman
8b5e3e2eb5
Add ML Lib scaladoc to API dropdown
2013-08-11 23:52:43 -07:00
jerryshao
09c7179e81
MetricsServlet code refactor according to comments
2013-08-12 13:23:23 +08:00
jerryshao
320e87e7ab
Add MetricsServlet for Spark metrics system
2013-08-12 13:23:23 +08:00
Reynold Xin
2a39d2ca25
Merge pull request #810 from pwendell/dead_doc_code
...
Remove now dead code inside of docs
2013-08-11 20:35:09 -07:00
Patrick Wendell
9244524146
Removing dead docs
2013-08-11 20:33:58 -07:00
Shivaram Venkataraman
4935a2558b
Clean up scaladoc in ML Lib.
...
Also build and copy ML Lib scaladoc in Spark docs build.
Some more minor cleanup with respect to naming, test locations etc.
2013-08-11 19:02:43 -07:00
Reynold Xin
e5b9ed2833
Merge pull request #808 from pwendell/ui_compressed_bytes
...
Report compressed bytes read when calculating TaskMetrics
2013-08-11 17:22:47 -07:00
Shivaram Venkataraman
ecc9bfe377
Create SparkContext in beforeAll for MLLib tests
...
This overcomes test failures that occur using Maven
2013-08-11 17:04:00 -07:00
Patrick Wendell
3d8f281604
Report compressed bytes read when calculating TaskMetrics
2013-08-11 16:25:57 -07:00