ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
4973 commits 6 branches 181 tags 397 MiB
Commit graph

4973 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin 2d2a556bdf Various UI improvements. 2013-08-14 23:23:09 -07:00
Reynold Xin 044a088c0d Merge pull request #831 from rxin/scheduler 
A few small scheduler / job description changes.
 2013-08-14 20:43:49 -07:00
Reynold Xin 290e3e6e65 Renamed setCurrentJobDescription to setJobDescription. 2013-08-14 18:40:53 -07:00
Reynold Xin 3886b54933 A few small scheduler / job description changes. 
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.

2. Renamed SparkContext.setDescription to setCurrentJobDescription.

3. Throw an exception if the fair scheduler allocation file is invalid.
 2013-08-14 17:19:42 -07:00
Joseph E. Gonzalez 54b54903c3 Adding testing code for indexedrdd 2013-08-14 16:35:20 -07:00
Matei Zaharia 839f2d4f3f Merge pull request #822 from pwendell/ui-features 
Adding GC Stats to TaskMetrics (and three small fixes)
 2013-08-14 16:17:23 -07:00
Joseph E. Gonzalez b71d4febbc Finished early prototype of IndexedRDD 2013-08-14 15:25:56 -07:00
Josh Rosen 7a9abb9ddc Fix PySpark unit tests on Python 2.6. 2013-08-14 15:12:12 -07:00
Patrick Wendell 04ad78b09d Style cleanup based on Matei feedback 2013-08-14 14:57:21 -07:00
Reynold Xin 63446f9208 Merge pull request #826 from kayousterhout/ui_fix 
Fixed 2 bugs in executor UI (incl. SPARK-877)
 2013-08-14 00:17:07 -07:00
Kay Ousterhout a88aa5e6ed Fixed 2 bugs in executor UI. 
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
 2013-08-13 23:44:58 -07:00
Matei Zaharia 3f14cbab05 Merge pull request #825 from shivaram/maven-repl-fix 
Set SPARK_CLASSPATH for maven repl tests
 2013-08-13 20:09:51 -07:00
Shivaram Venkataraman a1227708e9 Set SPARK_CLASSPATH for maven repl tests 2013-08-13 20:06:47 -07:00
Matei Zaharia 596adc63be Merge pull request #824 from mateiz/mesos-0.12.1 
Update to Mesos 0.12.1
 2013-08-13 19:41:34 -07:00
Matei Zaharia d9588183fa Update to Mesos 0.12.1 2013-08-13 18:51:35 -07:00
Patrick Wendell c223176388 Small style clean-up 2013-08-13 16:56:37 -07:00
Shivaram Venkataraman c874625354 Specify label format in LogisticRegression. 2013-08-13 16:55:53 -07:00
Patrick Wendell fab5cee111 Correcting terminology in RDD page 2013-08-13 16:25:55 -07:00
Patrick Wendell 024e5c5ce1 Correct sorting order for stages 2013-08-13 16:25:55 -07:00
Patrick Wendell 4e9f0c2df6 Capturing GC detials in TaskMetrics 2013-08-13 16:25:55 -07:00
Patrick Wendell f0382007dc Bug fix for display of shuffle read/write metrics. 
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
 2013-08-13 16:25:55 -07:00
Matei Zaharia d316af9c84 Merge pull request #821 from pwendell/print-launch-command 
Print run command to stderr rather than stdout
 2013-08-13 15:31:01 -07:00
Matei Zaharia 1f79d21f33 Merge pull request #818 from kayousterhout/killed_fix 
Properly account for killed tasks.
 2013-08-13 15:23:54 -07:00
Patrick Wendell a7feb69ae8 Print run command to stderr rather than stdout 2013-08-13 15:07:03 -07:00
Kay Ousterhout 1beb843a6f Reuse the set of failed states rather than creating a new object each time 2013-08-13 14:27:40 -07:00
Joseph E. Gonzalez f2b8dd3929 second indexedrdd design 2013-08-13 14:21:49 -07:00
Shivaram Venkataraman 0ab6ff4c32 Fix SVM model and unit test to work with {0,1}. 
Also rename validateFuncs to validators.
 2013-08-13 13:57:06 -07:00
Kay Ousterhout c92dd627ca Properly account for killed tasks. 
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
 2013-08-13 12:40:15 -07:00
Shivaram Venkataraman 654087194d Change SVM to use {0,1} labels. 
Also add a data validation check to make sure classification labels
are always 0 or 1 and add an appropriate test case.
 2013-08-13 11:44:47 -07:00
Patrick Wendell 622f83ce1c Merge pull request #817 from pwendell/pr_784 
Minor clean-up in metrics servlet code
 2013-08-13 09:58:52 -07:00
Patrick Wendell ed6a1646e6 Slight change to pr-784 2013-08-13 09:29:40 -07:00
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
Holden Karau d145da818e Code review feedback :) 2013-08-12 22:13:08 -07:00
Holden Karau 705c9ace2a Use less instances of the random class during ALS setup 2013-08-12 22:08:36 -07:00
Matei Zaharia e2fdac60da Merge pull request #802 from stayhf/SPARK-760-Python 
Simple PageRank algorithm implementation in Python for SPARK-760
 2013-08-12 21:26:59 -07:00
Matei Zaharia d3525babee Merge pull request #813 from AndreSchumacher/add_files_pyspark 
Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
 2013-08-12 21:02:39 -07:00
Andre Schumacher 8fd5c7bc00 Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark 
Now ADD_FILES uses a comma as file name separator.
 2013-08-12 20:22:52 -07:00
Matei Zaharia 9e02da2763 Merge pull request #812 from shivaram/maven-mllib-tests 
Create SparkContext in beforeAll for MLLib tests
 2013-08-12 20:22:27 -07:00
Matei Zaharia 65d0d91fba Merge pull request #807 from JoshRosen/guava-optional 
Change scala.Option to Guava Optional in Java APIs
 2013-08-12 19:00:57 -07:00
Josh Rosen cf08bb7a3e Fix import organization. 2013-08-12 18:55:02 -07:00
Evan Sparks 4346f0a1e9 Merge pull request #809 from shivaram/sgd-cleanup 
Clean up scaladoc in ML Lib.
 2013-08-12 12:12:12 -07:00
Matei Zaharia ea1b4baabd Merge pull request #806 from apivovarov/yarn-205 
Changed yarn.version to 2.0.5 in pom.xml
 2013-08-12 08:09:58 -07:00
Shivaram Venkataraman 8b5e3e2eb5 Add ML Lib scaladoc to API dropdown 2013-08-11 23:52:43 -07:00
jerryshao 09c7179e81 MetricsServlet code refactor according to comments 2013-08-12 13:23:23 +08:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Reynold Xin 2a39d2ca25 Merge pull request #810 from pwendell/dead_doc_code 
Remove now dead code inside of docs
 2013-08-11 20:35:09 -07:00
Patrick Wendell 9244524146 Removing dead docs 2013-08-11 20:33:58 -07:00
Shivaram Venkataraman 4935a2558b Clean up scaladoc in ML Lib. 
Also build and copy ML Lib scaladoc in Spark docs build.
Some more minor cleanup with respect to naming, test locations etc.
 2013-08-11 19:02:43 -07:00
Reynold Xin e5b9ed2833 Merge pull request #808 from pwendell/ui_compressed_bytes 
Report compressed bytes read when calculating TaskMetrics
 2013-08-11 17:22:47 -07:00
Shivaram Venkataraman ecc9bfe377 Create SparkContext in beforeAll for MLLib tests 
This overcomes test failures that occur using Maven
 2013-08-11 17:04:00 -07:00