ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3654 commits 6 branches 181 tags 397 MiB
Commit graph

3654 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin 290e3e6e65 Renamed setCurrentJobDescription to setJobDescription. 2013-08-14 18:40:53 -07:00
Reynold Xin 3886b54933 A few small scheduler / job description changes. 
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.

2. Renamed SparkContext.setDescription to setCurrentJobDescription.

3. Throw an exception if the fair scheduler allocation file is invalid.
 2013-08-14 17:19:42 -07:00
Matei Zaharia 839f2d4f3f Merge pull request #822 from pwendell/ui-features 
Adding GC Stats to TaskMetrics (and three small fixes)
 2013-08-14 16:17:23 -07:00
Patrick Wendell 04ad78b09d Style cleanup based on Matei feedback 2013-08-14 14:57:21 -07:00
Reynold Xin 63446f9208 Merge pull request #826 from kayousterhout/ui_fix 
Fixed 2 bugs in executor UI (incl. SPARK-877)
 2013-08-14 00:17:07 -07:00
Kay Ousterhout a88aa5e6ed Fixed 2 bugs in executor UI. 
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
 2013-08-13 23:44:58 -07:00
Matei Zaharia 3f14cbab05 Merge pull request #825 from shivaram/maven-repl-fix 
Set SPARK_CLASSPATH for maven repl tests
 2013-08-13 20:09:51 -07:00
Shivaram Venkataraman a1227708e9 Set SPARK_CLASSPATH for maven repl tests 2013-08-13 20:06:47 -07:00
Matei Zaharia 596adc63be Merge pull request #824 from mateiz/mesos-0.12.1 
Update to Mesos 0.12.1
 2013-08-13 19:41:34 -07:00
Matei Zaharia d9588183fa Update to Mesos 0.12.1 2013-08-13 18:51:35 -07:00
Patrick Wendell c223176388 Small style clean-up 2013-08-13 16:56:37 -07:00
Patrick Wendell fab5cee111 Correcting terminology in RDD page 2013-08-13 16:25:55 -07:00
Patrick Wendell 024e5c5ce1 Correct sorting order for stages 2013-08-13 16:25:55 -07:00
Patrick Wendell 4e9f0c2df6 Capturing GC detials in TaskMetrics 2013-08-13 16:25:55 -07:00
Patrick Wendell f0382007dc Bug fix for display of shuffle read/write metrics. 
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
 2013-08-13 16:25:55 -07:00
Matei Zaharia d316af9c84 Merge pull request #821 from pwendell/print-launch-command 
Print run command to stderr rather than stdout
 2013-08-13 15:31:01 -07:00
Matei Zaharia 1f79d21f33 Merge pull request #818 from kayousterhout/killed_fix 
Properly account for killed tasks.
 2013-08-13 15:23:54 -07:00
Patrick Wendell a7feb69ae8 Print run command to stderr rather than stdout 2013-08-13 15:07:03 -07:00
Kay Ousterhout 1beb843a6f Reuse the set of failed states rather than creating a new object each time 2013-08-13 14:27:40 -07:00
Kay Ousterhout c92dd627ca Properly account for killed tasks. 
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
 2013-08-13 12:40:15 -07:00
Patrick Wendell 622f83ce1c Merge pull request #817 from pwendell/pr_784 
Minor clean-up in metrics servlet code
 2013-08-13 09:58:52 -07:00
Patrick Wendell ed6a1646e6 Slight change to pr-784 2013-08-13 09:29:40 -07:00
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
Matei Zaharia e2fdac60da Merge pull request #802 from stayhf/SPARK-760-Python 
Simple PageRank algorithm implementation in Python for SPARK-760
 2013-08-12 21:26:59 -07:00
Matei Zaharia d3525babee Merge pull request #813 from AndreSchumacher/add_files_pyspark 
Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
 2013-08-12 21:02:39 -07:00
Andre Schumacher 8fd5c7bc00 Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark 
Now ADD_FILES uses a comma as file name separator.
 2013-08-12 20:22:52 -07:00
Matei Zaharia 9e02da2763 Merge pull request #812 from shivaram/maven-mllib-tests 
Create SparkContext in beforeAll for MLLib tests
 2013-08-12 20:22:27 -07:00
Matei Zaharia 65d0d91fba Merge pull request #807 from JoshRosen/guava-optional 
Change scala.Option to Guava Optional in Java APIs
 2013-08-12 19:00:57 -07:00
Josh Rosen cf08bb7a3e Fix import organization. 2013-08-12 18:55:02 -07:00
Evan Sparks 4346f0a1e9 Merge pull request #809 from shivaram/sgd-cleanup 
Clean up scaladoc in ML Lib.
 2013-08-12 12:12:12 -07:00
Matei Zaharia ea1b4baabd Merge pull request #806 from apivovarov/yarn-205 
Changed yarn.version to 2.0.5 in pom.xml
 2013-08-12 08:09:58 -07:00
Shivaram Venkataraman 8b5e3e2eb5 Add ML Lib scaladoc to API dropdown 2013-08-11 23:52:43 -07:00
jerryshao 09c7179e81 MetricsServlet code refactor according to comments 2013-08-12 13:23:23 +08:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Reynold Xin 2a39d2ca25 Merge pull request #810 from pwendell/dead_doc_code 
Remove now dead code inside of docs
 2013-08-11 20:35:09 -07:00
Patrick Wendell 9244524146 Removing dead docs 2013-08-11 20:33:58 -07:00
Shivaram Venkataraman 4935a2558b Clean up scaladoc in ML Lib. 
Also build and copy ML Lib scaladoc in Spark docs build.
Some more minor cleanup with respect to naming, test locations etc.
 2013-08-11 19:02:43 -07:00
Reynold Xin e5b9ed2833 Merge pull request #808 from pwendell/ui_compressed_bytes 
Report compressed bytes read when calculating TaskMetrics
 2013-08-11 17:22:47 -07:00
Shivaram Venkataraman ecc9bfe377 Create SparkContext in beforeAll for MLLib tests 
This overcomes test failures that occur using Maven
 2013-08-11 17:04:00 -07:00
Patrick Wendell 3d8f281604 Report compressed bytes read when calculating TaskMetrics 2013-08-11 16:25:57 -07:00
stayhf 24f02082c7 Code update for Matei's suggestions 2013-08-11 22:54:05 +00:00
Matei Zaharia 379648630b Merge pull request #805 from woggle/hadoop-rdd-jobconf 
Use new Configuration() instead of slower new JobConf() in SerializableWritable
 2013-08-11 14:51:47 -07:00
Josh Rosen d7f78b443b Change scala.Option to Guava Optional in Java APIs. 2013-08-11 12:05:09 -07:00
Evan Sparks ff9ebfabb4 Merge pull request #762 from shivaram/sgd-cleanup 
Refactor SGD options into a new class.
 2013-08-11 10:52:55 -07:00
shivaram 95c62ca306 Merge pull request #804 from apivovarov/master 
Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profi...
 2013-08-11 10:30:52 -07:00
Alexander Pivovarov 2d97cc46af Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profile 2013-08-10 23:04:50 -07:00
Alexander Pivovarov ca28f2e639 Changed yarn.version to 2.0.5 in pom.xml 2013-08-10 22:50:04 -07:00
Charles Reiss 6402b539d0 Use new Configuration() instead of new JobConf() for ObjectWritable. 
JobConf's constructor loads default config files in some verisons of
Hadoop, which is quite slow, and we only need the Configuration object
to pass the correct ClassLoader.
 2013-08-10 21:31:05 -07:00
Shivaram Venkataraman a65a6ed514 Fix GLM code review comments and move java tests 2013-08-10 18:54:10 -07:00
Matei Zaharia b13dae3ac6 Add a sample data file for PageRank 2013-08-10 18:13:49 -07:00