ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3682 commits 6 branches 181 tags 397 MiB
Commit graph

3682 commits

This branch
This branch
All branches
Author SHA1 Message Date
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
Matei Zaharia e2fdac60da Merge pull request #802 from stayhf/SPARK-760-Python 
Simple PageRank algorithm implementation in Python for SPARK-760
 2013-08-12 21:26:59 -07:00
Matei Zaharia d3525babee Merge pull request #813 from AndreSchumacher/add_files_pyspark 
Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
 2013-08-12 21:02:39 -07:00
Andre Schumacher 8fd5c7bc00 Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark 
Now ADD_FILES uses a comma as file name separator.
 2013-08-12 20:22:52 -07:00
Matei Zaharia 9e02da2763 Merge pull request #812 from shivaram/maven-mllib-tests 
Create SparkContext in beforeAll for MLLib tests
 2013-08-12 20:22:27 -07:00
Matei Zaharia 65d0d91fba Merge pull request #807 from JoshRosen/guava-optional 
Change scala.Option to Guava Optional in Java APIs
 2013-08-12 19:00:57 -07:00
Josh Rosen cf08bb7a3e Fix import organization. 2013-08-12 18:55:02 -07:00
Evan Sparks 4346f0a1e9 Merge pull request #809 from shivaram/sgd-cleanup 
Clean up scaladoc in ML Lib.
 2013-08-12 12:12:12 -07:00
Matei Zaharia ea1b4baabd Merge pull request #806 from apivovarov/yarn-205 
Changed yarn.version to 2.0.5 in pom.xml
 2013-08-12 08:09:58 -07:00
Shivaram Venkataraman 8b5e3e2eb5 Add ML Lib scaladoc to API dropdown 2013-08-11 23:52:43 -07:00
jerryshao 09c7179e81 MetricsServlet code refactor according to comments 2013-08-12 13:23:23 +08:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Reynold Xin 2a39d2ca25 Merge pull request #810 from pwendell/dead_doc_code 
Remove now dead code inside of docs
 2013-08-11 20:35:09 -07:00
Patrick Wendell 9244524146 Removing dead docs 2013-08-11 20:33:58 -07:00
Shivaram Venkataraman 4935a2558b Clean up scaladoc in ML Lib. 
Also build and copy ML Lib scaladoc in Spark docs build.
Some more minor cleanup with respect to naming, test locations etc.
 2013-08-11 19:02:43 -07:00
Reynold Xin e5b9ed2833 Merge pull request #808 from pwendell/ui_compressed_bytes 
Report compressed bytes read when calculating TaskMetrics
 2013-08-11 17:22:47 -07:00
Shivaram Venkataraman ecc9bfe377 Create SparkContext in beforeAll for MLLib tests 
This overcomes test failures that occur using Maven
 2013-08-11 17:04:00 -07:00
Patrick Wendell 3d8f281604 Report compressed bytes read when calculating TaskMetrics 2013-08-11 16:25:57 -07:00
stayhf 24f02082c7 Code update for Matei's suggestions 2013-08-11 22:54:05 +00:00
Matei Zaharia 379648630b Merge pull request #805 from woggle/hadoop-rdd-jobconf 
Use new Configuration() instead of slower new JobConf() in SerializableWritable
 2013-08-11 14:51:47 -07:00
Josh Rosen d7f78b443b Change scala.Option to Guava Optional in Java APIs. 2013-08-11 12:05:09 -07:00
Evan Sparks ff9ebfabb4 Merge pull request #762 from shivaram/sgd-cleanup 
Refactor SGD options into a new class.
 2013-08-11 10:52:55 -07:00
shivaram 95c62ca306 Merge pull request #804 from apivovarov/master 
Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profi...
 2013-08-11 10:30:52 -07:00
Alexander Pivovarov 2d97cc46af Fixed path to JavaALS.java and JavaKMeans.java, fixed hadoop2-yarn profile 2013-08-10 23:04:50 -07:00
Alexander Pivovarov ca28f2e639 Changed yarn.version to 2.0.5 in pom.xml 2013-08-10 22:50:04 -07:00
Charles Reiss 6402b539d0 Use new Configuration() instead of new JobConf() for ObjectWritable. 
JobConf's constructor loads default config files in some verisons of
Hadoop, which is quite slow, and we only need the Configuration object
to pass the correct ClassLoader.
 2013-08-10 21:31:05 -07:00
Shivaram Venkataraman a65a6ed514 Fix GLM code review comments and move java tests 2013-08-10 18:54:10 -07:00
Matei Zaharia b13dae3ac6 Add a sample data file for PageRank 2013-08-10 18:13:49 -07:00
Matei Zaharia 4c4f769187 Optimize Scala PageRank to use reduceByKey 2013-08-10 18:09:54 -07:00
Matei Zaharia 06e4f2a8f2 Merge pull request #789 from MLnick/master 
Adding Scala version of PageRank example
 2013-08-10 18:06:23 -07:00
stayhf 55d9bde2fa Simple PageRank algorithm implementation in Python for SPARK-760 2013-08-10 23:48:51 +00:00
Matei Zaharia 71c63de22f Merge pull request #795 from mridulm/master 
Fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
 2013-08-10 10:21:20 -07:00
Matei Zaharia d3277a0daf Merge remote-tracking branch 'origin/pr/792' 
Conflicts:
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/jobs/StagePage.scala
 2013-08-10 10:18:50 -07:00
Patrick Wendell d17eeb997d Merge pull request #785 from anfeng/master 
expose HDFS file system stats via Executor metrics
 2013-08-10 09:02:27 -07:00
Kay Ousterhout 14d14f451a Shortened names, as per Matei's suggestion 2013-08-10 07:50:27 -07:00
Matei Zaharia dce5e47435 Merge pull request #800 from dlyubimov/HBASE_VERSION 
Pull HBASE_VERSION in the head of sbt build
 2013-08-09 21:53:45 -07:00
Matei Zaharia cd247ba5bb Merge pull request #786 from shivaram/mllib-java 
Java fixes, tests and examples for ALS, KMeans
 2013-08-09 20:41:13 -07:00
Kay Ousterhout 7810a76512 Only print event queue full error message once 2013-08-09 18:20:48 -07:00
Kay Ousterhout 44ca8629d8 Style fix: removing unnecessary return type 2013-08-09 17:22:50 -07:00
Kay Ousterhout 29b79714f9 Style fixes based on code review 2013-08-09 16:46:34 -07:00
Dmitriy Lyubimov 27f674f82b fewer words 2013-08-09 13:54:41 -07:00
Kay Ousterhout 81e1d4a7d1 Refactored SparkListener to process all events asynchronously. 
This commit fixes issues where SparkListeners that take a while to
process events slow the DAGScheduler.

This commit also fixes a bug in the UI where if a user goes to a
web page of a stage that does not exist, they can create a memory
leak (granted, this is not an issue at small scale -- probably only
an issue if someone actively tried to DOS the UI).
 2013-08-09 13:27:41 -07:00
Matei Zaharia b09d4b79e8 Merge pull request #799 from woggle/sync-fix 
Remove extra synchronization in ResultTask
 2013-08-09 13:17:08 -07:00
Patrick Wendell cc6b92e80e Merge pull request #775 from pwendell/print-launch-command 
Log the launch command for Spark daemons
 2013-08-09 13:00:33 -07:00
Dmitriy Lyubimov ae95b57469 Pull HBASE_VERSION in the head of sbt build 2013-08-09 12:45:18 -07:00
Patrick Wendell 3970b580c2 Using quotes when printing out command 2013-08-09 11:53:32 -07:00
Charles Reiss 9dfc280f74 Remove extra synchronization in ResultTask 2013-08-09 11:09:02 -07:00
Matei Zaharia f94fc75c3f Merge pull request #788 from shane-huang/sparkjavaopts 
For standalone mode, add worker local env setting of SPARK_JAVA_OPTS as ...
 2013-08-09 10:04:03 -07:00
shane-huang 34c1f8d2da add comment in spark-env.sh.template for SPARK_JAVA_OPTS 
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
 2013-08-09 15:03:48 +08:00
Matei Zaharia 06303a62e5 Optimize JavaPageRank to use reduceByKey instead of groupByKey 2013-08-08 18:50:00 -07:00