ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3828 commits 6 branches 181 tags 397 MiB
Commit graph

3828 commits

This branch
This branch
All branches
Author SHA1 Message Date
stayhf 882baee489 Got rid of unnecessary map function 2013-08-06 21:34:39 +00:00
Karen Feng 254a930730 Reverse sorts StageTable by submitted time 2013-08-06 14:18:38 -07:00
stayhf 326a7a82e0 changes as reviewer requested 2013-08-06 21:03:24 +00:00
Karen Feng 5ed5b73026 Sorts first column of env tables 2013-08-06 13:59:53 -07:00
anfeng 0748c60817 expose HDFS file system stats via Executor metrics 2013-08-06 11:47:06 -07:00
Reynold Xin d031f73679 Merge pull request #782 from WANdisco/master 
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
 2013-08-05 22:33:00 -07:00
Matei Zaharia 1b63dea816 Merge pull request #769 from markhamstra/NegativeCores 
SPARK-847 + SPARK-845: Zombie workers and negative cores
 2013-08-05 22:21:26 -07:00
Alexander Pivovarov a30866438b SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD 2013-08-05 21:48:43 -07:00
Matei Zaharia 828aff744d Merge pull request #776 from gingsmith/master 
adding matrix factorization data generator
 2013-08-05 21:37:33 -07:00
Ginger Smith bf7033f3eb fixing formatting, style, and input 2013-08-05 21:26:24 -07:00
Matei Zaharia 8b277892c9 Merge pull request #774 from pwendell/job-description 
Show user-defined job name in UI
 2013-08-05 19:14:52 -07:00
Christopher Nguyen b1bbbe699c [HOTFIX] Mark lastSetSparkEnv @volatile in case it gets HotSpot-cached 
On branch adatao-global-SparkEnv
Changes to be committed:

	modified:   core/src/main/scala/spark/SparkEnv.scala
 2013-08-05 17:22:27 -07:00
Mark Hamstra 35d8f5ee52 Moved handling of timed out workers within the Master actor 2013-08-05 13:13:56 -07:00
Mark Hamstra 37ccf9301a milliseconds -> seconds in timeOutDeadWorkers logging 2013-08-05 13:13:56 -07:00
Mark Hamstra cdd1af562e Timeout zombie workers 2013-08-05 13:13:56 -07:00
Mikhail Bautin e8bec8365f Only reduce the number of cores once when removing an executor 2013-08-05 13:13:56 -07:00
Karen Feng 95025afdec Made most small fixes for SPARK-849 except for table sort, task progress overlay 2013-08-05 13:04:56 -07:00
Patrick Wendell 550b0cf48a Merge pull request #780 from cybermaster/master 
SPARK-850
 2013-08-05 12:10:32 -07:00
Bill Zhao 33b9b155de JBoss repository working now 2013-08-05 12:02:36 -07:00
Bill Zhao 9df66cd831 Merge branch 'master' of github.com:cybermaster/spark 2013-08-05 11:56:30 -07:00
Bill Zhao 87134b3648 SPARK-850: give better console message 2013-08-05 11:55:35 -07:00
Ginger Smith 8c8947e2b6 fixing formatting 2013-08-05 11:22:18 -07:00
Bill Zhao d93d5fcaac SPARK-850: Give better error message on the console 2013-08-05 10:09:03 -07:00
Christopher Nguyen 39e4fda76f [HOTFIX] Extend thread safety for SparkEnv.get() 
A ThreadLocal SparkEnv.env is facing various situations leading to
NullPointerExceptions, where SparkEnv.env set in one thread is not
gettable in another thread, but often assumed to be available.

See, e.g., https://groups.google.com/forum/#!topic/spark-developers/GLx8yunSj0A

This hotfixes SparkEnv.env to return either (a) the ThreadLocal
value if non-null, or (b) the previously set value in any thread.

This approach preserves SparkEnv.set() thread safety needed by
RDD.compute() and possibly other places. A refactoring that
parameterizes SparkEnv should be addressed subsequently.

On branch adatao-global-SparkEnv
Changes to be committed:

	modified:   core/src/main/scala/spark/SparkEnv.scala
 2013-08-05 02:09:54 -07:00
stayhf 98fd62605d Updated code with reviewer's suggestions 2013-08-05 00:30:28 +00:00
Patrick Wendell f3660d5ab8 Make output formatting consistent between bash/scala 2013-08-03 21:30:15 -07:00
Shivaram Venkataraman 7388e27668 Move implicit arg to constructor for Java access. 2013-08-03 18:08:43 -07:00
Patrick Wendell ad94fbb322 Log the launch command for Spark executors 2013-08-03 09:19:46 -07:00
stayhf a682637301 Simple PageRank algorithm implementation in Java for SPARK-760 2013-08-03 06:01:16 +00:00
Ginger Smith 4ab4df5edb adding matrix factorization data generator 2013-08-02 22:22:36 -07:00
Shivaram Venkataraman 00339cc032 Refactor optimizers and create GLMs 
This change refactors the structure of GLMs to use mixins which maintain
a similar interface to other ML lib algorithms. This change also creates
an Optimizer trait which allows GLMs to be extended to use other optimization
techniques.
 2013-08-02 19:15:34 -07:00
Patrick Wendell b4905c383b Log the launch command for Spark daemons 
For debugging and analysis purposes, it's nice to have the exact command
used to launch Spark contained within the logs. This adds the necessary
hooks to make that possible.
 2013-08-02 16:58:19 -07:00
Matei Zaharia 22abbc10d6 Merge pull request #772 from karenfeng/ui-843 
Show app duration
 2013-08-02 16:37:59 -07:00
Matei Zaharia abfa9e6f70 Increase Kryo buffer size in ALS since some arrays become big 2013-08-02 16:17:32 -07:00
Patrick Wendell 5b3784a79c Show user-defined job name in UI 2013-08-02 15:47:41 -07:00
Karen Feng b3ae5b25d5 Shows time the app has been running 2013-08-02 13:25:14 -07:00
Patrick Wendell 9d7dfd2d5a Merge pull request #743 from pwendell/app-metrics 
Add application metrics to standalone master
 2013-08-01 17:41:58 -07:00
Patrick Wendell 6d7afd7ced Merge pull request #768 from pwendell/pr-695 
Minor clean-up of fair scheduler UI
 2013-08-01 17:13:28 -07:00
Karen Feng 999eaac78c Merge branch 'master' of https://github.com/mesos/spark 2013-08-01 15:54:56 -07:00
Matei Zaharia e466a55a6b Revert Mesos version to 0.9 since the 0.12 artifact has target Java 7 2013-08-01 15:45:21 -07:00
Patrick Wendell 1105766295 Point to new repository/branch 2013-08-01 15:43:41 -07:00
Patrick Wendell d3c37ff120 Improving documentation in config file example 2013-08-01 15:26:26 -07:00
Patrick Wendell f1d2ad550e under_scores --> camelCase for config options 2013-08-01 15:26:26 -07:00
Patrick Wendell 12d9c82c9b Small style fix 2013-08-01 15:25:52 -07:00
Patrick Wendell 37bc64a205 Adding application-level metrics. 
This adds metrics for applications in the deploy Master.
 2013-08-01 15:25:52 -07:00
Karen Feng 73692f3cb9 Unify, reduce body font size 2013-08-01 15:10:30 -07:00
Patrick Wendell 87fd321a5a Minor refactoring and code cleanup 2013-08-01 15:02:31 -07:00
Patrick Wendell b10199413a Slight refactoring to SparkContext functions 2013-08-01 15:00:42 -07:00
Patrick Wendell cfcd77b5da Increasing inter job arrival 2013-08-01 15:00:42 -07:00
Patrick Wendell 5faac7f4f3 Minor style fixes 2013-08-01 15:00:42 -07:00