ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3510 commits 6 branches 181 tags 397 MiB
Commit graph

1653 commits

Author SHA1 Message Date
Matei Zaharia 7c4b7a53b1 Merge remote-tracking branch 'origin/pr/781' 
Conflicts:
	core/src/main/resources/spark/ui/static/webui.css
 2013-08-06 17:19:49 -07:00
Karen Feng 908032e79b Used saturated colors for progress bars 2013-08-06 16:52:21 -07:00
Karen Feng 8bc497fa10 Lightened color of progress bars 2013-08-06 16:33:05 -07:00
Karen Feng ca1903ea63 Overlays progress text on top of bar 2013-08-06 15:45:42 -07:00
Matei Zaharia df4d10d630 Merge pull request #779 from adatao/adatao-global-SparkEnv 
[HOTFIX] Extend thread safety for SparkEnv.get()
 2013-08-06 15:44:05 -07:00
Karen Feng 099528b6c4 Pre-sorts stage/env tables, changes text/link of stage summaries 2013-08-06 14:52:12 -07:00
Karen Feng 254a930730 Reverse sorts StageTable by submitted time 2013-08-06 14:18:38 -07:00
Karen Feng 5ed5b73026 Sorts first column of env tables 2013-08-06 13:59:53 -07:00
Reynold Xin d031f73679 Merge pull request #782 from WANdisco/master 
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
 2013-08-05 22:33:00 -07:00
Matei Zaharia 1b63dea816 Merge pull request #769 from markhamstra/NegativeCores 
SPARK-847 + SPARK-845: Zombie workers and negative cores
 2013-08-05 22:21:26 -07:00
Alexander Pivovarov a30866438b SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD 2013-08-05 21:48:43 -07:00
Matei Zaharia 8b277892c9 Merge pull request #774 from pwendell/job-description 
Show user-defined job name in UI
 2013-08-05 19:14:52 -07:00
Christopher Nguyen b1bbbe699c [HOTFIX] Mark lastSetSparkEnv @volatile in case it gets HotSpot-cached 
On branch adatao-global-SparkEnv
Changes to be committed:

	modified:   core/src/main/scala/spark/SparkEnv.scala
 2013-08-05 17:22:27 -07:00
Mark Hamstra 35d8f5ee52 Moved handling of timed out workers within the Master actor 2013-08-05 13:13:56 -07:00
Mark Hamstra 37ccf9301a milliseconds -> seconds in timeOutDeadWorkers logging 2013-08-05 13:13:56 -07:00
Mark Hamstra cdd1af562e Timeout zombie workers 2013-08-05 13:13:56 -07:00
Mikhail Bautin e8bec8365f Only reduce the number of cores once when removing an executor 2013-08-05 13:13:56 -07:00
Karen Feng 95025afdec Made most small fixes for SPARK-849 except for table sort, task progress overlay 2013-08-05 13:04:56 -07:00
Bill Zhao 87134b3648 SPARK-850: give better console message 2013-08-05 11:55:35 -07:00
Christopher Nguyen 39e4fda76f [HOTFIX] Extend thread safety for SparkEnv.get() 
A ThreadLocal SparkEnv.env is facing various situations leading to
NullPointerExceptions, where SparkEnv.env set in one thread is not
gettable in another thread, but often assumed to be available.

See, e.g., https://groups.google.com/forum/#!topic/spark-developers/GLx8yunSj0A

This hotfixes SparkEnv.env to return either (a) the ThreadLocal
value if non-null, or (b) the previously set value in any thread.

This approach preserves SparkEnv.set() thread safety needed by
RDD.compute() and possibly other places. A refactoring that
parameterizes SparkEnv should be addressed subsequently.

On branch adatao-global-SparkEnv
Changes to be committed:

	modified:   core/src/main/scala/spark/SparkEnv.scala
 2013-08-05 02:09:54 -07:00
Matei Zaharia 22abbc10d6 Merge pull request #772 from karenfeng/ui-843 
Show app duration
 2013-08-02 16:37:59 -07:00
Patrick Wendell 5b3784a79c Show user-defined job name in UI 2013-08-02 15:47:41 -07:00
Karen Feng b3ae5b25d5 Shows time the app has been running 2013-08-02 13:25:14 -07:00
Patrick Wendell 9d7dfd2d5a Merge pull request #743 from pwendell/app-metrics 
Add application metrics to standalone master
 2013-08-01 17:41:58 -07:00
Patrick Wendell f1d2ad550e under_scores --> camelCase for config options 2013-08-01 15:26:26 -07:00
Patrick Wendell 12d9c82c9b Small style fix 2013-08-01 15:25:52 -07:00
Patrick Wendell 37bc64a205 Adding application-level metrics. 
This adds metrics for applications in the deploy Master.
 2013-08-01 15:25:52 -07:00
Patrick Wendell 87fd321a5a Minor refactoring and code cleanup 2013-08-01 15:02:31 -07:00
Patrick Wendell b10199413a Slight refactoring to SparkContext functions 2013-08-01 15:00:42 -07:00
Patrick Wendell cfcd77b5da Increasing inter job arrival 2013-08-01 15:00:42 -07:00
Patrick Wendell 5faac7f4f3 Minor style fixes 2013-08-01 15:00:42 -07:00
Patrick Wendell 5e7b38fbb3 Merge pull request #695 from xiajunluan/pool_ui 
Enhance job ui in spark ui system with adding pool information
 2013-08-01 14:59:33 -07:00
Matei Zaharia 0a96493ac6 Merge pull request #760 from karenfeng/heading-update 
Clean up web UI page headers
 2013-08-01 11:27:17 -07:00
Patrick Wendell 9177bea2b4 Removing extra imports 2013-08-01 10:42:50 -07:00
Patrick Wendell 3e4d5e5f8b Merge branch 'master' into master-json 
Conflicts:
	core/src/main/scala/spark/deploy/master/ui/IndexPage.scala
 2013-08-01 10:42:07 -07:00
Patrick Wendell ffc034e4fb Import cleanup 2013-08-01 10:39:56 -07:00
Andrew xia d58502a156 fix bug of spark "SubmitStage" listener as unit test error 2013-08-01 23:21:41 +08:00
Andrew xia 3b5a11e765 change function name "setName" to "setProperties" as "setName" is also member of Thread class 2013-08-01 19:37:15 +08:00
Matei Zaharia 3097d75d6f Merge remote-tracking branch 'dlyubimov/SPARK-827' 
Conflicts:
	docs/configuration.md
 2013-07-31 18:36:43 -07:00
Matei Zaharia 39c75f3033 Merge pull request #757 from BlackNiuza/result_task_generation 
Bug fix: SPARK-837
 2013-07-31 15:52:36 -07:00
Matei Zaharia 14bf2fe039 Merge pull request #749 from benh/spark-executor-uri 
Added property 'spark.executor.uri' for launching on Mesos.
 2013-07-31 14:18:16 -07:00
Benjamin Hindman 4692ea4892 Used 'uri.split('/').last' instead of 'new File(uri).getName()'. 2013-07-31 12:29:44 -07:00
Karen Feng c453967f9a Reduced size of heading 2013-07-31 11:57:50 -07:00
Matei Zaharia a386ced2c6 Merge pull request #754 from rxin/compression 
Compression codec change
 2013-07-31 11:22:50 -07:00
Karen Feng 49e6344142 Removed master URL from job UI, reduced heading size of basic spark pages 2013-07-31 11:17:59 -07:00
Reynold Xin c61843a69f Changed other LZF uses to use the compression codec interface. 2013-07-31 10:32:13 -07:00
Patrick Wendell 89da9d94b3 Add JSON path to master index page 2013-07-31 09:47:53 -07:00
BlackNiuza 9a815de4bf write and read generation in ResultTask 2013-08-01 00:36:47 +08:00
Roman Tkalenko 0c6553714a Refactored Vector.apply(length, initializer) replacing excessive code with library method 
(also removed unused variable ```ans``` as minor change)
 2013-07-31 19:05:46 +03:00
Matei Zaharia 12553e5c55 Simplified nonNegativeMod to match previous version 2013-07-31 08:50:28 -07:00