ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3632 commits 6 branches 181 tags 397 MiB
Commit graph

1861 commits

Author SHA1 Message Date
Patrick Wendell a0133bfbad Merge pull request #784 from jerryshao/dev-metrics-servlet 
Add MetricsServlet for Spark metrics system
 2013-08-13 09:28:18 -07:00
Matei Zaharia 65d0d91fba Merge pull request #807 from JoshRosen/guava-optional 
Change scala.Option to Guava Optional in Java APIs
 2013-08-12 19:00:57 -07:00
Josh Rosen cf08bb7a3e Fix import organization. 2013-08-12 18:55:02 -07:00
jerryshao 09c7179e81 MetricsServlet code refactor according to comments 2013-08-12 13:23:23 +08:00
jerryshao 320e87e7ab Add MetricsServlet for Spark metrics system 2013-08-12 13:23:23 +08:00
Reynold Xin e5b9ed2833 Merge pull request #808 from pwendell/ui_compressed_bytes 
Report compressed bytes read when calculating TaskMetrics
 2013-08-11 17:22:47 -07:00
Patrick Wendell 3d8f281604 Report compressed bytes read when calculating TaskMetrics 2013-08-11 16:25:57 -07:00
Matei Zaharia 379648630b Merge pull request #805 from woggle/hadoop-rdd-jobconf 
Use new Configuration() instead of slower new JobConf() in SerializableWritable
 2013-08-11 14:51:47 -07:00
Josh Rosen d7f78b443b Change scala.Option to Guava Optional in Java APIs. 2013-08-11 12:05:09 -07:00
Charles Reiss 6402b539d0 Use new Configuration() instead of new JobConf() for ObjectWritable. 
JobConf's constructor loads default config files in some verisons of
Hadoop, which is quite slow, and we only need the Configuration object
to pass the correct ClassLoader.
 2013-08-10 21:31:05 -07:00
Matei Zaharia 71c63de22f Merge pull request #795 from mridulm/master 
Fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
 2013-08-10 10:21:20 -07:00
Matei Zaharia d3277a0daf Merge remote-tracking branch 'origin/pr/792' 
Conflicts:
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/jobs/StagePage.scala
 2013-08-10 10:18:50 -07:00
Patrick Wendell d17eeb997d Merge pull request #785 from anfeng/master 
expose HDFS file system stats via Executor metrics
 2013-08-10 09:02:27 -07:00
Kay Ousterhout 14d14f451a Shortened names, as per Matei's suggestion 2013-08-10 07:50:27 -07:00
Matei Zaharia cd247ba5bb Merge pull request #786 from shivaram/mllib-java 
Java fixes, tests and examples for ALS, KMeans
 2013-08-09 20:41:13 -07:00
Kay Ousterhout 7810a76512 Only print event queue full error message once 2013-08-09 18:20:48 -07:00
Kay Ousterhout 44ca8629d8 Style fix: removing unnecessary return type 2013-08-09 17:22:50 -07:00
Kay Ousterhout 29b79714f9 Style fixes based on code review 2013-08-09 16:46:34 -07:00
Kay Ousterhout 81e1d4a7d1 Refactored SparkListener to process all events asynchronously. 
This commit fixes issues where SparkListeners that take a while to
process events slow the DAGScheduler.

This commit also fixes a bug in the UI where if a user goes to a
web page of a stage that does not exist, they can create a memory
leak (granted, this is not an issue at small scale -- probably only
an issue if someone actively tried to DOS the UI).
 2013-08-09 13:27:41 -07:00
Matei Zaharia b09d4b79e8 Merge pull request #799 from woggle/sync-fix 
Remove extra synchronization in ResultTask
 2013-08-09 13:17:08 -07:00
Patrick Wendell cc6b92e80e Merge pull request #775 from pwendell/print-launch-command 
Log the launch command for Spark daemons
 2013-08-09 13:00:33 -07:00
Patrick Wendell 3970b580c2 Using quotes when printing out command 2013-08-09 11:53:32 -07:00
Charles Reiss 9dfc280f74 Remove extra synchronization in ResultTask 2013-08-09 11:09:02 -07:00
Matei Zaharia f94fc75c3f Merge pull request #788 from shane-huang/sparkjavaopts 
For standalone mode, add worker local env setting of SPARK_JAVA_OPTS as ...
 2013-08-09 10:04:03 -07:00
Matei Zaharia d1e1c1b24d Add test for Kryo with WrappedArray (which was failing in Chill 0.3.0) 2013-08-08 13:34:11 -07:00
Mridul Muralidharan c230ca3b4e Change line size 2013-08-08 22:28:40 +05:30
Mridul Muralidharan dc47084f4e Attempt to fix bug reported in PR 791 : a race condition in ConnectionManager and Connection 2013-08-08 22:19:27 +05:30
Kay Ousterhout 88049a214d Fixed 3 bugs that caused UI to crash (including SPARK-810). 
One bug caused the UI to crash if you try to look at a job's status
before any of the tasks have finished.

The second bug was a concurrency issue where two different threads
(the scheduling thread and a UI thread) could be reading/updating
the data structures in JobProgressListener concurrently.

The third bug mis-used an Option, also causing the UI to crash
under certain conditions.
 2013-08-07 23:09:25 -07:00
Patrick Wendell b4321edf68 Reverting boostrap change 2013-08-07 22:18:18 -07:00
Patrick Wendell 21392f2a73 Change I forgot to merge in 2013-08-07 21:45:32 -07:00
Patrick Wendell 706394b370 Bumping font size to 14px and fixing sytle issue in progress bars 2013-08-07 21:27:04 -07:00
Patrick Wendell 8c0d668468 Merge branch 'master' into bootstrap-design 
Conflicts:
	core/src/main/scala/spark/ui/UIUtils.scala
	core/src/main/scala/spark/ui/jobs/IndexPage.scala
	core/src/main/scala/spark/ui/storage/RDDPage.scala
 2013-08-07 21:06:03 -07:00
Kay Ousterhout b88e26248e Fixed issue in UI that limited scheduler throughput. 
Removal of items from ArrayBuffers in the UI code was slow and
significantly impacted scheduler throughput. This commit
improves scheduler throughput by 5x.
 2013-08-07 14:42:05 -07:00
shane-huang cbc5107e36 For standalone mode, add worker local env setting of SPARK_JAVA_OPTS as default and let application env override default options if applicable 
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
 2013-08-07 14:36:48 +08:00
Matei Zaharia 6b043a6f11 Merge pull request #724 from dlyubimov/SPARK-826 
SPARK-826: fold(), reduce(), collect() always attempt to use java serialization
 2013-08-06 22:31:02 -07:00
Matei Zaharia 7c4b7a53b1 Merge remote-tracking branch 'origin/pr/781' 
Conflicts:
	core/src/main/resources/spark/ui/static/webui.css
 2013-08-06 17:19:49 -07:00
Karen Feng 908032e79b Used saturated colors for progress bars 2013-08-06 16:52:21 -07:00
Karen Feng 8bc497fa10 Lightened color of progress bars 2013-08-06 16:33:05 -07:00
Karen Feng ca1903ea63 Overlays progress text on top of bar 2013-08-06 15:45:42 -07:00
Matei Zaharia df4d10d630 Merge pull request #779 from adatao/adatao-global-SparkEnv 
[HOTFIX] Extend thread safety for SparkEnv.get()
 2013-08-06 15:44:05 -07:00
Shivaram Venkataraman 471fbadd0c Java examples, tests for KMeans and ALS 
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
  easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
  called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
  examples project.

Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
  KMeans init
 2013-08-06 15:43:46 -07:00
anfeng dda2ac8b5d reformat registerFileSystemStat() 2013-08-06 15:22:25 -07:00
Karen Feng 099528b6c4 Pre-sorts stage/env tables, changes text/link of stage summaries 2013-08-06 14:52:12 -07:00
Karen Feng 254a930730 Reverse sorts StageTable by submitted time 2013-08-06 14:18:38 -07:00
Karen Feng 5ed5b73026 Sorts first column of env tables 2013-08-06 13:59:53 -07:00
anfeng 0748c60817 expose HDFS file system stats via Executor metrics 2013-08-06 11:47:06 -07:00
Reynold Xin d031f73679 Merge pull request #782 from WANdisco/master 
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
 2013-08-05 22:33:00 -07:00
Matei Zaharia 1b63dea816 Merge pull request #769 from markhamstra/NegativeCores 
SPARK-847 + SPARK-845: Zombie workers and negative cores
 2013-08-05 22:21:26 -07:00
Alexander Pivovarov a30866438b SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD 2013-08-05 21:48:43 -07:00
Matei Zaharia 8b277892c9 Merge pull request #774 from pwendell/job-description 
Show user-defined job name in UI
 2013-08-05 19:14:52 -07:00