Patrick Wendell
4e9f0c2df6
Capturing GC detials in TaskMetrics
2013-08-13 16:25:55 -07:00
Patrick Wendell
f0382007dc
Bug fix for display of shuffle read/write metrics.
...
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
2013-08-13 16:25:55 -07:00
Patrick Wendell
ed6a1646e6
Slight change to pr-784
2013-08-13 09:29:40 -07:00
Patrick Wendell
a0133bfbad
Merge pull request #784 from jerryshao/dev-metrics-servlet
...
Add MetricsServlet for Spark metrics system
2013-08-13 09:28:18 -07:00
Matei Zaharia
65d0d91fba
Merge pull request #807 from JoshRosen/guava-optional
...
Change scala.Option to Guava Optional in Java APIs
2013-08-12 19:00:57 -07:00
Josh Rosen
cf08bb7a3e
Fix import organization.
2013-08-12 18:55:02 -07:00
jerryshao
09c7179e81
MetricsServlet code refactor according to comments
2013-08-12 13:23:23 +08:00
jerryshao
320e87e7ab
Add MetricsServlet for Spark metrics system
2013-08-12 13:23:23 +08:00
Reynold Xin
e5b9ed2833
Merge pull request #808 from pwendell/ui_compressed_bytes
...
Report compressed bytes read when calculating TaskMetrics
2013-08-11 17:22:47 -07:00
Patrick Wendell
3d8f281604
Report compressed bytes read when calculating TaskMetrics
2013-08-11 16:25:57 -07:00
Matei Zaharia
379648630b
Merge pull request #805 from woggle/hadoop-rdd-jobconf
...
Use new Configuration() instead of slower new JobConf() in SerializableWritable
2013-08-11 14:51:47 -07:00
Josh Rosen
d7f78b443b
Change scala.Option to Guava Optional in Java APIs.
2013-08-11 12:05:09 -07:00
Charles Reiss
6402b539d0
Use new Configuration() instead of new JobConf() for ObjectWritable.
...
JobConf's constructor loads default config files in some verisons of
Hadoop, which is quite slow, and we only need the Configuration object
to pass the correct ClassLoader.
2013-08-10 21:31:05 -07:00
Matei Zaharia
71c63de22f
Merge pull request #795 from mridulm/master
...
Fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
2013-08-10 10:21:20 -07:00
Matei Zaharia
d3277a0daf
Merge remote-tracking branch 'origin/pr/792'
...
Conflicts:
core/src/main/scala/spark/ui/jobs/IndexPage.scala
core/src/main/scala/spark/ui/jobs/StagePage.scala
2013-08-10 10:18:50 -07:00
Patrick Wendell
d17eeb997d
Merge pull request #785 from anfeng/master
...
expose HDFS file system stats via Executor metrics
2013-08-10 09:02:27 -07:00
Kay Ousterhout
14d14f451a
Shortened names, as per Matei's suggestion
2013-08-10 07:50:27 -07:00
Matei Zaharia
cd247ba5bb
Merge pull request #786 from shivaram/mllib-java
...
Java fixes, tests and examples for ALS, KMeans
2013-08-09 20:41:13 -07:00
Kay Ousterhout
7810a76512
Only print event queue full error message once
2013-08-09 18:20:48 -07:00
Kay Ousterhout
44ca8629d8
Style fix: removing unnecessary return type
2013-08-09 17:22:50 -07:00
Kay Ousterhout
29b79714f9
Style fixes based on code review
2013-08-09 16:46:34 -07:00
Kay Ousterhout
81e1d4a7d1
Refactored SparkListener to process all events asynchronously.
...
This commit fixes issues where SparkListeners that take a while to
process events slow the DAGScheduler.
This commit also fixes a bug in the UI where if a user goes to a
web page of a stage that does not exist, they can create a memory
leak (granted, this is not an issue at small scale -- probably only
an issue if someone actively tried to DOS the UI).
2013-08-09 13:27:41 -07:00
Matei Zaharia
b09d4b79e8
Merge pull request #799 from woggle/sync-fix
...
Remove extra synchronization in ResultTask
2013-08-09 13:17:08 -07:00
Patrick Wendell
cc6b92e80e
Merge pull request #775 from pwendell/print-launch-command
...
Log the launch command for Spark daemons
2013-08-09 13:00:33 -07:00
Patrick Wendell
3970b580c2
Using quotes when printing out command
2013-08-09 11:53:32 -07:00
Charles Reiss
9dfc280f74
Remove extra synchronization in ResultTask
2013-08-09 11:09:02 -07:00
Matei Zaharia
f94fc75c3f
Merge pull request #788 from shane-huang/sparkjavaopts
...
For standalone mode, add worker local env setting of SPARK_JAVA_OPTS as ...
2013-08-09 10:04:03 -07:00
Mridul Muralidharan
c230ca3b4e
Change line size
2013-08-08 22:28:40 +05:30
Mridul Muralidharan
dc47084f4e
Attempt to fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
2013-08-08 22:19:27 +05:30
Kay Ousterhout
88049a214d
Fixed 3 bugs that caused UI to crash (including SPARK-810).
...
One bug caused the UI to crash if you try to look at a job's status
before any of the tasks have finished.
The second bug was a concurrency issue where two different threads
(the scheduling thread and a UI thread) could be reading/updating
the data structures in JobProgressListener concurrently.
The third bug mis-used an Option, also causing the UI to crash
under certain conditions.
2013-08-07 23:09:25 -07:00
Patrick Wendell
b4321edf68
Reverting boostrap change
2013-08-07 22:18:18 -07:00
Patrick Wendell
21392f2a73
Change I forgot to merge in
2013-08-07 21:45:32 -07:00
Patrick Wendell
706394b370
Bumping font size to 14px and fixing sytle issue in progress bars
2013-08-07 21:27:04 -07:00
Patrick Wendell
8c0d668468
Merge branch 'master' into bootstrap-design
...
Conflicts:
core/src/main/scala/spark/ui/UIUtils.scala
core/src/main/scala/spark/ui/jobs/IndexPage.scala
core/src/main/scala/spark/ui/storage/RDDPage.scala
2013-08-07 21:06:03 -07:00
Kay Ousterhout
b88e26248e
Fixed issue in UI that limited scheduler throughput.
...
Removal of items from ArrayBuffers in the UI code was slow and
significantly impacted scheduler throughput. This commit
improves scheduler throughput by 5x.
2013-08-07 14:42:05 -07:00
shane-huang
cbc5107e36
For standalone mode, add worker local env setting of SPARK_JAVA_OPTS as default and let application env override default options if applicable
...
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
2013-08-07 14:36:48 +08:00
Matei Zaharia
6b043a6f11
Merge pull request #724 from dlyubimov/SPARK-826
...
SPARK-826: fold(), reduce(), collect() always attempt to use java serialization
2013-08-06 22:31:02 -07:00
Matei Zaharia
7c4b7a53b1
Merge remote-tracking branch 'origin/pr/781'
...
Conflicts:
core/src/main/resources/spark/ui/static/webui.css
2013-08-06 17:19:49 -07:00
Karen Feng
908032e79b
Used saturated colors for progress bars
2013-08-06 16:52:21 -07:00
Karen Feng
8bc497fa10
Lightened color of progress bars
2013-08-06 16:33:05 -07:00
Karen Feng
ca1903ea63
Overlays progress text on top of bar
2013-08-06 15:45:42 -07:00
Matei Zaharia
df4d10d630
Merge pull request #779 from adatao/adatao-global-SparkEnv
...
[HOTFIX] Extend thread safety for SparkEnv.get()
2013-08-06 15:44:05 -07:00
Shivaram Venkataraman
471fbadd0c
Java examples, tests for KMeans and ALS
...
- Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it
easier to call from Java
- Renames class methods from `train` to `run` to enable static methods to be
called from Java.
- Add unit tests which check if both static / class methods can be called.
- Also add examples which port the main() function in ALS, KMeans to the
examples project.
Couple of minor changes to existing code:
- Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily
- Workaround a bug where using double[] from Java leads to class cast exception in
KMeans init
2013-08-06 15:43:46 -07:00
anfeng
dda2ac8b5d
reformat registerFileSystemStat()
2013-08-06 15:22:25 -07:00
Karen Feng
099528b6c4
Pre-sorts stage/env tables, changes text/link of stage summaries
2013-08-06 14:52:12 -07:00
Karen Feng
254a930730
Reverse sorts StageTable by submitted time
2013-08-06 14:18:38 -07:00
Karen Feng
5ed5b73026
Sorts first column of env tables
2013-08-06 13:59:53 -07:00
anfeng
0748c60817
expose HDFS file system stats via Executor metrics
2013-08-06 11:47:06 -07:00
Reynold Xin
d031f73679
Merge pull request #782 from WANdisco/master
...
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
2013-08-05 22:33:00 -07:00
Matei Zaharia
1b63dea816
Merge pull request #769 from markhamstra/NegativeCores
...
SPARK-847 + SPARK-845: Zombie workers and negative cores
2013-08-05 22:21:26 -07:00