Matei Zaharia
d4556f4207
Merge pull request #751 from cdshines/master
...
Cleaned Partitioner & PythonPartitioner source by taking out non-related logic to Utils
2013-07-31 08:48:14 -07:00
Andrew xia
5670c96f29
Merge branch 'master' into Pool_UI
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/scheduler/DAGScheduler.scala
core/src/main/scala/spark/scheduler/SparkListener.scala
core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
core/src/main/scala/spark/ui/jobs/IndexPage.scala
core/src/main/scala/spark/ui/jobs/JobProgressUI.scala
2013-07-31 19:36:36 +08:00
cdshines
fefb03cbd7
Eliminated code duplication, refactored to pattern-matching style Partitioner and PythonPartitioner
2013-07-31 13:19:42 +03:00
jerryshao
bf9318091a
Add Apache license header to metrics system
2013-07-31 09:42:16 +08:00
Reynold Xin
98024eadc3
Renamed compressionOutputStream and compressionInputStream to compressedOutputStream and compressedInputStream.
2013-07-30 18:28:46 -07:00
Reynold Xin
dae12fef9e
Updated the configuration option for Snappy block size to be consistent with the documentation.
2013-07-30 17:49:31 -07:00
Reynold Xin
ad7e9d0d64
CompressionCodec cleanup. Moved it to spark.io package.
2013-07-30 17:11:54 -07:00
Reynold Xin
368c58eac5
Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression
...
Conflicts:
project/SparkBuild.scala
2013-07-30 16:04:18 -07:00
Patrick Wendell
e87de037d6
Merge pull request #744 from karenfeng/bootstrap-update
...
Use Bootstrap progress bars in web UI
2013-07-30 15:00:08 -07:00
Karen Feng
26144c400f
Fixed wrap style
2013-07-30 12:40:41 -07:00
Karen Feng
218d7c4ed8
Fixed style, lowered height of progress bars
2013-07-30 12:39:17 -07:00
Karen Feng
f1cab31b73
Removed intermediate set for activeTasks, removed progress bar margin
2013-07-30 11:06:47 -07:00
Benjamin Hindman
f6f46455eb
Added property 'spark.executor.uri' for launching on Mesos without
...
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
2013-07-29 23:32:52 -07:00
Josh Rosen
49be084ed3
Use File.pathSeparator instead of hardcoding ':'.
2013-07-29 22:08:57 -07:00
Josh Rosen
b95732632b
Do not inherit master's PYTHONPATH on workers.
...
This fixes SPARK-832, an issue where PySpark
would not work when the master and workers used
different SPARK_HOME paths.
This change may potentially break code that relied
on the master's PYTHONPATH being used on workers.
To have custom PYTHONPATH additions used on the
workers, users should set a custom PYTHONPATH in
spark-env.sh rather than setting it in the shell.
2013-07-29 22:08:57 -07:00
Andrew xia
5406013997
refactor codes less than 100 character per line
2013-07-30 11:41:38 +08:00
Andrew xia
614ee16cc4
refactor job ui with pool information
2013-07-30 10:57:26 +08:00
Reynold Xin
81720e13fc
Moved all StandaloneClusterMessage's into StandaloneClusterMessages object.
2013-07-29 17:53:01 -07:00
Reynold Xin
23b5da14ed
Moved block manager messages into BlockManagerMessages object.
2013-07-29 17:42:05 -07:00
Reynold Xin
105f4d22e9
Removed Cache and SoftReferenceCache since they are no longer used.
2013-07-29 17:30:38 -07:00
Reynold Xin
17e62113d4
Moved DeployMessage's into its own DeployMessages object.
...
Also renamed MasterState to MasterStateResponse and WorkerState to WorkerStateResponse for clarity.
2013-07-29 17:14:44 -07:00
Karen Feng
87b821dc39
Fixed continuity of executorToTasksActive, changed color of progress bars
2013-07-29 16:50:51 -07:00
Karen Feng
c7b2788948
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
...
Conflicts:
core/src/main/scala/spark/ui/jobs/IndexPage.scala
2013-07-29 16:36:07 -07:00
Patrick Wendell
c99b674405
Merge pull request #735 from karenfeng/ui-807
...
Totals for shuffle data and CPU time
2013-07-29 16:32:55 -07:00
Karen Feng
2d6da9195a
Alphabetized imports
2013-07-29 15:50:52 -07:00
Karen Feng
478a2886d9
Added started tasks to progress bar
2013-07-29 14:51:07 -07:00
Karen Feng
e04a37a332
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
...
cially if it merges an updated upstream into a topic branch.
2013-07-29 14:32:48 -07:00
Reynold Xin
fe7298b587
Merge pull request #741 from pwendell/usability
...
Fix two small usability issues
2013-07-29 14:01:00 -07:00
Karen Feng
43a2cc15c0
Use Bootstrap progress bars in web UI
2013-07-29 13:37:24 -07:00
Matei Zaharia
b9d6783f36
Optimize Python take() to not compute entire first partition
2013-07-29 02:51:43 -04:00
Dmitriy Lyubimov
f5067abe85
changes per comments.
2013-07-27 23:08:00 -07:00
Karen Feng
077f2dad22
Fixed outdated bugs
2013-07-27 16:39:36 -07:00
Patrick Wendell
bcafb36c1e
Slight wording change
2013-07-27 16:03:50 -07:00
Patrick Wendell
8177165ac4
Log executor on finish
2013-07-27 16:02:06 -07:00
Patrick Wendell
c2223e6801
Improve catch scope and logging for client stop()
...
This does two things:
1. Catches the more general `TimeoutException`, since those can be thrown.
2. Logs at info level when a timeout is detected.
2013-07-27 16:02:06 -07:00
Karen Feng
5a93e3c58c
Cleaned up code based on pwendell's suggestions
2013-07-27 15:55:26 -07:00
Karen Feng
dcc4743a95
Moved val now to render
2013-07-27 12:52:53 -07:00
Karen Feng
1714693324
Current time called once with value now
2013-07-27 12:24:41 -07:00
Dmitriy Lyubimov
6a47cee721
style
2013-07-26 22:35:13 -07:00
Dmitriy Lyubimov
0c391feb73
Maximum task failures configurable
2013-07-26 22:34:43 -07:00
Karen Feng
bd4cc52e30
Made metrics Option instead of Some, fixed NullPointerException
2013-07-26 17:23:18 -07:00
Reynold Xin
cb366774c8
Merge pull request #738 from harsha2010/pruning
...
Fix bug in Partition Pruning.
2013-07-26 16:59:30 -07:00
harshars
392d7474fd
Code review
2013-07-26 15:23:15 -07:00
harshars
72cf7ec0e5
Indentation
2013-07-26 15:16:41 -07:00
harshars
822aac8f5a
Indentation
2013-07-26 15:10:32 -07:00
harshars
743fc4e7aa
Fix Bug in Partition Pruning, index of Pruned Partitions should inherit from parent
2013-07-26 14:35:17 -07:00
Karen Feng
3fbe9eaac0
Displys shuffle read/write only if exists, wraps if statements, trims old vals, grabs current time once
2013-07-26 11:51:38 -07:00
Karen Feng
22faeab261
Split Shuffle Activity overview column for read/write
2013-07-25 17:14:18 -07:00
Karen Feng
d4bbc8bd25
Shows totals for shuffle data and CPU time in Stage, homepage overviews including active time
2013-07-25 15:59:52 -07:00
Charles Reiss
a6de90c927
For standalone mode, get JAVA_HOME, SPARK_JAVA_OPTS, SPARK_LIBRARY_PATH from application env, not worker env
2013-07-25 12:42:30 -07:00
ryanlecompte
e56aa75de0
fix wrapping
2013-07-24 22:08:09 -07:00
ryanlecompte
8e0939f5a9
refactor Kryo serializer support to use chill/chill-java
2013-07-24 20:43:57 -07:00
Karen Feng
57009eef90
Fixed consistency of "success" status string
2013-07-24 13:43:09 -07:00
Karen Feng
4280e1768d
Removed finished status for task info, changed name of success case
2013-07-24 12:48:48 -07:00
Karen Feng
bd3931c874
Changed ifs with returns to if/else
2013-07-24 11:27:17 -07:00
Karen Feng
93c6015f82
Shows task status and running tasks on Stage Page: fixes SPARK-804 and 811
2013-07-24 10:53:02 -07:00
jerryshao
31ec72b243
Code refactor according to comments
2013-07-24 14:57:47 +08:00
jerryshao
8d1ef7f2df
Code style changes
2013-07-24 14:57:47 +08:00
Andrew xia
05637de842
Change class xxxInstrumentation to class xxxSource
2013-07-24 14:57:47 +08:00
Andrew xia
ed1a3bc206
continue to refactor code style and functions
2013-07-24 14:57:47 +08:00
jerryshao
5730193e0c
Fix some typos
2013-07-24 14:57:47 +08:00
jerryshao
a79f6077f0
Add Maven metrics library dependency and code changes
2013-07-24 14:57:47 +08:00
jerryshao
1daff54b2e
Change Executor MetricsSystem initialize code to SparkEnv
2013-07-24 14:57:47 +08:00
Andrew xia
5f8802c1fb
Register and init metricsSystem in SparkContext
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/SparkEnv.scala
2013-07-24 14:57:47 +08:00
Andrew xia
7d2eada451
Add metrics source of DAGScheduler and blockManager
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/SparkEnv.scala
2013-07-24 14:57:47 +08:00
jerryshao
e9ac88754d
Remove twice add Source bug and code clean
2013-07-24 14:57:47 +08:00
jerryshao
5ce5dc9fcd
Add default properties to deal with no configure file situation
2013-07-24 14:57:47 +08:00
jerryshao
871bc1687e
Add Executor instrumentation
2013-07-24 14:57:46 +08:00
jerryshao
7fb574bf66
Code clean and remarshal
2013-07-24 14:57:46 +08:00
Andrew xia
4d6dd67fa1
refactor metrics system
...
1.change source abstract class to support MetricRegistry
2.change master/work/jvm source class
2013-07-24 14:57:46 +08:00
jerryshao
03f9871116
MetricsSystem refactor
2013-07-24 14:57:46 +08:00
jerryshao
c3daad3f65
Update metric source support for instrumentation
2013-07-24 14:57:46 +08:00
jerryshao
9dec8c73e6
Add Master and Worker instrumentation support
2013-07-24 14:57:46 +08:00
jerryshao
503acd3a37
Build metrics system framwork
2013-07-24 14:57:46 +08:00
Matei Zaharia
b011329040
Merge pull request #727 from rxin/scheduler
...
Scheduler code style cleanup.
2013-07-23 22:50:09 -07:00
Matei Zaharia
876125b997
Merge pull request #726 from rxin/spark-826
...
SPARK-829: scheduler shouldn't hang if a task contains unserializable objects in its closure
2013-07-23 22:28:21 -07:00
Reynold Xin
3dae1df66f
Moved non-serializable closure catching exception from submitStage to submitMissingTasks
2013-07-23 20:29:07 -07:00
Reynold Xin
d33b8a2a0f
Added comments on task closure serialization.
2013-07-23 20:28:39 -07:00
Reynold Xin
85ab8114bc
Moved non-serializable closure catching exception from submitStage to submitMissingTasks
2013-07-23 20:25:58 -07:00
Matei Zaharia
6a31b7191d
Small bug fix
2013-07-23 16:20:24 -07:00
Matei Zaharia
2f1736c396
Merge pull request #725 from karenfeng/task-start
...
Creates task start events
2013-07-23 15:53:30 -07:00
Karen Feng
abc78cd331
Modifies instead of copies HashSets, fixes comment style
2013-07-23 15:47:16 -07:00
Karen Feng
383684daaa
Replaces Seq with HashSet, removes redundant import
2013-07-23 15:33:27 -07:00
Reynold Xin
f2422d4f29
SPARK-829: scheduler shouldn't hang if a task contains unserializable objects in its closure.
2013-07-23 15:30:20 -07:00
Reynold Xin
5ed38b4d1d
Scheduler code style cleanup.
2013-07-23 15:28:59 -07:00
Reynold Xin
101b8cc78a
SPARK-829: scheduler shouldn't hang if a task contains unserializable objects in its closure.
2013-07-23 15:28:20 -07:00
Karen Feng
9f2dbb2a7c
Adds/removes active tasks only once
2013-07-23 15:10:09 -07:00
Karen Feng
0200801a55
Tracks task start events and shows number of active tasks on Executor UI
2013-07-23 13:35:43 -07:00
Matei Zaharia
f369e0e51b
Merge pull request #720 from ooyala/2013-07/persistent-rdds-api
...
Add a public method getCachedRdds to SparkContext
2013-07-23 13:22:27 -07:00
Evan Chan
4830e22562
Rename method per rxin feedback
2013-07-23 09:50:13 -07:00
Evan Chan
2c2bfbe294
Add toMap method to TimeStampedHashMap and use it
2013-07-23 01:36:44 -07:00
Matei Zaharia
401aac8b18
Merge pull request #719 from karenfeng/ui-808
...
Creates Executors tab for Jobs UI
2013-07-22 16:57:16 -07:00
Karen Feng
872c97ad82
Split task columns, memory columns sort by numeric value
2013-07-22 16:54:37 -07:00
Karen Feng
2eea974795
Executors UI now calls executor ID from TaskInfo instead of TaskMetrics
2013-07-22 15:15:54 -07:00
Karen Feng
85c4d7bf3b
Shows number of complete/total/failed tasks (bug: failed tasks assigned to null executor)
2013-07-22 14:35:47 -07:00
Josh Rosen
f649dabb4a
Fix bug: DoubleRDDFunctions.sampleStdev() computed non-sample stdev().
...
Update JavaDoubleRDD to add new methods and docs.
Fixes SPARK-825.
2013-07-22 13:21:48 -07:00
Karen Feng
8901f379c9
Fixed memory used/remaining/total bug
2013-07-22 09:58:03 -07:00
Karen Feng
636b19f833
Merge branch 'master' of https://github.com/mesos/spark into ui-808
2013-07-22 09:53:26 -07:00
Evan Chan
0337d88321
Add a public method getCachedRdds to SparkContext
2013-07-21 18:26:14 -07:00
Karen Feng
865dc63bac
Changed table format for executors
2013-07-19 15:57:01 -07:00