Alexander Pivovarov
a30866438b
SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD
2013-08-05 21:48:43 -07:00
Matei Zaharia
8b277892c9
Merge pull request #774 from pwendell/job-description
...
Show user-defined job name in UI
2013-08-05 19:14:52 -07:00
Christopher Nguyen
b1bbbe699c
[HOTFIX] Mark lastSetSparkEnv @volatile in case it gets HotSpot-cached
...
On branch adatao-global-SparkEnv
Changes to be committed:
modified: core/src/main/scala/spark/SparkEnv.scala
2013-08-05 17:22:27 -07:00
Mark Hamstra
35d8f5ee52
Moved handling of timed out workers within the Master actor
2013-08-05 13:13:56 -07:00
Mark Hamstra
37ccf9301a
milliseconds -> seconds in timeOutDeadWorkers logging
2013-08-05 13:13:56 -07:00
Mark Hamstra
cdd1af562e
Timeout zombie workers
2013-08-05 13:13:56 -07:00
Mikhail Bautin
e8bec8365f
Only reduce the number of cores once when removing an executor
2013-08-05 13:13:56 -07:00
Karen Feng
95025afdec
Made most small fixes for SPARK-849 except for table sort, task progress overlay
2013-08-05 13:04:56 -07:00
Bill Zhao
87134b3648
SPARK-850: give better console message
2013-08-05 11:55:35 -07:00
Christopher Nguyen
39e4fda76f
[HOTFIX] Extend thread safety for SparkEnv.get()
...
A ThreadLocal SparkEnv.env is facing various situations leading to
NullPointerExceptions, where SparkEnv.env set in one thread is not
gettable in another thread, but often assumed to be available.
See, e.g., https://groups.google.com/forum/#!topic/spark-developers/GLx8yunSj0A
This hotfixes SparkEnv.env to return either (a) the ThreadLocal
value if non-null, or (b) the previously set value in any thread.
This approach preserves SparkEnv.set() thread safety needed by
RDD.compute() and possibly other places. A refactoring that
parameterizes SparkEnv should be addressed subsequently.
On branch adatao-global-SparkEnv
Changes to be committed:
modified: core/src/main/scala/spark/SparkEnv.scala
2013-08-05 02:09:54 -07:00
Patrick Wendell
f3660d5ab8
Make output formatting consistent between bash/scala
2013-08-03 21:30:15 -07:00
Patrick Wendell
ad94fbb322
Log the launch command for Spark executors
2013-08-03 09:19:46 -07:00
Matei Zaharia
22abbc10d6
Merge pull request #772 from karenfeng/ui-843
...
Show app duration
2013-08-02 16:37:59 -07:00
Patrick Wendell
5b3784a79c
Show user-defined job name in UI
2013-08-02 15:47:41 -07:00
Karen Feng
b3ae5b25d5
Shows time the app has been running
2013-08-02 13:25:14 -07:00
Patrick Wendell
9d7dfd2d5a
Merge pull request #743 from pwendell/app-metrics
...
Add application metrics to standalone master
2013-08-01 17:41:58 -07:00
Patrick Wendell
f1d2ad550e
under_scores --> camelCase for config options
2013-08-01 15:26:26 -07:00
Patrick Wendell
12d9c82c9b
Small style fix
2013-08-01 15:25:52 -07:00
Patrick Wendell
37bc64a205
Adding application-level metrics.
...
This adds metrics for applications in the deploy Master.
2013-08-01 15:25:52 -07:00
Karen Feng
73692f3cb9
Unify, reduce body font size
2013-08-01 15:10:30 -07:00
Patrick Wendell
87fd321a5a
Minor refactoring and code cleanup
2013-08-01 15:02:31 -07:00
Patrick Wendell
b10199413a
Slight refactoring to SparkContext functions
2013-08-01 15:00:42 -07:00
Patrick Wendell
cfcd77b5da
Increasing inter job arrival
2013-08-01 15:00:42 -07:00
Patrick Wendell
5faac7f4f3
Minor style fixes
2013-08-01 15:00:42 -07:00
Patrick Wendell
5e7b38fbb3
Merge pull request #695 from xiajunluan/pool_ui
...
Enhance job ui in spark ui system with adding pool information
2013-08-01 14:59:33 -07:00
Karen Feng
47600e9579
Removed hr margin
2013-08-01 14:57:04 -07:00
Karen Feng
e648a62fc8
Inserted needed line break for log paging
2013-08-01 14:46:19 -07:00
Karen Feng
686d6266c4
Use nav pills instead of default
2013-08-01 14:41:49 -07:00
Karen Feng
86d372d17f
Removed line breaks
2013-08-01 14:37:21 -07:00
Karen Feng
99803d88b9
Reduced all header sizes
2013-08-01 14:18:33 -07:00
Karen Feng
d216d687ef
Reduced size of table text to compact
2013-08-01 13:27:23 -07:00
Karen Feng
5dae283996
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
2013-08-01 11:28:28 -07:00
Matei Zaharia
0a96493ac6
Merge pull request #760 from karenfeng/heading-update
...
Clean up web UI page headers
2013-08-01 11:27:17 -07:00
Patrick Wendell
9177bea2b4
Removing extra imports
2013-08-01 10:42:50 -07:00
Patrick Wendell
3e4d5e5f8b
Merge branch 'master' into master-json
...
Conflicts:
core/src/main/scala/spark/deploy/master/ui/IndexPage.scala
2013-08-01 10:42:07 -07:00
Patrick Wendell
ffc034e4fb
Import cleanup
2013-08-01 10:39:56 -07:00
Andrew xia
d58502a156
fix bug of spark "SubmitStage" listener as unit test error
2013-08-01 23:21:41 +08:00
Andrew xia
3b5a11e765
change function name "setName" to "setProperties" as "setName" is also member of Thread class
2013-08-01 19:37:15 +08:00
Dmitriy Lyubimov
d29ee3689b
Merge fixes merge commit hasn't picked
2013-08-01 00:21:26 -07:00
Dmitriy Lyubimov
cb6be5bd7e
Merge remote-tracking branch 'mesos/master' into SPARK-826
...
Conflicts:
core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
core/src/test/scala/spark/KryoSerializerSuite.scala
2013-07-31 22:09:22 -07:00
Dmitriy Lyubimov
28f1550f01
More elegant rewrite of the same.
2013-07-31 21:41:00 -07:00
Dmitriy Lyubimov
7c52ecc6a4
(1) added reduce test case.
...
(2) added nested streaming in ParallelCollectionRDD
(3) added kryo with fold test which still doesn't work
2013-07-31 19:27:30 -07:00
Matei Zaharia
3097d75d6f
Merge remote-tracking branch 'dlyubimov/SPARK-827'
...
Conflicts:
docs/configuration.md
2013-07-31 18:36:43 -07:00
Karen Feng
7c9c5ef6c6
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
2013-07-31 16:39:26 -07:00
Karen Feng
02cde8efdf
Replaces theme with Bootswatch Spacelab theme
2013-07-31 16:34:07 -07:00
Karen Feng
09cd67bf98
Changed bootstrap colors, fixed logpaging buttons
2013-07-31 16:18:53 -07:00
Matei Zaharia
39c75f3033
Merge pull request #757 from BlackNiuza/result_task_generation
...
Bug fix: SPARK-837
2013-07-31 15:52:36 -07:00
Matei Zaharia
14bf2fe039
Merge pull request #749 from benh/spark-executor-uri
...
Added property 'spark.executor.uri' for launching on Mesos.
2013-07-31 14:18:16 -07:00
Benjamin Hindman
4692ea4892
Used 'uri.split('/').last' instead of 'new File(uri).getName()'.
2013-07-31 12:29:44 -07:00
Karen Feng
c453967f9a
Reduced size of heading
2013-07-31 11:57:50 -07:00
Matei Zaharia
a386ced2c6
Merge pull request #754 from rxin/compression
...
Compression codec change
2013-07-31 11:22:50 -07:00
Karen Feng
49e6344142
Removed master URL from job UI, reduced heading size of basic spark pages
2013-07-31 11:17:59 -07:00
Reynold Xin
c61843a69f
Changed other LZF uses to use the compression codec interface.
2013-07-31 10:32:13 -07:00
Patrick Wendell
89da9d94b3
Add JSON path to master index page
2013-07-31 09:47:53 -07:00
BlackNiuza
9a815de4bf
write and read generation in ResultTask
2013-08-01 00:36:47 +08:00
Roman Tkalenko
0c6553714a
Refactored Vector.apply(length, initializer) replacing excessive code with library method
...
(also removed unused variable ```ans``` as minor change)
2013-07-31 19:05:46 +03:00
Matei Zaharia
12553e5c55
Simplified nonNegativeMod to match previous version
2013-07-31 08:50:28 -07:00
Matei Zaharia
d4556f4207
Merge pull request #751 from cdshines/master
...
Cleaned Partitioner & PythonPartitioner source by taking out non-related logic to Utils
2013-07-31 08:48:14 -07:00
Andrew xia
5670c96f29
Merge branch 'master' into Pool_UI
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/scheduler/DAGScheduler.scala
core/src/main/scala/spark/scheduler/SparkListener.scala
core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
core/src/main/scala/spark/ui/jobs/IndexPage.scala
core/src/main/scala/spark/ui/jobs/JobProgressUI.scala
2013-07-31 19:36:36 +08:00
cdshines
fefb03cbd7
Eliminated code duplication, refactored to pattern-matching style Partitioner and PythonPartitioner
2013-07-31 13:19:42 +03:00
Dmitriy Lyubimov
96664431cb
IDEA flipped JavaSerialized import at some point to a wrong class.
2013-07-30 23:10:09 -07:00
Dmitriy Lyubimov
c219fc94fd
Minor, style
2013-07-30 22:08:39 -07:00
Dmitriy Lyubimov
f4b4b8836e
reverting back to one-by-one serialization for parallelize()
2013-07-30 19:00:58 -07:00
jerryshao
bf9318091a
Add Apache license header to metrics system
2013-07-31 09:42:16 +08:00
Reynold Xin
98024eadc3
Renamed compressionOutputStream and compressionInputStream to compressedOutputStream and compressedInputStream.
2013-07-30 18:28:46 -07:00
Dmitriy Lyubimov
abada94ebf
removing default constructor (not Externalizable any more)
2013-07-30 18:04:02 -07:00
Dmitriy Lyubimov
943c6590c9
realiging "extends" back manually
2013-07-30 18:01:35 -07:00
Dmitriy Lyubimov
ca33b12e98
resetting wrap and continuation indent = 4
2013-07-30 17:51:44 -07:00
Reynold Xin
dae12fef9e
Updated the configuration option for Snappy block size to be consistent with the documentation.
2013-07-30 17:49:31 -07:00
Dmitriy Lyubimov
984b56155a
changing approaches for parallelize(): java serialization needs to avoid writing headers!
2013-07-30 17:36:59 -07:00
Reynold Xin
311aae76a2
Added Snappy dependency to Maven build files.
2013-07-30 17:25:42 -07:00
Reynold Xin
56774b176e
Added unit test for compression codecs.
2013-07-30 17:12:33 -07:00
Reynold Xin
ad7e9d0d64
CompressionCodec cleanup. Moved it to spark.io package.
2013-07-30 17:11:54 -07:00
Dmitriy Lyubimov
ef9529a943
refactoring using writeByteBuffer() from Utils.
2013-07-30 16:24:23 -07:00
Dmitriy Lyubimov
43394b9a6d
fixing formatting
2013-07-30 16:13:41 -07:00
Dmitriy Lyubimov
13a9d66645
adding ===
2013-07-30 16:10:55 -07:00
Reynold Xin
368c58eac5
Merge branch 'lazy_file_open' of github.com:lyogavin/spark into compression
...
Conflicts:
project/SparkBuild.scala
2013-07-30 16:04:18 -07:00
Patrick Wendell
e87de037d6
Merge pull request #744 from karenfeng/bootstrap-update
...
Use Bootstrap progress bars in web UI
2013-07-30 15:00:08 -07:00
Karen Feng
26144c400f
Fixed wrap style
2013-07-30 12:40:41 -07:00
Karen Feng
218d7c4ed8
Fixed style, lowered height of progress bars
2013-07-30 12:39:17 -07:00
Karen Feng
f1cab31b73
Removed intermediate set for activeTasks, removed progress bar margin
2013-07-30 11:06:47 -07:00
Dmitriy Lyubimov
1bca91633e
+ bug fixes;
...
test added
Conflicts:
core/src/test/scala/spark/KryoSerializerSuite.scala
2013-07-30 11:04:11 -07:00
Benjamin Hindman
f6f46455eb
Added property 'spark.executor.uri' for launching on Mesos without
...
requiring Spark to be installed. Using 'make_distribution.sh' a user
can put a Spark distribution at a URI supported by Mesos (e.g.,
'hdfs://...') and then set that when launching their job. Also added
SPARK_EXECUTOR_URI for the REPL.
2013-07-29 23:32:52 -07:00
Josh Rosen
49be084ed3
Use File.pathSeparator instead of hardcoding ':'.
2013-07-29 22:08:57 -07:00
Josh Rosen
b95732632b
Do not inherit master's PYTHONPATH on workers.
...
This fixes SPARK-832, an issue where PySpark
would not work when the master and workers used
different SPARK_HOME paths.
This change may potentially break code that relied
on the master's PYTHONPATH being used on workers.
To have custom PYTHONPATH additions used on the
workers, users should set a custom PYTHONPATH in
spark-env.sh rather than setting it in the shell.
2013-07-29 22:08:57 -07:00
Andrew xia
5406013997
refactor codes less than 100 character per line
2013-07-30 11:41:38 +08:00
Andrew xia
614ee16cc4
refactor job ui with pool information
2013-07-30 10:57:26 +08:00
Dmitriy Lyubimov
8e5cd041bb
initial externalization of ParallelCollectionRDD's split
2013-07-29 19:02:53 -07:00
Reynold Xin
81720e13fc
Moved all StandaloneClusterMessage's into StandaloneClusterMessages object.
2013-07-29 17:53:01 -07:00
Reynold Xin
23b5da14ed
Moved block manager messages into BlockManagerMessages object.
2013-07-29 17:42:05 -07:00
Reynold Xin
105f4d22e9
Removed Cache and SoftReferenceCache since they are no longer used.
2013-07-29 17:30:38 -07:00
Reynold Xin
17e62113d4
Moved DeployMessage's into its own DeployMessages object.
...
Also renamed MasterState to MasterStateResponse and WorkerState to WorkerStateResponse for clarity.
2013-07-29 17:14:44 -07:00
Karen Feng
87b821dc39
Fixed continuity of executorToTasksActive, changed color of progress bars
2013-07-29 16:50:51 -07:00
Karen Feng
c7b2788948
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
...
Conflicts:
core/src/main/scala/spark/ui/jobs/IndexPage.scala
2013-07-29 16:36:07 -07:00
Patrick Wendell
c99b674405
Merge pull request #735 from karenfeng/ui-807
...
Totals for shuffle data and CPU time
2013-07-29 16:32:55 -07:00
Karen Feng
2d6da9195a
Alphabetized imports
2013-07-29 15:50:52 -07:00
Karen Feng
478a2886d9
Added started tasks to progress bar
2013-07-29 14:51:07 -07:00
Karen Feng
e04a37a332
Merge branch 'master' of https://github.com/mesos/spark into bootstrap-update
...
cially if it merges an updated upstream into a topic branch.
2013-07-29 14:32:48 -07:00
Reynold Xin
fe7298b587
Merge pull request #741 from pwendell/usability
...
Fix two small usability issues
2013-07-29 14:01:00 -07:00
Karen Feng
43a2cc15c0
Use Bootstrap progress bars in web UI
2013-07-29 13:37:24 -07:00