Karen Feng
3955711250
Added field to master UI with link to job UI
2013-07-15 15:47:21 -07:00
Karen Feng
0d78b6d9cd
Links to job UI from standalone deploy cluster web UI: fixes SPARK-802
2013-07-15 13:47:38 -07:00
Karen Feng
b2aaa1199e
Adds app name in HTML page titles on job web UI: fixes SPARK-806
2013-07-15 11:44:42 -07:00
Matei Zaharia
d47c16f78d
Add an option to disable reference tracking in Kryo
2013-07-15 01:55:54 +00:00
Matei Zaharia
10c05937bd
Merge pull request #699 from pwendell/ui-env
...
Add `Environment` tab to SparkUI.
2013-07-14 11:45:18 -07:00
Patrick Wendell
4883586838
Responding to Matei's review
2013-07-14 10:37:26 -07:00
Matei Zaharia
b91a218cea
Cosmetic fixes to web UI
2013-07-14 07:31:33 +00:00
Matei Zaharia
a44a7b1238
Determine Spark core classes better in getCallSite
2013-07-14 07:23:09 +00:00
root
e271fde10b
Fixed a delay scheduling bug in the YARN branch, found by Patrick
2013-07-14 06:24:29 +00:00
Patrick Wendell
ddb97f0fdf
Add Environment
tab to SparkUI.
...
This adds a tab which displays system property and classpath information. This
can be useful in debugging various types of issues such as:
1. Extra/incorrect Hadoop jars being included in the classpath
2. Spark launching with a different JRE version than intended
3. Spark system properties not being set to intended values
4. User added jars that conflict with Spark jars
2013-07-13 16:14:40 -07:00
Matei Zaharia
77c69ae5a0
Merge pull request #697 from pwendell/block-locations
...
Show block locations in Web UI.
2013-07-12 23:05:21 -07:00
Matei Zaharia
5a7835c152
Merge pull request #691 from karenfeng/logpaging
...
Create log pages
2013-07-12 20:28:21 -07:00
Matei Zaharia
71ccca0cc1
Merge pull request #696 from woggle/executor-env
...
Pass executor env vars (e.g. SPARK_CLASSPATH) to compute-classpath.sh
2013-07-12 20:25:06 -07:00
Matei Zaharia
90fc3f30cd
Merge pull request #692 from Reinvigorate/takeOrdered
...
adding takeOrdered() to RDD
2013-07-12 20:23:36 -07:00
Patrick Wendell
08150f19ab
Minor style fix
2013-07-12 19:32:35 -07:00
Patrick Wendell
6855338e14
Show block locations in Web UI.
...
This fixes SPARK-769. Support is added for enumerating the locations of blocks
in the UI. There is also some minor cleanup in StorageUtils.
2013-07-12 19:30:32 -07:00
Charles Reiss
531a7e5574
Pass executor env vars (e.g. SPARK_CLASSPATH) to compute-classpath.
2013-07-12 12:58:25 -07:00
seanm
a1662326e9
comment adjustment to takeOrdered
2013-07-12 08:38:19 -07:00
Andrew xia
2080e25006
Enhance job ui in spark ui system with adding pool information
2013-07-12 14:25:18 +08:00
seanm
a2c915fba8
giving order to top and making tests more clear
2013-07-11 18:55:00 -07:00
Karen Feng
5c67ca0278
Remove "Bytes" in lieu of String notation
2013-07-11 17:31:59 -07:00
Karen Feng
6d054487bf
Replace default buffer value to 100 GB, changed buttons to use String notation, removed default buffer parameter in UI URLs
2013-07-11 17:12:17 -07:00
Karen Feng
a32784109d
Fixed links for "Back to Master"
2013-07-11 16:57:55 -07:00
Karen Feng
ece2388585
Removed logPageLength from logPage
2013-07-11 16:35:56 -07:00
Karen Feng
9ed036ccdb
Replaced logPageLength with byteLength to prevent buffer shrink bug
2013-07-11 16:33:53 -07:00
Karen Feng
fdc226a14c
Clarified start and end byte variable names
2013-07-11 15:36:43 -07:00
Karen Feng
5d5dbc39f6
getByteRange moved to WorkerWebUI, takes converted parameters, returns only start/end offset
2013-07-11 15:22:45 -07:00
Karen Feng
15fd11d657
Removed redundant calls to request by logPage
2013-07-11 15:01:50 -07:00
Karen Feng
11872888ca
Created getByteRange function for logs and log pages, removed lastNBytes function
2013-07-11 14:56:37 -07:00
Matei Zaharia
018d04c64e
Merge pull request #684 from woggle/mesos-classloader
...
Explicitly set class loader for MesosSchedulerDriver callbacks.
2013-07-11 12:48:37 -07:00
Karen Feng
e3a3fcf61b
Scrollbar on log pages appear automatically
2013-07-11 12:16:38 -07:00
Karen Feng
044d4577ec
Fixed capitalization of log page
2013-07-11 12:02:15 -07:00
Karen Feng
0ecc33f0c8
Added byte range, page title with log name, previous/next bytes buttons, initialization to end of log, large default buffer, buggy back to master link
2013-07-11 11:25:58 -07:00
Karen Feng
74bd3fc680
Added byte range on log pages
2013-07-10 15:44:28 -07:00
Karen Feng
24196c91f0
Changed buffer to 10,000 bytes, created scrollbar for fixed-height log
2013-07-10 15:27:52 -07:00
Karen Feng
f5f3b272f8
Fixed mixup of start/end, moved more import files
2013-07-10 14:52:29 -07:00
Karen Feng
0d4580360b
Fixed docstring of offsetBytes to match params and wrapped for 100+ character lines
2013-07-10 13:24:26 -07:00
Karen Feng
04263e4d46
Made some minor style changes
2013-07-10 13:15:42 -07:00
Karen Feng
cfb6447ac4
Fixed for nonexistent bytes, added unit tests, changed stdout-page to stdout
2013-07-10 11:47:57 -07:00
seanm
ee4ce2fc51
adding takeOrdered to java API
2013-07-10 10:46:04 -07:00
seanm
24705d0f46
adding takeOrdered() to RDD
2013-07-10 10:33:11 -07:00
Karen Feng
620a6974c6
Allows for larger files, refactors lastNBytes, removes old Log column, fixes imports, uses map
2013-07-10 10:20:53 -07:00
Karen Feng
b6072b58bf
Fixes style, makes "std__-page" consistent, reads only parts of files
2013-07-09 17:25:10 -07:00
Karen Feng
13fc6f248c
Clean commit of log paging
2013-07-09 14:17:15 -07:00
Charles Reiss
e47253e0cc
Reset ClassLoader in MesosSchedulerBackend, too. (per review comments).
...
Also set ClassLoader for all mesos callbacks, not just statusUpdate,
registered.
2013-07-09 01:23:23 -07:00
Charles Reiss
8c1d1c98e0
Explicitly set class loader for MesosSchedulerDriver callbacks.
2013-07-08 12:25:46 -07:00
Shivaram Venkataraman
4af0d63cb1
Remove akka LogLevel fix as we no longer use spray
2013-07-07 10:42:43 -07:00
Shivaram Venkataraman
a948f06725
Suppress log messages in sbt test with two changes:
...
1. Set akka log level to ERROR before shutting down the actorSystem.
This avoids akka log messages (like Spray) from falling back to INFO
on the Stdout logger
2. Initialize netty to use SLF4J in LocalSparkContext. This ensures that
stack trace thrown during shutdown is handled by SLF4J instead of stdout
2013-07-07 04:09:08 -07:00
Patrick Wendell
32b9d21a97
Fix occasional failure in UI listener.
...
If a task fails before the metrics are initialized, it remains possible
that the metrics field will be `None`. This patch accounts for that possbility
by keeping metrics as an `Option` at all times.
2013-07-06 16:40:02 -07:00
Matei Zaharia
1ffadb2d9e
Merge remote-tracking branch 'pwendell/ui-updates'
...
Conflicts:
core/src/main/scala/spark/scheduler/DAGScheduler.scala
core/src/main/scala/spark/util/AkkaUtils.scala
pom.xml
2013-07-06 15:51:41 -07:00
Matei Zaharia
94871e4703
Merge pull request #655 from tgravescs/master
...
Add support for running Spark on Yarn on a secure Hadoop Cluster
2013-07-06 15:26:19 -07:00
Matei Zaharia
3f918b33f8
Merge pull request #672 from holdenk/master
...
s/ActorSystemImpl/ExtendedActorSystem/ as ActorSystemImpl results in a warning
2013-07-06 12:45:18 -07:00
Matei Zaharia
7ba7fa110b
Merge pull request #674 from liancheng/master
...
Bug fix: SPARK-789
2013-07-06 11:45:08 -07:00
BlackNiuza
44a2440039
Remove active job from idToActiveJob when job finished or aborted
2013-07-07 01:33:09 +08:00
Patrick Wendell
37abe84212
Tracking some task metrics even during failures.
2013-07-06 09:19:59 -07:00
Patrick Wendell
84b7fc54e6
Enforcing correct sort order for formatted strings
2013-07-05 17:21:08 -07:00
Matei Zaharia
652ea0f1d8
Allow RDD.takeSample to give samples bigger than the RDD
...
Before, when withReplacement was set to true, we would not get a sample
bigger than the RDD's count().
Conflicts:
core/src/main/scala/spark/RDD.scala
core/src/test/scala/spark/RDDSuite.scala
2013-07-05 11:15:13 -07:00
Matei Zaharia
6586c5e28b
Added a SparkContext accessor to RDD
2013-07-05 11:13:46 -07:00
jerryshao
e4ff544a8d
Clean StageToInfos periodically when spark.cleaner.ttl is enabled
2013-07-05 10:34:45 +08:00
Lian Cheng
c0c3155c3c
Bug fix: SPARK-789
...
https://spark-project.atlassian.net/browse/SPARK-789
2013-07-05 00:54:10 +08:00
Holden Karau
0f06d6217d
s/ActorSystemImpl/ExtendedActorSystem/ as ActorSystemImpl results in a warning
2013-07-04 01:05:39 -07:00
Gavin Li
94238aae57
fix dependencies
2013-07-03 18:08:38 +00:00
Gavin Li
96130c30d9
add compression codec trait and snappy compression
2013-07-03 05:49:04 +00:00
Y.CORP.YAHOO.COM\tgraves
923cf92900
Rework from pull request. Removed --user option from Spark on Yarn Client, made the user of JAVA_HOME environment
...
variable conditional on if its set, and created addCredentials in each of the SparkHadoopUtil classes
to only add the credentials when the profile is hadoop2-yarn.
2013-07-02 21:18:59 -05:00
Patrick Wendell
39e2325675
Removing dead code
2013-07-02 16:28:40 -07:00
Patrick Wendell
8ca1cc1786
Adding truncation for log files
2013-07-02 16:10:50 -07:00
Patrick Wendell
9a42d04efa
Throw exception for missing resource
2013-07-01 14:43:13 -07:00
Patrick Wendell
1025d7d1ef
Package refactoring
2013-07-01 14:40:53 -07:00
Patrick Wendell
30b9034241
Fixing bug where logs aren't shown
2013-07-01 13:48:01 -07:00
Patrick Wendell
8688689387
Various formatting changes
2013-07-01 13:40:12 -07:00
Patrick Wendell
735c951a09
Adding test script
2013-07-01 09:33:22 -07:00
Patrick Wendell
5de326db7d
Print exception message
2013-07-01 09:19:45 -07:00
root
ec31e68d5d
Fixed PySpark perf regression by not using socket.makefile(), and improved
...
debuggability by letting "print" statements show up in the executor's stderr
Conflicts:
core/src/main/scala/spark/api/python/PythonRDD.scala
2013-07-01 06:26:31 +00:00
root
3296d132b6
Fix performance bug with new Python code not using buffered streams
2013-07-01 06:25:43 +00:00
Matei Zaharia
03d0b858c8
Made use of spark.executor.memory setting consistent and documented it
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
2013-06-30 15:46:46 -07:00
Patrick Wendell
e721ff7e5a
Allowing details for failed stages
2013-06-29 11:26:30 -07:00
Patrick Wendell
473961d82e
Styling for progress bar
2013-06-29 08:38:04 -07:00
Patrick Wendell
249f0e54ba
Minor changes from Matei's review
2013-06-28 13:25:26 -07:00
Patrick Wendell
62c2c6b856
Forcing Jetty to run as daemon
2013-06-27 21:47:22 -07:00
Patrick Wendell
a55190d314
Adding better tabs for UI headers.
2013-06-27 19:14:51 -07:00
Patrick Wendell
362d996c81
Handful of changes based on matei's review
...
- Avoid exception when no tasks have finished for a stage
- Adding DOCTYPE so css renders properly
- Adding progress slider
2013-06-27 19:14:28 -07:00
Patrick Wendell
92a4c2a5f6
Fixing bug in local scheduler time recording
2013-06-27 12:33:06 -07:00
Stephen Haberman
d7011632d1
Wrap lines.
2013-06-26 12:35:57 -05:00
Patrick Wendell
ee692482a6
One more private class
2013-06-26 09:07:32 -07:00
Patrick Wendell
a59c15a37e
Adding config option for retained stages
2013-06-26 08:54:57 -07:00
Patrick Wendell
274193664a
Bumping timeouts
2013-06-26 08:51:28 -07:00
Patrick Wendell
b14ad509ba
Moving static ui package
2013-06-26 08:46:51 -07:00
Patrick Wendell
2cbaa0734b
Making all new classes package private
2013-06-26 08:44:55 -07:00
Stephen Haberman
d11025dc6a
Be cute with Option and getenv.
2013-06-26 09:53:35 -05:00
Matei Zaharia
6c8d1b2ca6
Fix computation of classpath when we launch java directly
...
The previous version assumed that a CLASSPATH environment variable was
set by the "run" script when launching the process that starts the
ExecutorRunner, but unfortunately this is not true in tests. Instead, we
factor the classpath calculation into an extenral script and call that.
NOTE: This includes a Windows version but hasn't yet been tested there.
2013-06-25 18:21:00 -04:00
Matei Zaharia
15b00914c5
Some fixes to the launch-java-directly change:
...
- Split SPARK_JAVA_OPTS into multiple command-line arguments if it
contains spaces; this splitting follows quoting rules in bash
- Add the Scala JARs to the classpath if they're not in the CLASSPATH
variable because the ExecutorRunner is launched with "scala" (this can
happen when using local-cluster URLs in spark-shell)
2013-06-25 17:17:27 -04:00
Matei Zaharia
7e0191c6ea
Merge remote-tracking branch 'cgrothaus/SPARK-698'
...
Conflicts:
run
2013-06-25 15:47:40 -04:00
Patrick Wendell
d66bd6f885
Adding another unit test to Web UI suite
2013-06-24 17:12:55 -07:00
Patrick Wendell
f7389330c3
Allowing for requested port on construction
2013-06-24 16:51:52 -07:00
Patrick Wendell
42157027f2
A few bug fixes and a unit test
2013-06-24 16:25:05 -07:00
Patrick Wendell
a4248138b4
Minor style cleanup
2013-06-24 14:22:28 -07:00
Patrick Wendell
b5e6e8bcc8
Cleaning up some code for Job Progress
2013-06-24 14:13:24 -07:00
Patrick Wendell
93e8ed85aa
Work around for initalization issue
2013-06-24 13:11:18 -07:00
Patrick Wendell
f6e64b5cd6
Updating based on changes to JobLogger (and one small change to JobLogger)
2013-06-24 12:40:41 -07:00
Matei Zaharia
78ffe164b3
Clone the zero value for each key in foldByKey
...
The old version reused the object within each task, leading to
overwriting of the object when a mutable type is used, which is expected
to be common in fold.
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2013-06-23 10:26:53 -07:00
Matei Zaharia
0e0f9d3069
Fix search path for REPL class loader to really find added JARs
2013-06-22 17:44:04 -07:00
Matei Zaharia
3e61beff7b
Merge pull request #648 from shivaram/netty-dbg
...
Shuffle fixes and cleanup
2013-06-22 16:22:47 -07:00
Patrick Wendell
7e9f1ed0de
Some cleanup of styling
2013-06-22 10:31:37 -07:00
Patrick Wendell
3b7ebdeeb8
Handling entirely failed stages
2013-06-22 10:31:37 -07:00
Patrick Wendell
be6107ce44
Some tweaking with shared page header
2013-06-22 10:31:37 -07:00
Patrick Wendell
9a24d1a2d0
Using scala in XML imports
2013-06-22 10:31:37 -07:00
Patrick Wendell
f91e1c4822
Linking RDD information when available in stages
2013-06-22 10:31:37 -07:00
Patrick Wendell
a86bb459e2
Showing shuffle status and purging old stages
2013-06-22 10:31:37 -07:00
Patrick Wendell
3485e73376
Style cleanup
2013-06-22 10:31:37 -07:00
Patrick Wendell
dd696f3a3d
Some renaming and comments
2013-06-22 10:31:37 -07:00
Patrick Wendell
5c872e9ef5
Documentation and some refactoring
2013-06-22 10:31:37 -07:00
Patrick Wendell
17776323a6
More work on percentile data:
2013-06-22 10:31:37 -07:00
Patrick Wendell
dcf6a68177
Refactoring into different modules
2013-06-22 10:31:36 -07:00
Patrick Wendell
ce81c320ac
Adding helper function to make listing tables
2013-06-22 10:31:36 -07:00
Patrick Wendell
9fd5dc3ea9
Initial steps towards job progress UI
2013-06-22 10:31:36 -07:00
Patrick Wendell
bc4a811c57
Stash
2013-06-22 10:31:36 -07:00
Patrick Wendell
77c53f7868
Refactoring UI packages
2013-06-22 10:31:36 -07:00
Patrick Wendell
8b5c7e71c4
Import cleanup
2013-06-22 10:31:36 -07:00
Patrick Wendell
32a45d01b1
Removing twirl files
2013-06-22 10:31:36 -07:00
Patrick Wendell
4e1f202481
Removing dead code
2013-06-22 10:31:36 -07:00
Patrick Wendell
d6fde4ffe4
Some JSON cleanup
2013-06-22 10:31:36 -07:00
Patrick Wendell
91ec5a1a04
Changing JSON protocol and removing spray code
2013-06-22 10:31:36 -07:00
Patrick Wendell
fc94576ece
Adding worker version of UI
2013-06-22 10:31:36 -07:00
Patrick Wendell
ee73c09ac9
Some comments
2013-06-22 10:31:36 -07:00
Patrick Wendell
9161db5478
Cleaning up master web UI
2013-06-22 10:31:36 -07:00
Patrick Wendell
e55cf0245f
Adding WebUI file
2013-06-22 10:31:35 -07:00
Patrick Wendell
f85fd7a793
Commenting unfinished part
2013-06-22 10:31:35 -07:00
Patrick Wendell
2c36a514aa
Spray refactoring for master web UI
2013-06-22 10:31:35 -07:00
Patrick Wendell
7e6977b6c5
Fix in storage status page
2013-06-22 10:31:35 -07:00
Patrick Wendell
950f83535a
Adding deterministic port
2013-06-22 10:31:35 -07:00
Patrick Wendell
7cd70dc2c1
Minor cleanup
2013-06-22 10:31:35 -07:00
Patrick Wendell
e66f570194
Completely hacked version of block manager UI in jetty
2013-06-22 10:31:35 -07:00
Patrick Wendell
60fbf7e461
Partially working checkpoint
2013-06-22 10:31:35 -07:00
Matei Zaharia
1ef5d0d2c9
Merge pull request #644 from shimingfei/joblogger
...
add Joblogger to Spark (on new Spark code)
2013-06-22 09:35:57 -07:00
Jey Kottalam
1ba3c17303
use parens when calling method with side-effects
2013-06-21 12:14:16 -04:00
Jey Kottalam
edb18ca928
Rename PythonWorker to PythonWorkerFactory
2013-06-21 12:14:16 -04:00
Jey Kottalam
62c4781400
Add tests and fixes for Python daemon shutdown
2013-06-21 12:14:16 -04:00
Jey Kottalam
c79a6078c3
Prefork Python worker processes
2013-06-21 12:14:16 -04:00
Jey Kottalam
40afe0d2a5
Add Python timing instrumentation
2013-06-21 12:14:16 -04:00
Mingfei
2fc794a6c7
small modify in DAGScheduler
2013-06-21 18:21:35 +08:00
Mingfei
4b9862ac9c
small format modification
2013-06-21 17:55:32 +08:00
Mingfei
aa7aa587be
some format modification
2013-06-21 17:48:41 +08:00
Mingfei
5240795154
edit according to comments
2013-06-21 17:38:23 +08:00
Matei Zaharia
71030ba3eb
Merge pull request #654 from lyogavin/enhance_pipe
...
fix typo and coding style in #638
2013-06-19 15:21:03 -07:00
Thomas Graves
bad51c7cb4
upmerge with latest mesos/spark master and fix hbase compile with hadoop2-yarn profile
2013-06-19 14:39:13 -05:00
Thomas Graves
75d78c7ac9
Add support for Spark on Yarn on a secure Hadoop cluster
2013-06-19 11:18:42 -05:00
Matei Zaharia
7902baddc7
Update ASM to version 4.0
2013-06-19 13:34:30 +02:00
Gavin Li
0a2a9bce1e
fix typo and coding style
2013-06-18 21:30:13 +00:00
jerryshao
1e9269c3ee
reduce ZippedPartitionsRDD's getPreferredLocations complexity
2013-06-18 09:49:06 +08:00
Matei Zaharia
db42451a52
Merge pull request #643 from adatao/master
...
Bug fix: Zero-length partitions result in NaN for overall mean & variance
2013-06-17 15:26:36 -07:00
Matei Zaharia
e82a2ffcc9
Merge pull request #653 from rxin/logging
...
SPARK-781: Log the temp directory path when Spark says "Failed to create temp directory."
2013-06-17 15:13:15 -07:00
Matei Zaharia
ec193c7d89
Merge remote-tracking branch 'xiajunluan/xiajunluan'
...
Conflicts:
core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
2013-06-18 00:11:50 +02:00
Reynold Xin
be3c406edf
Fixed the typo pointed out by Matei.
2013-06-17 17:07:51 -04:00
Reynold Xin
1450296797
SPARK-781: Log the temp directory path when Spark says "Failed to create
...
temp directory".
2013-06-17 16:58:23 -04:00
Gavin Li
4508089fc3
refine comments and add sc.clean
2013-06-17 05:23:46 +00:00
Gavin Li
e6ae049283
Merge remote-tracking branch 'upstream1/master' into enhance_pipe
2013-06-16 22:53:39 +00:00
Gavin Li
fb6d733fa8
update according to comments
2013-06-16 22:32:55 +00:00
Matei Zaharia
f961aac8b2
Merge pull request #649 from ryanlecompte/master
...
Add top K method to RDD using a bounded priority queue
2013-06-15 00:53:41 -07:00
ryanlecompte
e8801d4490
use delegation for BoundedPriorityQueue, add Java API
2013-06-14 23:39:05 -07:00
Reynold Xin
2cc188fd54
SPARK-774: cogroup should also disable map side combine by default
2013-06-14 00:10:54 -07:00
Reynold Xin
6738178d0d
SPARK-772: groupByKey should disable map side combine.
2013-06-13 23:59:42 -07:00
ryanlecompte
93b3f5e535
drop unneeded ClassManifest implicit
2013-06-13 16:26:35 -07:00
ryanlecompte
44b8dbaede
use Iterator.single(elem) instead of Iterator(elem) for improved performance based on scaladocs
2013-06-13 16:23:15 -07:00
Shivaram Venkataraman
1d9f0df065
Fix some comments and style
2013-06-13 14:46:25 -07:00
Mingfei
967a6a699d
modify sparklister function interface according to comments
2013-06-13 14:36:07 +08:00
Shivaram Venkataraman
5da4287b1d
Merge branch 'netty-dbg' of github.com:shivaram/spark into netty-dbg
2013-06-12 16:38:37 -07:00
Shivaram Venkataraman
5e9a9317c5
Merge branch 'master' of git://github.com/mesos/spark into netty-dbg
2013-06-12 16:38:01 -07:00
ryanlecompte
db5bca08ff
add a new top K method to RDD using a bounded priority queue
2013-06-12 10:54:16 -07:00
Andrew xia
190ec61799
change code style and debug info
2013-06-10 15:27:02 +08:00
Patrick Wendell
ef14dc2e77
Adding Java-API version of compression codec
2013-06-09 18:09:46 -07:00
Patrick Wendell
df592192e7
Monads FTW
2013-06-09 18:09:24 -07:00
Patrick Wendell
d1bbcebae5
Adding compression to Hadoop save functions
2013-06-09 11:39:35 -07:00
Mingfei
ade822011d
not check return value of eventQueue.take
2013-06-08 16:26:45 +08:00
Mingfei
4fd86e0e10
delete test code for joblogger in SparkContext
2013-06-08 15:45:47 +08:00
Mingfei
362f0f93ac
Merge branch 'master' of https://github.com/mesos/spark
2013-06-08 15:20:13 +08:00
Mingfei
1a4d93c025
modify to pass job annotation by localProperties and use daeamon thread to do joblogger's work
2013-06-08 14:23:39 +08:00
Matei Zaharia
b58a29295b
Small formatting and style fixes
2013-06-07 22:51:28 -07:00
Matei Zaharia
c8fc423bc2
Merge pull request #631 from jerryshao/master
...
Fix block manager UI display issue when enable spark.cleaner.ttl
2013-06-07 22:43:18 -07:00
Matei Zaharia
c9ca0a4a58
Small code style fix to SchedulingAlgorithm.scala
2013-06-07 22:40:44 -07:00
Matei Zaharia
1ae60bcb36
Merge pull request #634 from xiajunluan/master
...
[Spark-753] Fix ClusterSchedulSuite unit test failed
2013-06-07 22:39:06 -07:00
Shivaram Venkataraman
ac480fd977
Clean up variables and counters in BlockFetcherIterator
2013-06-06 16:34:27 -07:00
Gavin Li
e179ff8a32
update according to comments
2013-06-05 22:41:05 +00:00
Shivaram Venkataraman
cb2f5046ee
Pass in bufferSize to BufferedOutputStream
2013-06-05 15:09:02 -07:00
Shivaram Venkataraman
c851957fe4
Don't write zero block files with java serializer
2013-06-05 14:28:38 -07:00
Christopher Nguyen
9d35904357
In the current code, when both partitions happen to have zero-length, the return mean will be NaN.
...
Consequently, the result of mean after reducing over all partitions will also be NaN,
which is not correct if there are partitions with non-zero length. This patch fixes this issue.
2013-06-04 22:12:47 -07:00
Matei Zaharia
fff3728552
Merge pull request #640 from pwendell/timeout-update
...
Fixing bug in BlockManager timeout
2013-06-04 16:09:50 -07:00
Patrick Wendell
061fd3ae36
Fixing bug in BlockManager timeout
2013-06-04 19:02:44 -04:00
Matei Zaharia
f420d4f228
Merge pull request #639 from pwendell/timeout-update
...
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 15:25:58 -07:00
Patrick Wendell
8bd4e12104
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 18:14:24 -04:00
Shivaram Venkataraman
96943a1cc0
var to val
2013-06-03 12:29:38 -07:00
Shivaram Venkataraman
cd347f547a
Reuse the file object as it is valid after delete
2013-06-03 12:27:51 -07:00
Shivaram Venkataraman
a058b0acf3
Delete a file for a block if it already exists.
2013-06-03 12:10:00 -07:00
Andrew xia
606bb1b450
Fix schedulingAlgorithm bugs for unit test
2013-06-03 10:29:23 +08:00
Shivaram Venkataraman
038cfc1a9a
Make connect timeout configurable
2013-05-31 23:32:18 -07:00
Shivaram Venkataraman
91aca92249
Another round of Netty fixes.
...
1. Avoid race condition between stop and copier completion
2. Handle socket exceptions by reporting them and filling in a failed
FetchResult
2013-05-31 23:21:38 -07:00
Gavin Li
9f84315c05
enhance pipe to support what we can do in hadoop streaming
2013-06-01 00:26:10 +00:00
Reynold Xin
de1167bf2c
Incorporated Charles' feedback to put rdd metadata removal in
...
BlockManagerMasterActor.
2013-05-31 15:54:57 -07:00
Reynold Xin
ba5e544461
More block manager cleanup.
...
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
2013-05-31 01:48:16 -07:00
jerryshao
926f41cc52
fix block manager UI display issue when enable spark.cleaner.ttl
2013-05-31 09:32:52 +08:00
Reynold Xin
bed1b08169
Do not create symlink for local add file. Instead, copy the file.
...
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
2013-05-30 16:21:49 -07:00