Dmitriy Lyubimov
7c52ecc6a4
(1) added reduce test case.
...
(2) added nested streaming in ParallelCollectionRDD
(3) added kryo with fold test which still doesn't work
2013-07-31 19:27:30 -07:00
Dmitriy Lyubimov
96664431cb
IDEA flipped JavaSerialized import at some point to a wrong class.
2013-07-30 23:10:09 -07:00
Dmitriy Lyubimov
c219fc94fd
Minor, style
2013-07-30 22:08:39 -07:00
Dmitriy Lyubimov
f4b4b8836e
reverting back to one-by-one serialization for parallelize()
2013-07-30 19:00:58 -07:00
Dmitriy Lyubimov
abada94ebf
removing default constructor (not Externalizable any more)
2013-07-30 18:04:02 -07:00
Dmitriy Lyubimov
943c6590c9
realiging "extends" back manually
2013-07-30 18:01:35 -07:00
Dmitriy Lyubimov
ca33b12e98
resetting wrap and continuation indent = 4
2013-07-30 17:51:44 -07:00
Dmitriy Lyubimov
483d8dd694
Reverting SparkBuild w.r.t to branch fork ratherh than current trunk to avoid diffs appearing
2013-07-30 17:41:24 -07:00
Dmitriy Lyubimov
7ef2551af0
reverting build file..again
2013-07-30 17:39:43 -07:00
Dmitriy Lyubimov
984b56155a
changing approaches for parallelize(): java serialization needs to avoid writing headers!
2013-07-30 17:36:59 -07:00
Dmitriy Lyubimov
ef9529a943
refactoring using writeByteBuffer() from Utils.
2013-07-30 16:24:23 -07:00
Dmitriy Lyubimov
43394b9a6d
fixing formatting
2013-07-30 16:13:41 -07:00
Dmitriy Lyubimov
13a9d66645
adding ===
2013-07-30 16:10:55 -07:00
Dmitriy Lyubimov
1bca91633e
+ bug fixes;
...
test added
Conflicts:
core/src/test/scala/spark/KryoSerializerSuite.scala
2013-07-30 11:04:11 -07:00
Dmitriy Lyubimov
8e5cd041bb
initial externalization of ParallelCollectionRDD's split
2013-07-29 19:02:53 -07:00
Dmitriy Lyubimov
b241fcfb35
reverting changes to SparkBuild.scala
2013-07-27 22:53:01 -07:00
Dmitriy Lyubimov
23f3e0f117
mixing in SharedSparkContext for the kryo-collect test
2013-07-26 19:15:11 -07:00
Dmitriy Lyubimov
72bac09c42
Leaking spark context in the test
2013-07-23 15:19:07 -07:00
Dmitriy Lyubimov
ef82ff8564
Merge branch 'master' into SPARK-826
...
Conflicts:
core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
2013-07-23 13:43:00 -07:00
Dmitriy Lyubimov
310e73d566
style
2013-07-23 13:23:25 -07:00
Matei Zaharia
f369e0e51b
Merge pull request #720 from ooyala/2013-07/persistent-rdds-api
...
Add a public method getCachedRdds to SparkContext
2013-07-23 13:22:27 -07:00
Dmitriy Lyubimov
ac60d06381
Re-working in terms of changes to TaskSetManager. Verified with Standalone and Local mode.
2013-07-23 13:13:19 -07:00
Evan Chan
efd6418c1b
Move getPersistentRDDs testing to a new Suite
2013-07-23 10:40:41 -07:00
Evan Chan
4830e22562
Rename method per rxin feedback
2013-07-23 09:50:13 -07:00
Evan Chan
2c2bfbe294
Add toMap method to TimeStampedHashMap and use it
2013-07-23 01:36:44 -07:00
Matei Zaharia
401aac8b18
Merge pull request #719 from karenfeng/ui-808
...
Creates Executors tab for Jobs UI
2013-07-22 16:57:16 -07:00
Karen Feng
872c97ad82
Split task columns, memory columns sort by numeric value
2013-07-22 16:54:37 -07:00
Matei Zaharia
ea1cfabfdd
Merge branch 'master' of github.com:mesos/spark
2013-07-22 16:22:02 -07:00
Josh Rosen
e17e1b388e
Remove annotation code that broke build.
2013-07-22 16:12:11 -07:00
Josh Rosen
c83680434b
Add JavaAPICompletenessChecker.
...
This is used to find methods in the Scala API that
need to be ported to the Java API. To use it:
./run spark.tools.JavaAPICompletenessChecker
Conflicts:
project/SparkBuild.scala
run
run2.cmd
2013-07-22 16:11:49 -07:00
Matei Zaharia
8e38e77232
Fix a test that was using an outdated config setting
2013-07-22 16:05:32 -07:00
Matei Zaharia
8ae1436981
Merge pull request #722 from JoshRosen/spark-825
...
Fix bug: DoubleRDDFunctions.sampleStdev() computed non-sample stdev()
2013-07-22 16:03:04 -07:00
Karen Feng
2eea974795
Executors UI now calls executor ID from TaskInfo instead of TaskMetrics
2013-07-22 15:15:54 -07:00
Dmitriy Lyubimov
8ca0c31944
removing non-pertinent comment
2013-07-22 14:48:46 -07:00
Dmitriy Lyubimov
b4b230e606
Fixing for LocalScheduler with test, that much works ..
2013-07-22 14:42:47 -07:00
Karen Feng
85c4d7bf3b
Shows number of complete/total/failed tasks (bug: failed tasks assigned to null executor)
2013-07-22 14:35:47 -07:00
Josh Rosen
f649dabb4a
Fix bug: DoubleRDDFunctions.sampleStdev() computed non-sample stdev().
...
Update JavaDoubleRDD to add new methods and docs.
Fixes SPARK-825.
2013-07-22 13:21:48 -07:00
Karen Feng
8901f379c9
Fixed memory used/remaining/total bug
2013-07-22 09:58:03 -07:00
Karen Feng
636b19f833
Merge branch 'master' of https://github.com/mesos/spark into ui-808
2013-07-22 09:53:26 -07:00
Evan Chan
0337d88321
Add a public method getCachedRdds to SparkContext
2013-07-21 18:26:14 -07:00
Matei Zaharia
15fb394833
Merge pull request #716 from c0s/webui-port
...
Regression: default webui-port can't be set via command line "--webui-port" anymore
2013-07-21 10:33:38 -07:00
Karen Feng
865dc63bac
Changed table format for executors
2013-07-19 15:57:01 -07:00
Karen Feng
81bb5dc640
Creates Executors tab for application with RDD block and memory/disk used, solves SPARK-808
2013-07-19 14:08:30 -07:00
Konstantin Boudnik
cfce9a6a36
Regression: default webui-port can't be set via command line "--webui-port" anymore
2013-07-19 14:00:58 -07:00
Matei Zaharia
c40f0f21f1
Merge pull request #711 from shivaram/ml-generators
...
Move ML lib data generator files to util/
2013-07-19 13:33:04 -07:00
Matei Zaharia
413b84172e
Merge pull request #717 from viirya/dev1
...
Do not copy local jars given to SparkContext in yarn mode
2013-07-19 13:31:38 -07:00
Liang-Chi Hsieh
d1738d72ba
also exclude asm for hadoop2. hadoop1 looks like no need to do that too.
2013-07-20 00:37:24 +08:00
Liang-Chi Hsieh
4530e8a9bf
fix typo.
2013-07-20 00:04:25 +08:00
Liang-Chi Hsieh
aa6f83289b
A better fix for giving local jars unde Yarn mode.
2013-07-19 22:25:28 +08:00
Liang-Chi Hsieh
a613628c50
Do not copy local jars given to SparkContext in yarn mode since the Context is not running on local. This bug causes failure when jars can not be found. Example codes (such as spark.examples.SparkPi) can not work without this fix under yarn mode.
2013-07-19 16:59:12 +08:00
