Reynold Xin
ae0eca5ec8
Merge branch 'graph' of github.com:rxin/spark into graph
2013-06-29 15:22:33 -07:00
Reynold Xin
ae12d163dc
Added the BytecodeUtils class for analyzing bytecode.
2013-06-29 15:22:15 -07:00
Joseph E. Gonzalez
a80b28a579
Renamed several functions and classes and improved documentation
2013-06-28 18:18:51 -07:00
Reynold Xin
8abf4a8447
Merge branch 'master' of github.com:mesos/spark into graph
2013-06-10 12:28:43 -07:00
Matei Zaharia
74b91d53bc
Merge pull request #629 from c0s/master
...
Sometime Maven build runs out of PermGen space.
2013-06-08 01:19:40 -07:00
Matei Zaharia
b58a29295b
Small formatting and style fixes
2013-06-07 22:51:28 -07:00
Matei Zaharia
c8fc423bc2
Merge pull request #631 from jerryshao/master
...
Fix block manager UI display issue when enable spark.cleaner.ttl
2013-06-07 22:43:18 -07:00
Matei Zaharia
c9ca0a4a58
Small code style fix to SchedulingAlgorithm.scala
2013-06-07 22:40:44 -07:00
Matei Zaharia
1ae60bcb36
Merge pull request #634 from xiajunluan/master
...
[Spark-753] Fix ClusterSchedulSuite unit test failed
2013-06-07 22:39:06 -07:00
Reynold Xin
aad785374c
Merge branch 'master' of github.com:mesos/spark into graph
2013-06-06 23:28:58 -07:00
Matei Zaharia
fff3728552
Merge pull request #640 from pwendell/timeout-update
...
Fixing bug in BlockManager timeout
2013-06-04 16:09:50 -07:00
Patrick Wendell
061fd3ae36
Fixing bug in BlockManager timeout
2013-06-04 19:02:44 -04:00
Matei Zaharia
f420d4f228
Merge pull request #639 from pwendell/timeout-update
...
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 15:25:58 -07:00
Patrick Wendell
8bd4e12104
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 18:14:24 -04:00
Konstantin Boudnik
d1286231e0
Sometime Maven build runs out of PermGen space.
2013-06-03 15:55:44 -07:00
Andrew xia
606bb1b450
Fix schedulingAlgorithm bugs for unit test
2013-06-03 10:29:23 +08:00
Reynold Xin
b0403d3f2b
Merge branch 'master' of github.com:mesos/spark into graph
...
Conflicts:
run
2013-06-01 00:48:27 -07:00
Matei Zaharia
84530ba6d9
Merge pull request #636 from rxin/unpersist
...
Unpersist More block manager cleanup.
2013-05-31 17:06:13 -07:00
Reynold Xin
de1167bf2c
Incorporated Charles' feedback to put rdd metadata removal in
...
BlockManagerMasterActor.
2013-05-31 15:54:57 -07:00
Reynold Xin
ba5e544461
More block manager cleanup.
...
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
2013-05-31 01:48:16 -07:00
jerryshao
926f41cc52
fix block manager UI display issue when enable spark.cleaner.ttl
2013-05-31 09:32:52 +08:00
Reynold Xin
f6ad3781b1
Fixed the flaky unpersist test in RDDSuite.
2013-05-30 16:28:08 -07:00
Reynold Xin
bed1b08169
Do not create symlink for local add file. Instead, copy the file.
...
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
2013-05-30 16:21:49 -07:00
Reynold Xin
ef77bb73c6
Merge pull request #627 from shivaram/master
...
Netty and shuffle bug fixes
2013-05-30 14:50:06 -07:00
Shivaram Venkataraman
3b0cd17343
Merge branch 'master' of git://github.com/mesos/spark
...
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2013-05-30 14:36:24 -07:00
Reynold Xin
8cb817820f
Merge pull request #628 from shivaram/zero-block-size
...
Skip fetching zero-sized blocks in NIO.
2013-05-30 14:17:44 -07:00
Shivaram Venkataraman
19fd6d54c0
Also flush serializer in revertPartialWrites
2013-05-29 17:29:34 -07:00
Shivaram Venkataraman
618c8cae1e
Skip fetching zero-sized blocks in OIO.
...
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
2013-05-29 13:18:54 -07:00
Matei Zaharia
6ed71390d9
Merge pull request #626 from stephenh/remove-add-if-no-port
...
Remove unused addIfNoPort.
2013-05-29 10:14:22 -07:00
Shivaram Venkataraman
b79b10a6d6
Flush serializer to fix zero-size kryo blocks bug.
...
Also convert the local-cluster test case to check for non-zero block sizes
2013-05-29 00:52:55 -07:00
Matei Zaharia
41d230ccb0
Merge pull request #611 from squito/classloader
...
Use default classloaders for akka & deserializing task results
2013-05-28 23:35:24 -07:00
Shivaram Venkataraman
fbc1ab3468
Couple of Netty fixes
...
a. Fix the port number by reading it from the bound channel
b. Fix the shutdown sequence to make sure we actually block on the channel
c. Fix the unit test to use two JVMs.
2013-05-28 16:27:16 -07:00
Stephen Haberman
4fe1fbdd51
Remove unused addIfNoPort.
2013-05-28 16:26:32 -05:00
Matei Zaharia
3db1e17baa
Merge pull request #620 from jerryshao/master
...
Fix CheckpointRDD java.io.FileNotFoundException when calling getPreferredLocations
2013-05-27 21:31:43 -07:00
Josh Rosen
3d4891d9fb
Merge pull request #621 from JoshRosen/spark-613
...
Use ec2-metadata in start-slave.sh to detect if running on EC2
2013-05-25 23:38:05 -07:00
Matei Zaharia
e8d4b6c296
Merge pull request #529 from xiajunluan/master
...
[SPARK-663]Implement Fair Scheduler in Spark Cluster Scheduler
2013-05-25 21:09:03 -07:00
Matei Zaharia
9a3c344679
Merge pull request #624 from rxin/master
...
NonJavaSerializableClass should not be Java serializable...
2013-05-25 17:53:43 -07:00
Reynold Xin
6bbbe01287
Fixed a stupid mistake that NonJavaSerializableClass was made Java
...
serializable.
2013-05-24 16:51:45 -07:00
Matei Zaharia
24e41aa423
Merge pull request #623 from rxin/master
...
Automatically configure Netty port.
2013-05-24 16:48:52 -07:00
Reynold Xin
26962c9340
Automatically configure Netty port. This makes unit tests using
...
local-cluster pass. Previously they were failing because Netty was
trying to bind to the same port for all processes.
Pair programmed with @shivaram.
2013-05-24 16:39:33 -07:00
Matei Zaharia
69161f9cbc
Merge pull request #622 from rxin/master
...
bug fix: Shuffle block iterator is ignoring the shuffle serializer setting.
2013-05-24 14:42:13 -07:00
Reynold Xin
6ea085169d
Fixed the bug that shuffle serializer is ignored by the new shuffle
...
block iterators for local blocks. Also added a unit test for that.
2013-05-24 14:08:37 -07:00
Josh Rosen
cda2b15041
Use ec2-metadata in start-slave.sh.
...
PR #419 applied the same change, but only to start-master.sh,
so some workers were still starting their web UI's using internal
addresses.
This should finally fix SPARK-613.
2013-05-24 13:05:06 -07:00
jerryshao
bd3ea8f2a6
fix CheckpointRDD getPreferredLocations java.io.FileNotFoundException
2013-05-24 14:26:19 +08:00
Matei Zaharia
dbbedfc535
Merge pull request #616 from jey/maven-netty-exclusion
...
Exclude old versions of Netty from Maven-based build
2013-05-23 23:11:06 -07:00
Matei Zaharia
a2b0a7975c
Merge pull request #619 from woggling/adjust-sampling
...
Use ARRAY_SAMPLE_SIZE constant instead of hard-coded 100.0 in SizeEstimator
2013-05-21 18:16:20 -07:00
Charles Reiss
f350f14084
Use ARRAY_SAMPLE_SIZE constant instead of 100.0
2013-05-21 18:11:33 -07:00
Reynold Xin
66dac449fb
Merge pull request #618 from woggling/dead-code-disttest
...
DistributedSuite: remove dead code
2013-05-21 11:41:42 -07:00
Charles Reiss
786c97b87c
DistributedSuite: remove dead test code
2013-05-21 11:35:49 -07:00
Andrew xia
ecd6d75c6a
fix bug of unit tests
2013-05-21 06:49:23 +08:00