ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2821 commits 6 branches 181 tags 397 MiB
Commit graph

2821 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin f6ad3781b1 Fixed the flaky unpersist test in RDDSuite. 2013-05-30 16:28:08 -07:00
Reynold Xin bed1b08169 Do not create symlink for local add file. Instead, copy the file. 
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
 2013-05-30 16:21:49 -07:00
Reynold Xin ef77bb73c6 Merge pull request #627 from shivaram/master 
Netty and shuffle  bug fixes
 2013-05-30 14:50:06 -07:00
Shivaram Venkataraman 3b0cd17343 Merge branch 'master' of git://github.com/mesos/spark 
Conflicts:
	core/src/test/scala/spark/ShuffleSuite.scala
 2013-05-30 14:36:24 -07:00
Reynold Xin 8cb817820f Merge pull request #628 from shivaram/zero-block-size 
Skip fetching zero-sized blocks in NIO.
 2013-05-30 14:17:44 -07:00
Shivaram Venkataraman 19fd6d54c0 Also flush serializer in revertPartialWrites 2013-05-29 17:29:34 -07:00
Shivaram Venkataraman 618c8cae1e Skip fetching zero-sized blocks in OIO. 
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
 2013-05-29 13:18:54 -07:00
Matei Zaharia 6ed71390d9 Merge pull request #626 from stephenh/remove-add-if-no-port 
Remove unused addIfNoPort.
 2013-05-29 10:14:22 -07:00
Shivaram Venkataraman b79b10a6d6 Flush serializer to fix zero-size kryo blocks bug. 
Also convert the local-cluster test case to check for non-zero block sizes
 2013-05-29 00:52:55 -07:00
Matei Zaharia 41d230ccb0 Merge pull request #611 from squito/classloader 
Use default classloaders for akka & deserializing task results
 2013-05-28 23:35:24 -07:00
Shivaram Venkataraman fbc1ab3468 Couple of Netty fixes 
a. Fix the port number by reading it from the bound channel
b. Fix the shutdown sequence to make sure we actually block on the channel
c. Fix the unit test to use two JVMs.
 2013-05-28 16:27:16 -07:00
Stephen Haberman 4fe1fbdd51 Remove unused addIfNoPort. 2013-05-28 16:26:32 -05:00
Matei Zaharia 3db1e17baa Merge pull request #620 from jerryshao/master 
Fix CheckpointRDD java.io.FileNotFoundException when calling getPreferredLocations
 2013-05-27 21:31:43 -07:00
Josh Rosen 3d4891d9fb Merge pull request #621 from JoshRosen/spark-613 
Use ec2-metadata in start-slave.sh to detect if running on EC2
 2013-05-25 23:38:05 -07:00
Matei Zaharia e8d4b6c296 Merge pull request #529 from xiajunluan/master 
[SPARK-663]Implement Fair Scheduler in Spark Cluster Scheduler
 2013-05-25 21:09:03 -07:00
Matei Zaharia 9a3c344679 Merge pull request #624 from rxin/master 
NonJavaSerializableClass should not be Java serializable...
 2013-05-25 17:53:43 -07:00
Reynold Xin 6bbbe01287 Fixed a stupid mistake that NonJavaSerializableClass was made Java 
serializable.
 2013-05-24 16:51:45 -07:00
Matei Zaharia 24e41aa423 Merge pull request #623 from rxin/master 
Automatically configure Netty port.
 2013-05-24 16:48:52 -07:00
Reynold Xin 26962c9340 Automatically configure Netty port. This makes unit tests using 
local-cluster pass. Previously they were failing because Netty was
trying to bind to the same port for all processes.

Pair programmed with @shivaram.
 2013-05-24 16:39:33 -07:00
Matei Zaharia 69161f9cbc Merge pull request #622 from rxin/master 
bug fix: Shuffle block iterator is ignoring the shuffle serializer setting.
 2013-05-24 14:42:13 -07:00
Reynold Xin 6ea085169d Fixed the bug that shuffle serializer is ignored by the new shuffle 
block iterators for local blocks. Also added a unit test for that.
 2013-05-24 14:08:37 -07:00
Josh Rosen cda2b15041 Use ec2-metadata in start-slave.sh. 
PR #419 applied the same change, but only to start-master.sh,
so some workers were still starting their web UI's using internal
addresses.

This should finally fix SPARK-613.
 2013-05-24 13:05:06 -07:00
jerryshao bd3ea8f2a6 fix CheckpointRDD getPreferredLocations java.io.FileNotFoundException 2013-05-24 14:26:19 +08:00
Matei Zaharia dbbedfc535 Merge pull request #616 from jey/maven-netty-exclusion 
Exclude old versions of Netty from Maven-based build
 2013-05-23 23:11:06 -07:00
Matei Zaharia a2b0a7975c Merge pull request #619 from woggling/adjust-sampling 
Use ARRAY_SAMPLE_SIZE constant instead of hard-coded 100.0 in SizeEstimator
 2013-05-21 18:16:20 -07:00
Charles Reiss f350f14084 Use ARRAY_SAMPLE_SIZE constant instead of 100.0 2013-05-21 18:11:33 -07:00
Reynold Xin 66dac449fb Merge pull request #618 from woggling/dead-code-disttest 
DistributedSuite: remove dead code
 2013-05-21 11:41:42 -07:00
Charles Reiss 786c97b87c DistributedSuite: remove dead test code 2013-05-21 11:35:49 -07:00
Andrew xia ecd6d75c6a fix bug of unit tests 2013-05-21 06:49:23 +08:00
Jey Kottalam e7982c798e Exclude old versions of Netty from Maven-based build 2013-05-18 21:24:58 -07:00
Reynold Xin 5912cc4967 Merge pull request #610 from JoshRosen/spark-747 
Throw exception if TaskResult exceeds Akka frame size
 2013-05-17 19:58:40 -07:00
Reynold Xin 8d78c5f89f Changed the logging level from info to warning when addJar(null) is 
called.
 2013-05-17 18:51:35 -07:00
Reynold Xin 0eab7a78b9 Fixed a couple typos and formating problems in the YARN documentation. 2013-05-17 18:05:46 -07:00
Reynold Xin 7760d78b3a Merge branch 'master' of https://github.com/mridulm/spark 2013-05-17 17:58:36 -07:00
Reynold Xin 6729c2ead8 Merge branch 'master' of github.com:mesos/spark 2013-05-17 17:58:06 -07:00
Andrew xia 3d4672eaa9 Merge branch 'master' into xiajunluan 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/scheduler/cluster/ClusterScheduler.scala
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
 2013-05-18 07:28:03 +08:00
Andrew xia d19753b9c7 expose TaskSetManager type to resourceOffer function in ClusterScheduler 2013-05-18 06:45:19 +08:00
Mridul Muralidharan da2642bead Fix example jar name 2013-05-17 06:58:46 +05:30
Reynold Xin 6c27c38451 Merge pull request #615 from rxin/build-fix 
Maven build fix & two other small changes
 2013-05-16 17:33:56 -07:00
Reynold Xin dc146406ae Updated Scala version in docs generation ruby script. 2013-05-16 17:07:14 -07:00
Reynold Xin 3b3300383a Updated Scala version in docs generation ruby script. 2013-05-16 16:51:28 -07:00
Reynold Xin 43644a293f Only check for repl classes if the user is running the repl. Otherwise, 
check for core classes in run. This fixed the problem that core tests
depend on whether repl module is compiled or not.
 2013-05-16 14:31:38 -07:00
Reynold Xin 61cf176238 Added dependency on netty-all in Maven. 2013-05-16 14:31:26 -07:00
Reynold Xin f742435f18 Removed the duplicated netty dependency in SBT build file. 2013-05-16 14:31:03 -07:00
Andrew xia c6e2770bfe Fix ClusterScheduler bug to avoid allocating tasks to same slave 2013-05-17 05:10:38 +08:00
Mridul Muralidharan f0881f8d48 Hope this does not turn into a bike shed change 2013-05-17 01:58:50 +05:30
Mridul Muralidharan f16c781709 Fix documentation to use yarn-standalone as master 2013-05-16 17:50:22 +05:30
Mridul Muralidharan feddd2530d Filter out nulls - prevent NPE 2013-05-16 17:49:14 +05:30
Mridul Muralidharan 87540a7b38 Fix running on yarn documentation 2013-05-16 15:27:58 +05:30
Josh Rosen b8e46b6074 Abort job if result exceeds Akka frame size; add test. 2013-05-16 01:57:57 -07:00