ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3012 commits 6 branches 181 tags 397 MiB
Commit graph

3012 commits

This branch
This branch
All branches
Author SHA1 Message Date
Rohit Rai b104c7f5c7 Example to write the output to cassandra 2013-06-03 15:15:52 +05:30
Rohit Rai 56c64c4033 A better way to read column value if you are sure the column exists in every row. 2013-06-03 12:48:35 +05:30
Andrew xia 606bb1b450 Fix schedulingAlgorithm bugs for unit test 2013-06-03 10:29:23 +08:00
Gavin Li 4a9913d66a add ut for pipe enhancement 2013-06-02 23:21:09 +00:00
Rohit Rai 6d8423fd1b Adding deps to examples/pom.xml 
Fixing exclusion in examples deps in SparkBuild.scala
 2013-06-02 13:03:45 +05:30
Rohit Rai 81c2adc15c Removing infix call 2013-06-02 12:51:15 +05:30
Rohit Rai 3be7bdcefd Adding example to make Spark RDD from Cassandra 2013-06-01 19:32:17 +05:30
Shivaram Venkataraman 038cfc1a9a Make connect timeout configurable 2013-05-31 23:32:18 -07:00
Shivaram Venkataraman 91aca92249 Another round of Netty fixes. 
1. Avoid race condition between stop and copier completion
2. Handle socket exceptions by reporting them and filling in a failed
FetchResult
 2013-05-31 23:21:38 -07:00
Gavin Li 9f84315c05 enhance pipe to support what we can do in hadoop streaming 2013-06-01 00:26:10 +00:00
Matei Zaharia 84530ba6d9 Merge pull request #636 from rxin/unpersist 
Unpersist More block manager cleanup.
 2013-05-31 17:06:13 -07:00
Reynold Xin de1167bf2c Incorporated Charles' feedback to put rdd metadata removal in 
BlockManagerMasterActor.
 2013-05-31 15:54:57 -07:00
Reynold Xin ba5e544461 More block manager cleanup. 
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
 2013-05-31 01:48:16 -07:00
jerryshao 926f41cc52 fix block manager UI display issue when enable spark.cleaner.ttl 2013-05-31 09:32:52 +08:00
Reynold Xin f6ad3781b1 Fixed the flaky unpersist test in RDDSuite. 2013-05-30 16:28:08 -07:00
Reynold Xin bed1b08169 Do not create symlink for local add file. Instead, copy the file. 
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
 2013-05-30 16:21:49 -07:00
Reynold Xin ef77bb73c6 Merge pull request #627 from shivaram/master 
Netty and shuffle  bug fixes
 2013-05-30 14:50:06 -07:00
Shivaram Venkataraman 3b0cd17343 Merge branch 'master' of git://github.com/mesos/spark 
Conflicts:
	core/src/test/scala/spark/ShuffleSuite.scala
 2013-05-30 14:36:24 -07:00
Reynold Xin 8cb817820f Merge pull request #628 from shivaram/zero-block-size 
Skip fetching zero-sized blocks in NIO.
 2013-05-30 14:17:44 -07:00
Andrew xia c3db3ea554 1. Add unit test for local scheduler 
2. Move localTaskSetManager to a new file
 2013-05-30 20:49:40 +08:00
Andrew xia ecceb101d3 implement FIFO and fair scheduler for spark local mode 2013-05-30 10:43:01 +08:00
Shivaram Venkataraman 19fd6d54c0 Also flush serializer in revertPartialWrites 2013-05-29 17:29:34 -07:00
Shivaram Venkataraman 618c8cae1e Skip fetching zero-sized blocks in OIO. 
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
 2013-05-29 13:18:54 -07:00
Matei Zaharia 6ed71390d9 Merge pull request #626 from stephenh/remove-add-if-no-port 
Remove unused addIfNoPort.
 2013-05-29 10:14:22 -07:00
Shivaram Venkataraman b79b10a6d6 Flush serializer to fix zero-size kryo blocks bug. 
Also convert the local-cluster test case to check for non-zero block sizes
 2013-05-29 00:52:55 -07:00
Matei Zaharia 41d230ccb0 Merge pull request #611 from squito/classloader 
Use default classloaders for akka & deserializing task results
 2013-05-28 23:35:24 -07:00
Shivaram Venkataraman fbc1ab3468 Couple of Netty fixes 
a. Fix the port number by reading it from the bound channel
b. Fix the shutdown sequence to make sure we actually block on the channel
c. Fix the unit test to use two JVMs.
 2013-05-28 16:27:16 -07:00
Stephen Haberman 4fe1fbdd51 Remove unused addIfNoPort. 2013-05-28 16:26:32 -05:00
Stephen Haberman a674d67c0a Fix start-slave not passing instance number to spark-daemon. 2013-05-28 16:24:19 -05:00
Matei Zaharia 3db1e17baa Merge pull request #620 from jerryshao/master 
Fix CheckpointRDD java.io.FileNotFoundException when calling getPreferredLocations
 2013-05-27 21:31:43 -07:00
Josh Rosen 3d4891d9fb Merge pull request #621 from JoshRosen/spark-613 
Use ec2-metadata in start-slave.sh to detect if running on EC2
 2013-05-25 23:38:05 -07:00
Matei Zaharia e8d4b6c296 Merge pull request #529 from xiajunluan/master 
[SPARK-663]Implement Fair Scheduler in Spark Cluster Scheduler
 2013-05-25 21:09:03 -07:00
Matei Zaharia 9a3c344679 Merge pull request #624 from rxin/master 
NonJavaSerializableClass should not be Java serializable...
 2013-05-25 17:53:43 -07:00
Reynold Xin 6bbbe01287 Fixed a stupid mistake that NonJavaSerializableClass was made Java 
serializable.
 2013-05-24 16:51:45 -07:00
Matei Zaharia 24e41aa423 Merge pull request #623 from rxin/master 
Automatically configure Netty port.
 2013-05-24 16:48:52 -07:00
Reynold Xin 26962c9340 Automatically configure Netty port. This makes unit tests using 
local-cluster pass. Previously they were failing because Netty was
trying to bind to the same port for all processes.

Pair programmed with @shivaram.
 2013-05-24 16:39:33 -07:00
Matei Zaharia 69161f9cbc Merge pull request #622 from rxin/master 
bug fix: Shuffle block iterator is ignoring the shuffle serializer setting.
 2013-05-24 14:42:13 -07:00
Reynold Xin 6ea085169d Fixed the bug that shuffle serializer is ignored by the new shuffle 
block iterators for local blocks. Also added a unit test for that.
 2013-05-24 14:08:37 -07:00
Josh Rosen cda2b15041 Use ec2-metadata in start-slave.sh. 
PR #419 applied the same change, but only to start-master.sh,
so some workers were still starting their web UI's using internal
addresses.

This should finally fix SPARK-613.
 2013-05-24 13:05:06 -07:00
jerryshao bd3ea8f2a6 fix CheckpointRDD getPreferredLocations java.io.FileNotFoundException 2013-05-24 14:26:19 +08:00
Matei Zaharia dbbedfc535 Merge pull request #616 from jey/maven-netty-exclusion 
Exclude old versions of Netty from Maven-based build
 2013-05-23 23:11:06 -07:00
Matei Zaharia a2b0a7975c Merge pull request #619 from woggling/adjust-sampling 
Use ARRAY_SAMPLE_SIZE constant instead of hard-coded 100.0 in SizeEstimator
 2013-05-21 18:16:20 -07:00
Charles Reiss f350f14084 Use ARRAY_SAMPLE_SIZE constant instead of 100.0 2013-05-21 18:11:33 -07:00
Reynold Xin 66dac449fb Merge pull request #618 from woggling/dead-code-disttest 
DistributedSuite: remove dead code
 2013-05-21 11:41:42 -07:00
Charles Reiss 786c97b87c DistributedSuite: remove dead test code 2013-05-21 11:35:49 -07:00
Ethan Jewett 3217d486f7 Add hBase dependency to examples POM 2013-05-20 19:41:38 -05:00
Andrew xia ecd6d75c6a fix bug of unit tests 2013-05-21 06:49:23 +08:00
Jey Kottalam e7982c798e Exclude old versions of Netty from Maven-based build 2013-05-18 21:24:58 -07:00
Reynold Xin 5912cc4967 Merge pull request #610 from JoshRosen/spark-747 
Throw exception if TaskResult exceeds Akka frame size
 2013-05-17 19:58:40 -07:00
Reynold Xin 8d78c5f89f Changed the logging level from info to warning when addJar(null) is 
called.
 2013-05-17 18:51:35 -07:00