ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1509 commits 6 branches 181 tags 397 MiB
Commit graph

1509 commits

This branch
This branch
All branches
Author SHA1 Message Date
Reynold Xin be7166146b Removed the use of getOrElse to avoid Scala wrapper for every call. 2013-01-13 15:27:28 -08:00
Reynold Xin bd336f5f40 Changed CoGroupRDD's hash map from Scala to Java. 2013-01-10 17:13:04 -08:00
Matei Zaharia 14972141f9 Merge pull request #344 from mbautin/log_preferred_hosts 
Log preferred hosts
 2013-01-08 22:26:34 -08:00
Matei Zaharia d0bae072ea Merge pull request #353 from stephenh/tupleBy 
Add RDD.tupleBy.
 2013-01-08 22:24:03 -08:00
Stephen Haberman 8ac0f35be4 Add JavaRDDLike.keyBy. 2013-01-08 09:57:45 -06:00
Stephen Haberman 4ee6b22775 Merge branch 'master' into tupleBy 
Conflicts:
	core/src/test/scala/spark/RDDSuite.scala
 2013-01-08 09:10:10 -06:00
Mikhail Bautin 4725b0f643 Fixing if/else coding style for preferred hosts logging 2013-01-07 20:09:26 -08:00
Mikhail Bautin c41042c816 Log preferred hosts 2013-01-07 20:06:09 -08:00
Matei Zaharia a37adfa67b Merge pull request #354 from shivaram/ibm-jdk-fixes 
Fixes to build and test spark on IBM JVM
 2013-01-07 17:37:03 -08:00
Shivaram Venkataraman b1336e2fe4 Update expected size of strings to match our dummy string class 2013-01-07 17:00:32 -08:00
Shivaram Venkataraman fb3d4d5e85 Make default hadoop version 1.0.3 in pom.xml 2013-01-07 16:46:06 -08:00
Shivaram Venkataraman 55c66d365f Use a dummy string class in Size Estimator tests to make it resistant to jdk 
versions
 2013-01-07 15:58:00 -08:00
Shivaram Venkataraman 77d751731c Remove unused BoundedMemoryCache file and associated test case. 2013-01-07 15:57:46 -08:00
Shivaram Venkataraman aed368a970 Update Hadoop dependency to 1.0.3 as 0.20 has Sun specific dependencies. Also 
fix SequenceFileRDDFunctions to pick the right type conversion across Hadoop
versions
 2013-01-07 15:57:33 -08:00
Shivaram Venkataraman f8d579a0c0 Remove dependencies on sun jvm classes. Instead use reflection to infer 
HotSpot options and total physical memory size
 2013-01-07 15:57:18 -08:00
Matei Zaharia 1941d9602d Merge branch 'master' of github.com:mesos/spark 2013-01-07 16:50:39 -05:00
Matei Zaharia 9c32f300fb Add Accumulable.setValue for easier use in Java 2013-01-07 16:50:23 -05:00
Stephen Haberman 8dc06069fe Rename RDD.tupleBy to keyBy. 2013-01-06 15:21:45 -06:00
Matei Zaharia 8fd3a70c18 Add PairRDD.keys() and values() to Java API 2013-01-05 22:46:45 -05:00
Matei Zaharia b1663752c6 Merge pull request #351 from stephenh/values 
Add PairRDDFunctions.keys and values.
 2013-01-05 19:15:54 -08:00
Matei Zaharia 0982572519 Add methods called just 'accumulator' for int/double in Java API 2013-01-05 22:11:28 -05:00
Matei Zaharia 86af64b0a6 Fix Accumulators in Java, and add a test for them 2013-01-05 20:55:17 -05:00
Stephen Haberman 1fdb6946b5 Add RDD.tupleBy. 2013-01-05 13:07:59 -06:00
Stephen Haberman 6a0db3b449 Fix typo. 2013-01-05 12:56:17 -06:00
Matei Zaharia 7ab9f09140 Merge pull request #352 from stephenh/collect 
Add RDD.collect(PartialFunction).
 2013-01-05 10:17:20 -08:00
Stephen Haberman f4e6b9361f Add RDD.collect(PartialFunction). 2013-01-05 12:14:08 -06:00
Stephen Haberman 8d57c78c83 Add PairRDDFunctions.keys and values. 2013-01-05 12:04:01 -06:00
Matei Zaharia 55809fbc6d Merge pull request #349 from woggling/cache-finally 
Avoid stalls when computation of cached RDD throws exception
 2013-01-01 08:21:33 -08:00
Matei Zaharia c593f6329e Merge pull request #348 from JoshRosen/spark-597 
Raise exception when hashing Java arrays (SPARK-597)
 2013-01-01 08:20:06 -08:00
Charles Reiss 58072a7340 Remove some dead comments 2013-01-01 08:07:44 -08:00
Charles Reiss 21636ee4fa Test with exception while computing cached RDD. 2013-01-01 08:07:40 -08:00
Charles Reiss feadaf72f4 Mark key as not loading in CacheTracker even when compute() fails 2013-01-01 07:57:20 -08:00
Josh Rosen f803953998 Raise exception when hashing Java arrays (SPARK-597) 2012-12-31 20:20:11 -08:00
Matei Zaharia 3f74f729a1 Merge pull request #345 from JoshRosen/fix/add-file 
Fix deletion of files in current working directory by clearFiles()
 2012-12-29 15:01:33 -08:00
Josh Rosen 397e67103c Change Utils.fetchFile() warning to SparkException. 2012-12-28 17:37:13 -08:00
Josh Rosen d64fa72d2e Add addFile() and addJar() to JavaSparkContext. 2012-12-28 17:00:57 -08:00
Josh Rosen bd237d4a9d Add synchronization to LocalScheduler.updateDependencies(). 2012-12-28 17:00:57 -08:00
Josh Rosen f1bf4f0385 Skip deletion of files in clearFiles(). 
This fixes an issue where Spark could delete
original files in the current working directory
that were added to the job using addFile().

There was also the potential for addFile() to
overwrite local files, which is addressed by
changing Utils.fetchFile() to log a warning
instead of overwriting a file with new contents.

This is a short-term fix; a better long-term
solution would be to remove the dependence on
storing files in the current working directory,
since we can't change the cwd from Java.
 2012-12-28 17:00:57 -08:00
Matei Zaharia 84587a9bf3 Merge pull request #343 from markhamstra/spark-601 
lookup() needn't fail when there is no partitioner
 2012-12-24 15:28:05 -08:00
Mark Hamstra 903f3518df fall back to filter-map-collect when calling lookup() on an RDD without a partitioner 2012-12-24 13:18:45 -08:00
Matei Zaharia b575cbe069 Merge pull request #342 from markhamstra/spark-645 
Allow distinct() to be called without parentheses
 2012-12-24 08:04:50 -08:00
Mark Hamstra 61be8566e2 Allow distinct() to be called without parentheses when using the default number of splits. 2012-12-24 02:36:47 -08:00
Reynold Xin a6bb41c6d3 Updated Kryo version for Maven pom file. 2012-12-21 16:25:50 -08:00
Reynold Xin c68a076037 Updated Kryo documentation for Kryo version update. 2012-12-21 16:03:17 -08:00
Reynold Xin 60f7338092 Remove the call to close input stream in Kryo serializer. 2012-12-21 15:49:33 -08:00
Matei Zaharia 3334b7c6b5 Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677 
Kryo2 update against Spark master
 2012-12-21 15:31:23 -08:00
Matei Zaharia 5e51b889fe Merge pull request #327 from rxin/spark-633 
Added the ability in block manager to remove blocks.
 2012-12-20 11:33:38 -08:00
Reynold Xin 9397c5014e Let the slave notify the master block removal. 2012-12-20 01:37:09 -08:00
Matei Zaharia e7051767f7 Merge pull request #337 from pwendell/worker-liveness-ui 
SPARK-616: Logging dead workers in Web UI.
 2012-12-19 15:31:32 -08:00
Reynold Xin 68c52d80ec Moved BlockManager's IdGenerator into BlockManager object. Removed some 
excessive debug messages.
 2012-12-19 15:27:23 -08:00