ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1488 commits 6 branches 181 tags 397 MiB
Commit graph

1488 commits

This branch
This branch
All branches
Author SHA1 Message Date
Shivaram Venkataraman 22ba3a3f77 Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-13 13:32:10 -07:00
Shivaram Venkataraman 1f68c4b03b Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-13 13:31:54 -07:00
Shivaram Venkataraman 1ea269110c Move object size and pointer size initialization into a function to enable unit-testing 2012-08-13 13:31:45 -07:00
Shivaram Venkataraman 44661df9cc If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-13 13:31:39 -07:00
Shivaram Venkataraman 0dd8fe73ba Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-13 13:31:29 -07:00
Shivaram Venkataraman 80104ce1da Add link to Java wiki which specifies what changes with compressed oops 2012-08-13 13:31:21 -07:00
Shivaram Venkataraman 00ab5490b3 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-13 13:31:11 -07:00
Matei Zaharia 680df96c43 Merge pull request #169 from shivaram/master 
Changes to SizeEstimator more accurate
 2012-08-12 22:29:11 -07:00
Shivaram Venkataraman 2ee731211a Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-12 17:18:01 -07:00
Shivaram Venkataraman 54502238a2 Move object size and pointer size initialization into a function to enable unit-testing 2012-08-12 17:16:27 -07:00
Matei Zaharia 6ae3c375a9 Renamed apply() to call() in Java API and allowed it to throw Exceptions 2012-08-12 23:10:19 +02:00
Matei Zaharia 0141879c40 Use Promises instead of having a Future wait on a thread in 
ConnectionManager.
 2012-08-12 22:16:32 +02:00
Matei Zaharia 845a870242 Return remotely fetched blocks in a pipelined fashion from BlockManager 2012-08-12 20:01:38 +02:00
Matei Zaharia e17ed9a21d Switch to Akka futures in connection manager. 
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
 2012-08-12 19:40:37 +02:00
Matei Zaharia ad8a7612a4 Changed multi-get method in BlockManager to return an iterator 2012-08-12 19:18:01 +02:00
Shivaram Venkataraman 73452cc649 Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-11 16:42:35 -07:00
Shivaram Venkataraman 64b8fd62f0 If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-11 16:40:33 -07:00
Shivaram Venkataraman c0e773aa01 Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-11 14:38:05 -07:00
Shivaram Venkataraman f2475ca95a Add link to Java wiki which specifies what changes with compressed oops 2012-08-11 02:34:20 -07:00
Shivaram Venkataraman 980585b220 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-11 02:18:39 -07:00
Matei Zaharia 3c94e5c188 Merge pull request #168 from shivaram/dev 
Use JavaConversion to get a scala iterator
 2012-08-10 00:57:33 -07:00
Matei Zaharia e463e7a333 Merge pull request #167 from JoshRosen/piped-rdd-fixes 
Detect non-zero exit status from PipedRDD process
 2012-08-10 00:56:42 -07:00
Josh Rosen 59c22fb444 Print exit status in PipedRDD failure exception. 2012-08-10 00:33:56 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Shivaram Venkataraman 1803cce692 Use an implicit conversion to get the scala iterator 2012-08-08 14:31:04 -07:00
Shivaram Venkataraman 674fcf56bf Use JavaConversion to get a scala iterator 2012-08-08 14:10:23 -07:00
Matei Zaharia bec4d362c8 Merge pull request #166 from shivaram/dev 
Avoid a copy in ShuffleMapTask
 2012-08-08 09:11:19 -07:00
Shivaram Venkataraman f4aaec7a48 Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the 
block manager.
 2012-08-08 00:47:02 -07:00
Mosharaf Chowdhury d821dd3ccc BroadcastManager is a class now (replaced Braodcast object) 2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury b4804119f9 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-04 20:42:12 -07:00
Matei Zaharia 88b016db2a Merge pull request #160 from dennybritz/clusterscripts 
Standalone cluster scripts
 2012-08-04 17:45:20 -07:00
Matei Zaharia 1c5ae3edf2 Merge pull request #151 from dennybritz/fix/examples_jar 
Examples ship to to cluster
 2012-08-04 17:39:41 -07:00
Denny 8fb955fd40 Add Apache license to non-trivial scripts taken from Hadoop. 2012-08-04 17:04:33 -07:00
Denny 38d86d2616 updated readme 2012-08-04 16:58:47 -07:00
Denny 48cac4171c Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR 2012-08-04 16:56:32 -07:00
Denny 63c2020f93 Merge branch 'master' into fix/examples_jar 2012-08-04 16:55:18 -07:00
Mosharaf Chowdhury 1b0534af8f Merge branch 'dev' into bc-bm 2012-08-04 00:30:08 -07:00
Mosharaf Chowdhury d11b457e67 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-04 00:28:10 -07:00
Mosharaf Chowdhury 24b7eb872c Bug fixed. Broadcast now works with BlockManager. 2012-08-04 00:27:28 -07:00
Matei Zaharia 5cefda9984 Merge pull request #165 from shivaram/dev 
Fix test checkpoint to reuse spark context defined in the class
 2012-08-03 19:17:50 -07:00
Shivaram Venkataraman ce3444d2cb Fix testcheckpoint to reuse spark context defined in the class 2012-08-03 18:52:26 -07:00
Matei Zaharia 62898b631f Made range partition balance tests more aggressive. 
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
 2012-08-03 16:46:48 -04:00
Matei Zaharia abca699378 Made range partition balance tests more aggressive. 
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
 2012-08-03 16:44:17 -04:00
Matei Zaharia 6601a6212b Added a unit test for cross-partition balancing in sort, and changes to 
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
 2012-08-03 16:40:45 -04:00
Harvey 1170de3757 Fix for partitioning when sorting in descending order 2012-08-03 16:40:38 -04:00
Paul Cavallaro d05c0f97ca Logging Throwables in Info and Debug 
Logging Throwables in logInfo and logDebug instead of swallowing them.

Conflicts:

	core/src/main/scala/spark/Logging.scala
 2012-08-03 16:40:21 -04:00
Matei Zaharia 6da2bcdba1 Added a unit test for cross-partition balancing in sort, and changes to 
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
 2012-08-03 16:37:35 -04:00
Matei Zaharia 508221b8e6 Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int) 2012-08-03 15:57:43 -04:00
Matei Zaharia c0d5bd6553 Merge pull request #164 from HarveyFeng/master 
Bug fix in RangePartitioner for partitioning when sorting in descending order.
 2012-08-03 12:27:17 -07:00
Harvey 5ec13327d4 Fix for partitioning when sorting in descending order 2012-08-03 12:22:07 -07:00