ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
801 commits 6 branches 181 tags 397 MiB
Commit graph

801 commits

This branch
This branch
All branches
Author SHA1 Message Date
Shivaram Venkataraman 2ee731211a Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-12 17:18:01 -07:00
Shivaram Venkataraman 54502238a2 Move object size and pointer size initialization into a function to enable unit-testing 2012-08-12 17:16:27 -07:00
Shivaram Venkataraman 73452cc649 Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-11 16:42:35 -07:00
Shivaram Venkataraman 64b8fd62f0 If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-11 16:40:33 -07:00
Shivaram Venkataraman c0e773aa01 Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-11 14:38:05 -07:00
Shivaram Venkataraman f2475ca95a Add link to Java wiki which specifies what changes with compressed oops 2012-08-11 02:34:20 -07:00
Shivaram Venkataraman 980585b220 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-11 02:18:39 -07:00
Matei Zaharia 1c5ae3edf2 Merge pull request #151 from dennybritz/fix/examples_jar 
Examples ship to to cluster
 2012-08-04 17:39:41 -07:00
Denny 38d86d2616 updated readme 2012-08-04 16:58:47 -07:00
Denny 48cac4171c Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR 2012-08-04 16:56:32 -07:00
Denny 63c2020f93 Merge branch 'master' into fix/examples_jar 2012-08-04 16:55:18 -07:00
Matei Zaharia abca699378 Made range partition balance tests more aggressive. 
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
 2012-08-03 16:44:17 -04:00
Matei Zaharia 6da2bcdba1 Added a unit test for cross-partition balancing in sort, and changes to 
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
 2012-08-03 16:37:35 -04:00
Matei Zaharia 508221b8e6 Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int) 2012-08-03 15:57:43 -04:00
Matei Zaharia c0d5bd6553 Merge pull request #164 from HarveyFeng/master 
Bug fix in RangePartitioner for partitioning when sorting in descending order.
 2012-08-03 12:27:17 -07:00
Harvey 5ec13327d4 Fix for partitioning when sorting in descending order 2012-08-03 12:22:07 -07:00
Matei Zaharia 4a9c58913d Merge pull request #157 from paulcavallaro/log-throwables 
Logging Throwables in Info and Debug
 2012-08-02 06:26:06 -07:00
Paul Cavallaro e3952f31de Logging Throwables in Info and Debug 
Logging Throwables in logInfo and logDebug instead of swallowing them.
 2012-07-30 13:41:09 -04:00
Matei Zaharia 1e2df26c33 Merge pull request #145 from squito/exp_accum 
add Accumulatable, add corresponding docs & tests for accumulators
 2012-07-26 17:25:45 -07:00
Imran Rashid 0384be3467 tasks cannot access value of accumulator 2012-07-26 12:38:51 -07:00
Matei Zaharia da4298135c Merge pull request #152 from dennybritz/fix/testbeforeafter 
Always destroy SparkContext in after block for the unit tests.
 2012-07-23 16:13:24 -07:00
Denny 5656dcdfe5 Stlystic changes 2012-07-23 10:36:52 -07:00
Matei Zaharia 840e1b21e4 Merge branch 'master' of github.com:mesos/spark 2012-07-21 21:58:40 -07:00
Matei Zaharia 6f44c0db74 Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-21 21:58:28 -07:00
Matei Zaharia d1759c0290 Merge pull request #149 from dennybritz/serfix 
Instantiating custom serializer using user's classpath
 2012-07-21 21:54:50 -07:00
Matei Zaharia 5122f11b05 Use full package name in import 2012-07-21 21:53:38 -07:00
Denny 5559608e6f Always destroy SparkContext in after block for the unit tests. 2012-07-18 13:09:50 -07:00
Denny e4dbaf653f syntax errors 2012-07-18 12:18:00 -07:00
Denny 1d98884548 Use extended constructor in the examples. 2012-07-18 11:46:03 -07:00
Denny 2132c541f0 Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader. 2012-07-17 14:05:26 -07:00
Denny 2b84b50a85 Use Context classloader for Serializer class 2012-07-17 13:55:23 -07:00
Imran Rashid 7f43ba7ffa one more minor cleanup to scaladoc 2012-07-16 18:26:48 -07:00
Imran Rashid 913d42c6a0 fix up scaladoc, naming of type parameters 2012-07-16 18:25:15 -07:00
Imran Rashid 85940a7d71 rename addToAccum to addAccumulator 2012-07-16 18:17:13 -07:00
Matei Zaharia a33ca6949c Merge branch 'master' of github.com:mesos/spark 2012-07-12 18:38:20 -07:00
Matei Zaharia 452330efb4 Allow null keys in Spark's reduce and group by 2012-07-12 18:37:37 -07:00
Imran Rashid 86024ca74d add some functionality to Vector, delete copy in AccumulatorSuite 2012-07-12 13:08:03 -07:00
Imran Rashid 42ce879486 move Vector class into core and spark.util package 2012-07-12 13:08:03 -07:00
Imran Rashid 13cc72cfb5 Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=) 2012-07-12 12:40:10 -07:00
Imran Rashid 73935629a1 improve scaladoc 2012-07-12 09:58:06 -07:00
Imran Rashid 30480e6dae add Accumulatable, add corresponding docs & tests for accumulators 2012-07-12 09:37:42 -07:00
Matei Zaharia 4259d37f84 Formatting 2012-07-10 11:16:34 -07:00
Matei Zaharia 6980b67557 Added more methods for loading/saving with new Hadoop API 2012-07-10 11:11:35 -07:00
Matei Zaharia cbe88465a9 Merge pull request #142 from rrmckinley/patch-1 
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/i...
 2012-06-29 16:40:57 -07:00
Matei Zaharia 3a326c0ddd Increase the default wait time for EC2 clusters to 2 minutes. 2012-06-29 16:25:06 -07:00
rrmckinley 25972b52cd Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24. Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too. 2012-06-29 12:00:23 -07:00
Matei Zaharia 800fcbfbca Revert "Fixed HttpBroadcast to work with this branch's Serializer." 
This reverts commit b3eeac55b8.
 2012-06-17 14:29:39 -07:00
Matei Zaharia 94d77f83d3 Revert "Merge branch 'master' into dev" 
This reverts commit f58da6164e, reversing
changes made to 4449eb9783.
 2012-06-17 14:27:45 -07:00
Matei Zaharia f46e867249 Revert "Update version number for dev branch" 
This reverts commit 08579ffa11.
 2012-06-17 14:27:32 -07:00
Matei Zaharia 0e84d620e1 Revert "Various fixes to get unit tests running. In particular, shut down" 
This reverts commit 2893b30550.
 2012-06-17 14:27:30 -07:00