ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
818 commits 6 branches 181 tags 397 MiB
Commit graph

818 commits

This branch
This branch
All branches
Author SHA1 Message Date
Shivaram Venkataraman 5975d2ee3b Fix SizeEstimator tests to work with String classes in JDK 6 and 7 2012-10-04 19:42:57 -07:00
Matei Zaharia 3d24281fbf Backport sampling fixes from dev (suggested by Henry Milner) 2012-09-29 21:55:52 -07:00
Matei Zaharia 915ab970b7 Make error reporting less scary if we can't look up UseCompressedOops 2012-09-28 14:52:37 -07:00
Matei Zaharia a417cd4d9d Look for Kryo registrator using context class loader 2012-09-26 14:14:17 -07:00
Matei Zaharia 2498f95199 Fixed GrowableAccumulatorParam.zero() to do a copy and to make sure it's 
empty instead of using the old initialValue
 2012-09-11 15:17:44 -07:00
Matei Zaharia 66e5362ce0 Merge pull request #175 from squito/collection_accumulators 
add accumulators for mutable collections, with correct typing!
 2012-09-11 14:29:10 -07:00
Imran Rashid 1490d09b5d Merge branch 'master' into collection_accumulators 
Conflicts:
	core/src/test/scala/spark/AccumulatorSuite.scala
 2012-09-11 11:20:56 -07:00
Imran Rashid f8f125eebb fix stupid typo and add documentation 2012-09-07 13:58:46 -07:00
Matei Zaharia 607b8fffcd End runJob with a SparkException when a Mesos task fails too many times 2012-08-31 11:40:12 -07:00
Matei Zaharia 62e5326af0 Wording 2012-08-30 08:37:43 -07:00
Matei Zaharia e8ac9221dc Update sbt build command to create JARs 2012-08-30 08:36:39 -07:00
Matei Zaharia 84bf7924d6 Made region used by spark-ec2 configurable. 2012-08-28 22:40:48 -07:00
Matei Zaharia 79c82b6cfd Merge pull request #173 from squito/accum_localValue 
make accumulator.localValue public, add tests
 2012-08-22 00:11:21 -07:00
Imran Rashid 4d2efe9555 change tests to show utility of localValue 2012-08-20 15:17:31 -07:00
Imran Rashid 823878c77f add accumulators for mutable collections, with correct typing! 2012-08-17 15:52:42 -07:00
Imran Rashid 206a3833ce make accumulator.localValue public, add tests 2012-08-14 14:08:22 -07:00
Matei Zaharia 680df96c43 Merge pull request #169 from shivaram/master 
Changes to SizeEstimator more accurate
 2012-08-12 22:29:11 -07:00
Shivaram Venkataraman 2ee731211a Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-12 17:18:01 -07:00
Shivaram Venkataraman 54502238a2 Move object size and pointer size initialization into a function to enable unit-testing 2012-08-12 17:16:27 -07:00
Shivaram Venkataraman 73452cc649 Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-11 16:42:35 -07:00
Shivaram Venkataraman 64b8fd62f0 If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-11 16:40:33 -07:00
Shivaram Venkataraman c0e773aa01 Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-11 14:38:05 -07:00
Shivaram Venkataraman f2475ca95a Add link to Java wiki which specifies what changes with compressed oops 2012-08-11 02:34:20 -07:00
Shivaram Venkataraman 980585b220 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-11 02:18:39 -07:00
Matei Zaharia 1c5ae3edf2 Merge pull request #151 from dennybritz/fix/examples_jar 
Examples ship to to cluster
 2012-08-04 17:39:41 -07:00
Denny 38d86d2616 updated readme 2012-08-04 16:58:47 -07:00
Denny 48cac4171c Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR 2012-08-04 16:56:32 -07:00
Denny 63c2020f93 Merge branch 'master' into fix/examples_jar 2012-08-04 16:55:18 -07:00
Matei Zaharia abca699378 Made range partition balance tests more aggressive. 
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
 2012-08-03 16:44:17 -04:00
Matei Zaharia 6da2bcdba1 Added a unit test for cross-partition balancing in sort, and changes to 
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
 2012-08-03 16:37:35 -04:00
Matei Zaharia 508221b8e6 Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int) 2012-08-03 15:57:43 -04:00
Matei Zaharia c0d5bd6553 Merge pull request #164 from HarveyFeng/master 
Bug fix in RangePartitioner for partitioning when sorting in descending order.
 2012-08-03 12:27:17 -07:00
Harvey 5ec13327d4 Fix for partitioning when sorting in descending order 2012-08-03 12:22:07 -07:00
Matei Zaharia 4a9c58913d Merge pull request #157 from paulcavallaro/log-throwables 
Logging Throwables in Info and Debug
 2012-08-02 06:26:06 -07:00
Paul Cavallaro e3952f31de Logging Throwables in Info and Debug 
Logging Throwables in logInfo and logDebug instead of swallowing them.
 2012-07-30 13:41:09 -04:00
Matei Zaharia 1e2df26c33 Merge pull request #145 from squito/exp_accum 
add Accumulatable, add corresponding docs & tests for accumulators
 2012-07-26 17:25:45 -07:00
Imran Rashid 0384be3467 tasks cannot access value of accumulator 2012-07-26 12:38:51 -07:00
Matei Zaharia da4298135c Merge pull request #152 from dennybritz/fix/testbeforeafter 
Always destroy SparkContext in after block for the unit tests.
 2012-07-23 16:13:24 -07:00
Denny 5656dcdfe5 Stlystic changes 2012-07-23 10:36:52 -07:00
Matei Zaharia 840e1b21e4 Merge branch 'master' of github.com:mesos/spark 2012-07-21 21:58:40 -07:00
Matei Zaharia 6f44c0db74 Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-21 21:58:28 -07:00
Matei Zaharia d1759c0290 Merge pull request #149 from dennybritz/serfix 
Instantiating custom serializer using user's classpath
 2012-07-21 21:54:50 -07:00
Matei Zaharia 5122f11b05 Use full package name in import 2012-07-21 21:53:38 -07:00
Denny 5559608e6f Always destroy SparkContext in after block for the unit tests. 2012-07-18 13:09:50 -07:00
Denny e4dbaf653f syntax errors 2012-07-18 12:18:00 -07:00
Denny 1d98884548 Use extended constructor in the examples. 2012-07-18 11:46:03 -07:00
Denny 2132c541f0 Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader. 2012-07-17 14:05:26 -07:00
Denny 2b84b50a85 Use Context classloader for Serializer class 2012-07-17 13:55:23 -07:00
Imran Rashid 7f43ba7ffa one more minor cleanup to scaladoc 2012-07-16 18:26:48 -07:00
Imran Rashid 913d42c6a0 fix up scaladoc, naming of type parameters 2012-07-16 18:25:15 -07:00