Shivaram Venkataraman
73452cc649
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-11 16:42:35 -07:00
Shivaram Venkataraman
64b8fd62f0
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-11 16:40:33 -07:00
Shivaram Venkataraman
c0e773aa01
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-11 14:38:05 -07:00
Shivaram Venkataraman
f2475ca95a
Add link to Java wiki which specifies what changes with compressed oops
2012-08-11 02:34:20 -07:00
Shivaram Venkataraman
980585b220
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-11 02:18:39 -07:00
Matei Zaharia
abca699378
Made range partition balance tests more aggressive.
...
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
2012-08-03 16:44:17 -04:00
Matei Zaharia
6da2bcdba1
Added a unit test for cross-partition balancing in sort, and changes to
...
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
2012-08-03 16:37:35 -04:00
Matei Zaharia
508221b8e6
Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int)
2012-08-03 15:57:43 -04:00
Harvey
5ec13327d4
Fix for partitioning when sorting in descending order
2012-08-03 12:22:07 -07:00
Paul Cavallaro
e3952f31de
Logging Throwables in Info and Debug
...
Logging Throwables in logInfo and logDebug instead of swallowing them.
2012-07-30 13:41:09 -04:00
Matei Zaharia
1e2df26c33
Merge pull request #145 from squito/exp_accum
...
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-26 17:25:45 -07:00
Imran Rashid
0384be3467
tasks cannot access value of accumulator
2012-07-26 12:38:51 -07:00
Matei Zaharia
da4298135c
Merge pull request #152 from dennybritz/fix/testbeforeafter
...
Always destroy SparkContext in after block for the unit tests.
2012-07-23 16:13:24 -07:00
Denny
5656dcdfe5
Stlystic changes
2012-07-23 10:36:52 -07:00
Matei Zaharia
840e1b21e4
Merge branch 'master' of github.com:mesos/spark
2012-07-21 21:58:40 -07:00
Matei Zaharia
6f44c0db74
Fix a bug where an input path was added to a Hadoop job configuration twice
2012-07-21 21:58:28 -07:00
Matei Zaharia
d1759c0290
Merge pull request #149 from dennybritz/serfix
...
Instantiating custom serializer using user's classpath
2012-07-21 21:54:50 -07:00
Matei Zaharia
5122f11b05
Use full package name in import
2012-07-21 21:53:38 -07:00
Denny
5559608e6f
Always destroy SparkContext in after block for the unit tests.
2012-07-18 13:09:50 -07:00
Denny
2132c541f0
Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader.
2012-07-17 14:05:26 -07:00
Denny
2b84b50a85
Use Context classloader for Serializer class
2012-07-17 13:55:23 -07:00
Imran Rashid
7f43ba7ffa
one more minor cleanup to scaladoc
2012-07-16 18:26:48 -07:00
Imran Rashid
913d42c6a0
fix up scaladoc, naming of type parameters
2012-07-16 18:25:15 -07:00
Imran Rashid
85940a7d71
rename addToAccum to addAccumulator
2012-07-16 18:17:13 -07:00
Matei Zaharia
a33ca6949c
Merge branch 'master' of github.com:mesos/spark
2012-07-12 18:38:20 -07:00
Matei Zaharia
452330efb4
Allow null keys in Spark's reduce and group by
2012-07-12 18:37:37 -07:00
Imran Rashid
86024ca74d
add some functionality to Vector, delete copy in AccumulatorSuite
2012-07-12 13:08:03 -07:00
Imran Rashid
42ce879486
move Vector class into core and spark.util package
2012-07-12 13:08:03 -07:00
Imran Rashid
13cc72cfb5
Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=)
2012-07-12 12:40:10 -07:00
Imran Rashid
73935629a1
improve scaladoc
2012-07-12 09:58:06 -07:00
Imran Rashid
30480e6dae
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-12 09:37:42 -07:00
Matei Zaharia
4259d37f84
Formatting
2012-07-10 11:16:34 -07:00
Matei Zaharia
6980b67557
Added more methods for loading/saving with new Hadoop API
2012-07-10 11:11:35 -07:00
Matei Zaharia
800fcbfbca
Revert "Fixed HttpBroadcast to work with this branch's Serializer."
...
This reverts commit b3eeac55b8
2012-06-17 14:29:39 -07:00
Matei Zaharia
94d77f83d3
Revert "Merge branch 'master' into dev"
...
This reverts commit f58da6164e4449eb9783
2012-06-17 14:27:45 -07:00
Matei Zaharia
0e84d620e1
Revert "Various fixes to get unit tests running. In particular, shut down"
...
This reverts commit 2893b30550
2012-06-17 14:27:30 -07:00
Matei Zaharia
4749ec063c
Revert "Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase."
...
This reverts commit 40536e3668
2012-06-17 14:27:18 -07:00
Tathagata Das
40536e3668
Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase.
2012-06-17 13:28:41 -07:00
Matei Zaharia
2893b30550
Various fixes to get unit tests running. In particular, shut down
...
ConnectionManager and DAGScheduler properly, plus a fix to
LocalScheduler that was not merged in from 0.5 and was actually caught
by one of the tests.
2012-06-17 00:28:45 -07:00
Matei Zaharia
b3eeac55b8
Fixed HttpBroadcast to work with this branch's Serializer.
2012-06-15 23:54:38 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Tathagata Das
5f54bdf98b
Added shutdown for akka to SparkContext.stop(). Helps a little, but many testsuites still fail.
2012-06-13 20:49:00 -04:00
Tathagata Das
c6156da9e2
Multiple bug fixes to pass the testsuites ShuffleSuite and BlockManagerSuite.
2012-06-13 16:26:49 -04:00
Matei Zaharia
879bc0bece
Merge branch 'master' into mesos-0.9
2012-06-09 16:24:16 -07:00
Matei Zaharia
4b05798c06
Further bug fix to HttpBroadcast
2012-06-09 16:24:03 -07:00
Matei Zaharia
587a16a7ef
Merge branch 'master' into mesos-0.9
2012-06-09 16:17:07 -07:00
Matei Zaharia
8ed662862e
Bug fix to HttpBroadcast
2012-06-09 16:16:55 -07:00
Matei Zaharia
2fd9f994ae
Merge branch 'master' into mesos-0.9
2012-06-09 15:58:35 -07:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
