Shivaram Venkataraman
0dd8fe73ba
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-13 13:31:29 -07:00
Shivaram Venkataraman
80104ce1da
Add link to Java wiki which specifies what changes with compressed oops
2012-08-13 13:31:21 -07:00
Shivaram Venkataraman
00ab5490b3
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-13 13:31:11 -07:00
Matei Zaharia
680df96c43
Merge pull request #169 from shivaram/master
...
Changes to SizeEstimator more accurate
2012-08-12 22:29:11 -07:00
Shivaram Venkataraman
2ee731211a
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-12 17:18:01 -07:00
Shivaram Venkataraman
54502238a2
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-12 17:16:27 -07:00
Matei Zaharia
6ae3c375a9
Renamed apply() to call() in Java API and allowed it to throw Exceptions
2012-08-12 23:10:19 +02:00
Matei Zaharia
0141879c40
Use Promises instead of having a Future wait on a thread in
...
ConnectionManager.
2012-08-12 22:16:32 +02:00
Matei Zaharia
845a870242
Return remotely fetched blocks in a pipelined fashion from BlockManager
2012-08-12 20:01:38 +02:00
Matei Zaharia
e17ed9a21d
Switch to Akka futures in connection manager.
...
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
2012-08-12 19:40:37 +02:00
Matei Zaharia
ad8a7612a4
Changed multi-get method in BlockManager to return an iterator
2012-08-12 19:18:01 +02:00
Shivaram Venkataraman
73452cc649
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-11 16:42:35 -07:00
Shivaram Venkataraman
64b8fd62f0
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-11 16:40:33 -07:00
Shivaram Venkataraman
c0e773aa01
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-11 14:38:05 -07:00
Shivaram Venkataraman
f2475ca95a
Add link to Java wiki which specifies what changes with compressed oops
2012-08-11 02:34:20 -07:00
Shivaram Venkataraman
980585b220
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-11 02:18:39 -07:00
Matei Zaharia
3c94e5c188
Merge pull request #168 from shivaram/dev
...
Use JavaConversion to get a scala iterator
2012-08-10 00:57:33 -07:00
Matei Zaharia
e463e7a333
Merge pull request #167 from JoshRosen/piped-rdd-fixes
...
Detect non-zero exit status from PipedRDD process
2012-08-10 00:56:42 -07:00
Josh Rosen
59c22fb444
Print exit status in PipedRDD failure exception.
2012-08-10 00:33:56 -07:00
Matei Zaharia
8069bd5b41
Removed separate launcher for EC2 standalone cluster
2012-08-09 22:45:24 +02:00
Shivaram Venkataraman
1803cce692
Use an implicit conversion to get the scala iterator
2012-08-08 14:31:04 -07:00
Shivaram Venkataraman
674fcf56bf
Use JavaConversion to get a scala iterator
2012-08-08 14:10:23 -07:00
Matei Zaharia
bec4d362c8
Merge pull request #166 from shivaram/dev
...
Avoid a copy in ShuffleMapTask
2012-08-08 09:11:19 -07:00
Shivaram Venkataraman
f4aaec7a48
Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the
...
block manager.
2012-08-08 00:47:02 -07:00
Tathagata Das
cae894ee7a
Added new Clock interface that is used by RecurringTimer to scheduler events on system time or manually-configured time.
2012-08-06 14:52:46 -07:00
Mosharaf Chowdhury
d821dd3ccc
BroadcastManager is a class now (replaced Braodcast object)
2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury
b4804119f9
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 20:42:12 -07:00
Matei Zaharia
88b016db2a
Merge pull request #160 from dennybritz/clusterscripts
...
Standalone cluster scripts
2012-08-04 17:45:20 -07:00
Matei Zaharia
1c5ae3edf2
Merge pull request #151 from dennybritz/fix/examples_jar
...
Examples ship to to cluster
2012-08-04 17:39:41 -07:00
Denny
8fb955fd40
Add Apache license to non-trivial scripts taken from Hadoop.
2012-08-04 17:04:33 -07:00
Denny
38d86d2616
updated readme
2012-08-04 16:58:47 -07:00
Denny
48cac4171c
Renamed EXAMPLES_JAR to SPARK_EXAMPLES_JAR
2012-08-04 16:56:32 -07:00
Denny
63c2020f93
Merge branch 'master' into fix/examples_jar
2012-08-04 16:55:18 -07:00
Mosharaf Chowdhury
1b0534af8f
Merge branch 'dev' into bc-bm
2012-08-04 00:30:08 -07:00
Mosharaf Chowdhury
d11b457e67
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 00:28:10 -07:00
Mosharaf Chowdhury
24b7eb872c
Bug fixed. Broadcast now works with BlockManager.
2012-08-04 00:27:28 -07:00
Matei Zaharia
5cefda9984
Merge pull request #165 from shivaram/dev
...
Fix test checkpoint to reuse spark context defined in the class
2012-08-03 19:17:50 -07:00
Shivaram Venkataraman
ce3444d2cb
Fix testcheckpoint to reuse spark context defined in the class
2012-08-03 18:52:26 -07:00
Matei Zaharia
62898b631f
Made range partition balance tests more aggressive.
...
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
2012-08-03 16:46:48 -04:00
Matei Zaharia
abca699378
Made range partition balance tests more aggressive.
...
This is because we pull out such a large sample (10x the number of
partitions) that we should expect pretty good balance. The tests are
also deterministic so there's no worry about them failing irreproducibly.
2012-08-03 16:44:17 -04:00
Matei Zaharia
6601a6212b
Added a unit test for cross-partition balancing in sort, and changes to
...
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
2012-08-03 16:40:45 -04:00
Harvey
1170de3757
Fix for partitioning when sorting in descending order
2012-08-03 16:40:38 -04:00
Paul Cavallaro
d05c0f97ca
Logging Throwables in Info and Debug
...
Logging Throwables in logInfo and logDebug instead of swallowing them.
Conflicts:
core/src/main/scala/spark/Logging.scala
2012-08-03 16:40:21 -04:00
Matei Zaharia
6da2bcdba1
Added a unit test for cross-partition balancing in sort, and changes to
...
RangePartitioner to make it pass. It turns out that the first partition
was always kind of small due to how we picked partition boundaries.
2012-08-03 16:37:35 -04:00
Matei Zaharia
508221b8e6
Fix to #154 (CacheTracker trying to cast a broadcast variable's ID to int)
2012-08-03 15:57:43 -04:00
Matei Zaharia
c0d5bd6553
Merge pull request #164 from HarveyFeng/master
...
Bug fix in RangePartitioner for partitioning when sorting in descending order.
2012-08-03 12:27:17 -07:00
Harvey
5ec13327d4
Fix for partitioning when sorting in descending order
2012-08-03 12:22:07 -07:00
Denny
c90c9ec208
Read config variables before to get the master port
2012-08-02 16:12:40 -07:00
Denny
0008994044
merged dev branch
2012-08-02 16:00:33 -07:00
Denny
53008c2d8a
Settings variables and bugfix for stop script.
2012-08-02 15:59:39 -07:00