ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1361 commits 6 branches 181 tags 397 MiB
Commit graph

1361 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury c74455f309 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-28 14:56:57 -07:00
Matei Zaharia bf2e9cb08e Fault tolerance and block store fixes discovered through streaming tests. 2012-08-27 23:07:50 -07:00
Matei Zaharia 291abc2c28 Merge pull request #181 from rxin/dev 
Removed the deserialization cache for ShuffleMapTask
 2012-08-27 22:38:22 -07:00
Reynold Xin 3a6a95dc24 Removed the deserialization cache for ShuffleMapTask because it was 
causing concurrency problems (some variables in Shark get set to null).
The cost of task deserialization on slaves is trivial compared with the
execution time of the task anyway.
 2012-08-27 22:33:15 -07:00
Matei Zaharia 51453eb87b Merge pull request #179 from JoshRosen/fix/sparklr-caching 
Cache points in SparkLR example
 2012-08-26 15:32:50 -07:00
Josh Rosen 566feafe1d Cache points in SparkLR example. 2012-08-26 15:24:43 -07:00
Mosharaf Chowdhury edd1a740a6 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-23 20:43:27 -07:00
Matei Zaharia 2c16ae36d7 Set log level in tests to WARN 2012-08-23 20:38:14 -07:00
Matei Zaharia deedb9e7b7 Fix further issues with tests and broadcast. 
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
 2012-08-23 20:31:49 -07:00
Mosharaf Chowdhury 3b1f5480a4 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-23 20:16:50 -07:00
Matei Zaharia 59b831b9d1 Fixed test failures due to broadcast not stopping correctly 2012-08-23 19:59:55 -07:00
Matei Zaharia 7310a6f499 Merge pull request #147 from mosharaf/dev 
Broadcast refactoring/cleaning up
 2012-08-23 19:38:28 -07:00
Mosharaf Chowdhury 995ad6ba36 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-23 09:51:38 -07:00
Matei Zaharia 79c82b6cfd Merge pull request #173 from squito/accum_localValue 
make accumulator.localValue public, add tests
 2012-08-22 00:11:21 -07:00
Imran Rashid 4d2efe9555 change tests to show utility of localValue 2012-08-20 15:17:31 -07:00
Matei Zaharia 25a6a39e6d Added other SparkContext constructors to JavaSparkContext 2012-08-19 18:59:16 -07:00
Imran Rashid 823878c77f add accumulators for mutable collections, with correct typing! 2012-08-17 15:52:42 -07:00
Imran Rashid 206a3833ce make accumulator.localValue public, add tests 2012-08-14 14:08:22 -07:00
Matei Zaharia 9a0c128fec Merge pull request #172 from dennybritz/dev 
Rsync root directory in EC2 script
 2012-08-14 13:05:22 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 942e604c62 Merge pull request #171 from shivaram/for-size-estimator-pull 
Size estimator changes for dev
 2012-08-13 15:29:40 -07:00
Shivaram Venkataraman 0f4fbb057b Change BlockManagerSuite test cases to use a deterministic size estimator and 
update the results to match the new estimates
 2012-08-13 13:32:23 -07:00
Shivaram Venkataraman 22ba3a3f77 Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-13 13:32:10 -07:00
Shivaram Venkataraman 1f68c4b03b Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-13 13:31:54 -07:00
Shivaram Venkataraman 1ea269110c Move object size and pointer size initialization into a function to enable unit-testing 2012-08-13 13:31:45 -07:00
Shivaram Venkataraman 44661df9cc If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-13 13:31:39 -07:00
Shivaram Venkataraman 0dd8fe73ba Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-13 13:31:29 -07:00
Shivaram Venkataraman 80104ce1da Add link to Java wiki which specifies what changes with compressed oops 2012-08-13 13:31:21 -07:00
Shivaram Venkataraman 00ab5490b3 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-13 13:31:11 -07:00
Matei Zaharia 680df96c43 Merge pull request #169 from shivaram/master 
Changes to SizeEstimator more accurate
 2012-08-12 22:29:11 -07:00
Shivaram Venkataraman 2ee731211a Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-12 17:18:01 -07:00
Shivaram Venkataraman 54502238a2 Move object size and pointer size initialization into a function to enable unit-testing 2012-08-12 17:16:27 -07:00
Matei Zaharia 6ae3c375a9 Renamed apply() to call() in Java API and allowed it to throw Exceptions 2012-08-12 23:10:19 +02:00
Matei Zaharia 0141879c40 Use Promises instead of having a Future wait on a thread in 
ConnectionManager.
 2012-08-12 22:16:32 +02:00
Matei Zaharia 845a870242 Return remotely fetched blocks in a pipelined fashion from BlockManager 2012-08-12 20:01:38 +02:00
Matei Zaharia e17ed9a21d Switch to Akka futures in connection manager. 
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
 2012-08-12 19:40:37 +02:00
Matei Zaharia ad8a7612a4 Changed multi-get method in BlockManager to return an iterator 2012-08-12 19:18:01 +02:00
Shivaram Venkataraman 73452cc649 Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-11 16:42:35 -07:00
Shivaram Venkataraman 64b8fd62f0 If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-11 16:40:33 -07:00
Shivaram Venkataraman c0e773aa01 Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-11 14:38:05 -07:00
Shivaram Venkataraman f2475ca95a Add link to Java wiki which specifies what changes with compressed oops 2012-08-11 02:34:20 -07:00
Shivaram Venkataraman 980585b220 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-11 02:18:39 -07:00
Matei Zaharia 3c94e5c188 Merge pull request #168 from shivaram/dev 
Use JavaConversion to get a scala iterator
 2012-08-10 00:57:33 -07:00
Matei Zaharia e463e7a333 Merge pull request #167 from JoshRosen/piped-rdd-fixes 
Detect non-zero exit status from PipedRDD process
 2012-08-10 00:56:42 -07:00
Josh Rosen 59c22fb444 Print exit status in PipedRDD failure exception. 2012-08-10 00:33:56 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Shivaram Venkataraman 1803cce692 Use an implicit conversion to get the scala iterator 2012-08-08 14:31:04 -07:00
Shivaram Venkataraman 674fcf56bf Use JavaConversion to get a scala iterator 2012-08-08 14:10:23 -07:00
Matei Zaharia bec4d362c8 Merge pull request #166 from shivaram/dev 
Avoid a copy in ShuffleMapTask
 2012-08-08 09:11:19 -07:00