ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
988 commits 6 branches 181 tags 397 MiB
Commit graph

988 commits

This branch
This branch
All branches
Author SHA1 Message Date
Tathagata Das 43e66146f7 Merge branch 'dev' of github.com/radlab/spark into dev 2012-08-28 13:51:05 -07:00
Tathagata Das b5b93a621c Added capabllity to take streaming input from network. Renamed SparkStreamContext to StreamingContext. 2012-08-28 12:35:19 -07:00
Matei Zaharia 17af2df0cd Log levels 2012-08-27 23:07:32 -07:00
Matei Zaharia a0b34d826a Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 22:49:52 -07:00
Matei Zaharia b4a2214218 More fault tolerance fixes to catch lost tasks 2012-08-27 22:49:29 -07:00
root e2cf197a0a Made WordCount2 even more configurable 2012-08-27 03:34:15 +00:00
root 9635823947 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 03:08:25 +00:00
Matei Zaharia b914cd0dfa Serialize generation correctly in ShuffleMapTask 2012-08-26 20:07:59 -07:00
root 20f6b0cfc9 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 03:01:03 +00:00
Matei Zaharia 69c2ab0408 logging 2012-08-26 20:00:58 -07:00
root 89c5c03035 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 02:53:07 +00:00
Matei Zaharia 117e3f8c86 Fix a bug that was causing FetchFailedException not to be thrown 2012-08-26 19:52:56 -07:00
root beb6456442 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 02:37:49 +00:00
Matei Zaharia 3c9c44a8d3 More helpful log messages 2012-08-26 19:37:43 -07:00
root 7b59943d79 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 01:57:12 +00:00
Matei Zaharia 26dfd20c9a Detect disconnected slaves in StandaloneScheduler 2012-08-26 18:56:56 -07:00
root b78c5ae803 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-27 01:16:39 +00:00
Matei Zaharia 29e83f39e9 Fix replication with MEMORY_ONLY_DESER_2 2012-08-26 18:16:25 -07:00
root 9de1c3abf9 Tweaks to WordCount2 2012-08-27 00:57:00 +00:00
Matei Zaharia 57796b183e Code style 2012-08-26 17:25:22 -07:00
Matei Zaharia 22b1a20e61 Made Time and Interval immutable 2012-08-26 17:04:34 -07:00
Matei Zaharia 23a29b6d19 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-26 16:45:37 -07:00
Matei Zaharia b120e24fe0 Add equals and hashCode to Time 2012-08-26 16:45:14 -07:00
root b08ff710af Added sliding word count, and some fixes to reduce window DStream 2012-08-26 23:40:50 +00:00
Matei Zaharia 06ef7c3d1b Less debug info 2012-08-26 16:29:20 -07:00
Matei Zaharia ad6537321e Make Time serializable 2012-08-26 16:27:23 -07:00
Matei Zaharia 741899b21e Fix sendMessageReliablySync 2012-08-26 16:26:06 -07:00
Matei Zaharia e7a5cbb543 Reduce log4j verbosity for streaming 2012-08-24 16:45:01 -07:00
Matei Zaharia 091b1438f5 Fix WordCount job name 2012-08-24 16:43:59 -07:00
Matei Zaharia 5a8015d2db Merge remote-tracking branch 'public/dev' into dev 2012-08-24 16:11:44 -07:00
Matei Zaharia 2c16ae36d7 Set log level in tests to WARN 2012-08-23 20:38:14 -07:00
Matei Zaharia deedb9e7b7 Fix further issues with tests and broadcast. 
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
 2012-08-23 20:31:49 -07:00
Matei Zaharia 59b831b9d1 Fixed test failures due to broadcast not stopping correctly 2012-08-23 19:59:55 -07:00
Matei Zaharia 7310a6f499 Merge pull request #147 from mosharaf/dev 
Broadcast refactoring/cleaning up
 2012-08-23 19:38:28 -07:00
Matei Zaharia 25a6a39e6d Added other SparkContext constructors to JavaSparkContext 2012-08-19 18:59:16 -07:00
Matei Zaharia 9a0c128fec Merge pull request #172 from dennybritz/dev 
Rsync root directory in EC2 script
 2012-08-14 13:05:22 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 942e604c62 Merge pull request #171 from shivaram/for-size-estimator-pull 
Size estimator changes for dev
 2012-08-13 15:29:40 -07:00
Shivaram Venkataraman 0f4fbb057b Change BlockManagerSuite test cases to use a deterministic size estimator and 
update the results to match the new estimates
 2012-08-13 13:32:23 -07:00
Shivaram Venkataraman 22ba3a3f77 Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-13 13:32:10 -07:00
Shivaram Venkataraman 1f68c4b03b Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-13 13:31:54 -07:00
Shivaram Venkataraman 1ea269110c Move object size and pointer size initialization into a function to enable unit-testing 2012-08-13 13:31:45 -07:00
Shivaram Venkataraman 44661df9cc If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-13 13:31:39 -07:00
Shivaram Venkataraman 0dd8fe73ba Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-13 13:31:29 -07:00
Shivaram Venkataraman 80104ce1da Add link to Java wiki which specifies what changes with compressed oops 2012-08-13 13:31:21 -07:00
Shivaram Venkataraman 00ab5490b3 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-13 13:31:11 -07:00
Matei Zaharia 6ae3c375a9 Renamed apply() to call() in Java API and allowed it to throw Exceptions 2012-08-12 23:10:19 +02:00
Matei Zaharia 0141879c40 Use Promises instead of having a Future wait on a thread in 
ConnectionManager.
 2012-08-12 22:16:32 +02:00
Matei Zaharia 845a870242 Return remotely fetched blocks in a pipelined fashion from BlockManager 2012-08-12 20:01:38 +02:00