ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
920 commits 6 branches 181 tags 397 MiB
Commit graph

920 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 29e83f39e9 Fix replication with MEMORY_ONLY_DESER_2 2012-08-26 18:16:25 -07:00
Matei Zaharia 57796b183e Code style 2012-08-26 17:25:22 -07:00
Matei Zaharia 22b1a20e61 Made Time and Interval immutable 2012-08-26 17:04:34 -07:00
Matei Zaharia 23a29b6d19 Merge branch 'dev' of github.com:radlab/spark into dev 2012-08-26 16:45:37 -07:00
Matei Zaharia b120e24fe0 Add equals and hashCode to Time 2012-08-26 16:45:14 -07:00
root b08ff710af Added sliding word count, and some fixes to reduce window DStream 2012-08-26 23:40:50 +00:00
Matei Zaharia 06ef7c3d1b Less debug info 2012-08-26 16:29:20 -07:00
Matei Zaharia ad6537321e Make Time serializable 2012-08-26 16:27:23 -07:00
Matei Zaharia 741899b21e Fix sendMessageReliablySync 2012-08-26 16:26:06 -07:00
Matei Zaharia e7a5cbb543 Reduce log4j verbosity for streaming 2012-08-24 16:45:01 -07:00
Matei Zaharia 091b1438f5 Fix WordCount job name 2012-08-24 16:43:59 -07:00
Matei Zaharia 5a8015d2db Merge remote-tracking branch 'public/dev' into dev 2012-08-24 16:11:44 -07:00
Matei Zaharia 2c16ae36d7 Set log level in tests to WARN 2012-08-23 20:38:14 -07:00
Matei Zaharia deedb9e7b7 Fix further issues with tests and broadcast. 
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
 2012-08-23 20:31:49 -07:00
Matei Zaharia 59b831b9d1 Fixed test failures due to broadcast not stopping correctly 2012-08-23 19:59:55 -07:00
Matei Zaharia 7310a6f499 Merge pull request #147 from mosharaf/dev 
Broadcast refactoring/cleaning up
 2012-08-23 19:38:28 -07:00
Matei Zaharia 25a6a39e6d Added other SparkContext constructors to JavaSparkContext 2012-08-19 18:59:16 -07:00
Matei Zaharia 9a0c128fec Merge pull request #172 from dennybritz/dev 
Rsync root directory in EC2 script
 2012-08-14 13:05:22 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 942e604c62 Merge pull request #171 from shivaram/for-size-estimator-pull 
Size estimator changes for dev
 2012-08-13 15:29:40 -07:00
Shivaram Venkataraman 0f4fbb057b Change BlockManagerSuite test cases to use a deterministic size estimator and 
update the results to match the new estimates
 2012-08-13 13:32:23 -07:00
Shivaram Venkataraman 22ba3a3f77 Add test-cases for 32-bit and no-compressed oops scenarios. 2012-08-13 13:32:10 -07:00
Shivaram Venkataraman 1f68c4b03b Update test cases to match the new size estimates. Uses 64-bit and compressed 
oops setting to get deterministic results
 2012-08-13 13:31:54 -07:00
Shivaram Venkataraman 1ea269110c Move object size and pointer size initialization into a function to enable unit-testing 2012-08-13 13:31:45 -07:00
Shivaram Venkataraman 44661df9cc If spark.test.useCompressedOops is set, use that to infer compressed oops 
setting. This is useful to get a deterministic test case
 2012-08-13 13:31:39 -07:00
Shivaram Venkataraman 0dd8fe73ba Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not 2012-08-13 13:31:29 -07:00
Shivaram Venkataraman 80104ce1da Add link to Java wiki which specifies what changes with compressed oops 2012-08-13 13:31:21 -07:00
Shivaram Venkataraman 00ab5490b3 Changes to make size estimator more accurate. Fixes object size, pointer size 
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
 2012-08-13 13:31:11 -07:00
Matei Zaharia 6ae3c375a9 Renamed apply() to call() in Java API and allowed it to throw Exceptions 2012-08-12 23:10:19 +02:00
Matei Zaharia 0141879c40 Use Promises instead of having a Future wait on a thread in 
ConnectionManager.
 2012-08-12 22:16:32 +02:00
Matei Zaharia 845a870242 Return remotely fetched blocks in a pipelined fashion from BlockManager 2012-08-12 20:01:38 +02:00
Matei Zaharia e17ed9a21d Switch to Akka futures in connection manager. 
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
 2012-08-12 19:40:37 +02:00
Matei Zaharia ad8a7612a4 Changed multi-get method in BlockManager to return an iterator 2012-08-12 19:18:01 +02:00
Matei Zaharia 3c94e5c188 Merge pull request #168 from shivaram/dev 
Use JavaConversion to get a scala iterator
 2012-08-10 00:57:33 -07:00
Matei Zaharia e463e7a333 Merge pull request #167 from JoshRosen/piped-rdd-fixes 
Detect non-zero exit status from PipedRDD process
 2012-08-10 00:56:42 -07:00
Josh Rosen 59c22fb444 Print exit status in PipedRDD failure exception. 2012-08-10 00:33:56 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Shivaram Venkataraman 1803cce692 Use an implicit conversion to get the scala iterator 2012-08-08 14:31:04 -07:00
Shivaram Venkataraman 674fcf56bf Use JavaConversion to get a scala iterator 2012-08-08 14:10:23 -07:00
Matei Zaharia bec4d362c8 Merge pull request #166 from shivaram/dev 
Avoid a copy in ShuffleMapTask
 2012-08-08 09:11:19 -07:00
Shivaram Venkataraman f4aaec7a48 Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the 
block manager.
 2012-08-08 00:47:02 -07:00
Tathagata Das cae894ee7a Added new Clock interface that is used by RecurringTimer to scheduler events on system time or manually-configured time. 2012-08-06 14:52:46 -07:00
Mosharaf Chowdhury d821dd3ccc BroadcastManager is a class now (replaced Braodcast object) 2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury b4804119f9 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-04 20:42:12 -07:00
Matei Zaharia 88b016db2a Merge pull request #160 from dennybritz/clusterscripts 
Standalone cluster scripts
 2012-08-04 17:45:20 -07:00
Denny 8fb955fd40 Add Apache license to non-trivial scripts taken from Hadoop. 2012-08-04 17:04:33 -07:00
Mosharaf Chowdhury 1b0534af8f Merge branch 'dev' into bc-bm 2012-08-04 00:30:08 -07:00
Mosharaf Chowdhury d11b457e67 Merge remote-tracking branch 'upstream/dev' into dev 2012-08-04 00:28:10 -07:00
Mosharaf Chowdhury 24b7eb872c Bug fixed. Broadcast now works with BlockManager. 2012-08-04 00:27:28 -07:00