root
7b59943d79
Merge branch 'dev' of github.com:radlab/spark into dev
2012-08-27 01:57:12 +00:00
Matei Zaharia
26dfd20c9a
Detect disconnected slaves in StandaloneScheduler
2012-08-26 18:56:56 -07:00
root
b78c5ae803
Merge branch 'dev' of github.com:radlab/spark into dev
2012-08-27 01:16:39 +00:00
Matei Zaharia
29e83f39e9
Fix replication with MEMORY_ONLY_DESER_2
2012-08-26 18:16:25 -07:00
root
9de1c3abf9
Tweaks to WordCount2
2012-08-27 00:57:00 +00:00
Matei Zaharia
57796b183e
Code style
2012-08-26 17:25:22 -07:00
Matei Zaharia
22b1a20e61
Made Time and Interval immutable
2012-08-26 17:04:34 -07:00
Matei Zaharia
23a29b6d19
Merge branch 'dev' of github.com:radlab/spark into dev
2012-08-26 16:45:37 -07:00
Matei Zaharia
b120e24fe0
Add equals and hashCode to Time
2012-08-26 16:45:14 -07:00
root
b08ff710af
Added sliding word count, and some fixes to reduce window DStream
2012-08-26 23:40:50 +00:00
Matei Zaharia
06ef7c3d1b
Less debug info
2012-08-26 16:29:20 -07:00
Matei Zaharia
ad6537321e
Make Time serializable
2012-08-26 16:27:23 -07:00
Matei Zaharia
741899b21e
Fix sendMessageReliablySync
2012-08-26 16:26:06 -07:00
Matei Zaharia
51453eb87b
Merge pull request #179 from JoshRosen/fix/sparklr-caching
...
Cache points in SparkLR example
2012-08-26 15:32:50 -07:00
Josh Rosen
566feafe1d
Cache points in SparkLR example.
2012-08-26 15:24:43 -07:00
Matei Zaharia
e7a5cbb543
Reduce log4j verbosity for streaming
2012-08-24 16:45:01 -07:00
Matei Zaharia
091b1438f5
Fix WordCount job name
2012-08-24 16:43:59 -07:00
Matei Zaharia
5a8015d2db
Merge remote-tracking branch 'public/dev' into dev
2012-08-24 16:11:44 -07:00
Mosharaf Chowdhury
edd1a740a6
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 20:43:27 -07:00
Matei Zaharia
2c16ae36d7
Set log level in tests to WARN
2012-08-23 20:38:14 -07:00
Matei Zaharia
deedb9e7b7
Fix further issues with tests and broadcast.
...
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
2012-08-23 20:31:49 -07:00
Mosharaf Chowdhury
3b1f5480a4
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 20:16:50 -07:00
Matei Zaharia
59b831b9d1
Fixed test failures due to broadcast not stopping correctly
2012-08-23 19:59:55 -07:00
Matei Zaharia
7310a6f499
Merge pull request #147 from mosharaf/dev
...
Broadcast refactoring/cleaning up
2012-08-23 19:38:28 -07:00
Mosharaf Chowdhury
995ad6ba36
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 09:51:38 -07:00
Matei Zaharia
79c82b6cfd
Merge pull request #173 from squito/accum_localValue
...
make accumulator.localValue public, add tests
2012-08-22 00:11:21 -07:00
Imran Rashid
4d2efe9555
change tests to show utility of localValue
2012-08-20 15:17:31 -07:00
Matei Zaharia
25a6a39e6d
Added other SparkContext constructors to JavaSparkContext
2012-08-19 18:59:16 -07:00
Imran Rashid
823878c77f
add accumulators for mutable collections, with correct typing!
2012-08-17 15:52:42 -07:00
Imran Rashid
206a3833ce
make accumulator.localValue public, add tests
2012-08-14 14:08:22 -07:00
Matei Zaharia
9a0c128fec
Merge pull request #172 from dennybritz/dev
...
Rsync root directory in EC2 script
2012-08-14 13:05:22 -07:00
Denny
8dc7242544
Use root login in standalone AMI
2012-08-14 10:18:24 -07:00
Denny
7152c7c12d
rsync root directory in EC2 script
2012-08-14 09:26:47 -07:00
Matei Zaharia
942e604c62
Merge pull request #171 from shivaram/for-size-estimator-pull
...
Size estimator changes for dev
2012-08-13 15:29:40 -07:00
Shivaram Venkataraman
0f4fbb057b
Change BlockManagerSuite test cases to use a deterministic size estimator and
...
update the results to match the new estimates
2012-08-13 13:32:23 -07:00
Shivaram Venkataraman
22ba3a3f77
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-13 13:32:10 -07:00
Shivaram Venkataraman
1f68c4b03b
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-13 13:31:54 -07:00
Shivaram Venkataraman
1ea269110c
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-13 13:31:45 -07:00
Shivaram Venkataraman
44661df9cc
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-13 13:31:39 -07:00
Shivaram Venkataraman
0dd8fe73ba
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-13 13:31:29 -07:00
Shivaram Venkataraman
80104ce1da
Add link to Java wiki which specifies what changes with compressed oops
2012-08-13 13:31:21 -07:00
Shivaram Venkataraman
00ab5490b3
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-13 13:31:11 -07:00
Matei Zaharia
680df96c43
Merge pull request #169 from shivaram/master
...
Changes to SizeEstimator more accurate
2012-08-12 22:29:11 -07:00
Shivaram Venkataraman
2ee731211a
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-12 17:18:01 -07:00
Shivaram Venkataraman
54502238a2
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-12 17:16:27 -07:00
Matei Zaharia
6ae3c375a9
Renamed apply() to call() in Java API and allowed it to throw Exceptions
2012-08-12 23:10:19 +02:00
Matei Zaharia
0141879c40
Use Promises instead of having a Future wait on a thread in
...
ConnectionManager.
2012-08-12 22:16:32 +02:00
Matei Zaharia
845a870242
Return remotely fetched blocks in a pipelined fashion from BlockManager
2012-08-12 20:01:38 +02:00
Matei Zaharia
e17ed9a21d
Switch to Akka futures in connection manager.
...
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
2012-08-12 19:40:37 +02:00
Matei Zaharia
ad8a7612a4
Changed multi-get method in BlockManager to return an iterator
2012-08-12 19:18:01 +02:00
