Tathagata Das
7c09ad0e04
Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage.
2012-09-04 19:11:49 -07:00
Tathagata Das
b8e9e8ea78
Merge branch 'dev' of github.com:radlab/spark into dev
2012-09-02 02:35:32 -07:00
root
ceabf71257
tweaks
2012-09-01 21:52:42 +00:00
root
6025889be0
More raw network receiver programs
2012-09-01 20:51:07 +00:00
Matei Zaharia
f84d2bbe55
Bug fixes to RateLimitedOutputStream
2012-09-01 00:31:15 -07:00
Matei Zaharia
44758aa8e2
First work towards a RawInputDStream and a sender program for it.
2012-09-01 00:17:59 -07:00
root
c42e7ac282
More block manager fixes
2012-09-01 04:31:11 +00:00
root
113277549c
Really fixed the replication-3 issue. The problem was a few buffers not being rewound.
2012-08-31 05:39:35 +00:00
Matei Zaharia
101ae493e2
Replicate serialized blocks properly, without sharing a ByteBuffer.
2012-08-30 22:24:14 -07:00
Matei Zaharia
1b3e3352eb
Deserialize multi-get results in the caller's thread. This fixes an
...
issue with shared buffers with the KryoSerializer.
2012-08-30 17:59:25 -07:00
root
c4366eb764
Fixes to ShuffleFetcher
2012-08-31 00:34:24 +00:00
Tathagata Das
4db3a96766
Made minor changes to reduce compilation errors in Eclipse. Twirl stuff still does not compile in Eclipse.
2012-08-29 13:04:01 -07:00
Matei Zaharia
17af2df0cd
Log levels
2012-08-27 23:07:32 -07:00
Matei Zaharia
b4a2214218
More fault tolerance fixes to catch lost tasks
2012-08-27 22:49:29 -07:00
Matei Zaharia
b914cd0dfa
Serialize generation correctly in ShuffleMapTask
2012-08-26 20:07:59 -07:00
Matei Zaharia
69c2ab0408
logging
2012-08-26 20:00:58 -07:00
Matei Zaharia
117e3f8c86
Fix a bug that was causing FetchFailedException not to be thrown
2012-08-26 19:52:56 -07:00
Matei Zaharia
3c9c44a8d3
More helpful log messages
2012-08-26 19:37:43 -07:00
Matei Zaharia
26dfd20c9a
Detect disconnected slaves in StandaloneScheduler
2012-08-26 18:56:56 -07:00
Matei Zaharia
29e83f39e9
Fix replication with MEMORY_ONLY_DESER_2
2012-08-26 18:16:25 -07:00
Matei Zaharia
06ef7c3d1b
Less debug info
2012-08-26 16:29:20 -07:00
Matei Zaharia
741899b21e
Fix sendMessageReliablySync
2012-08-26 16:26:06 -07:00
Matei Zaharia
5a8015d2db
Merge remote-tracking branch 'public/dev' into dev
2012-08-24 16:11:44 -07:00
Matei Zaharia
2c16ae36d7
Set log level in tests to WARN
2012-08-23 20:38:14 -07:00
Matei Zaharia
deedb9e7b7
Fix further issues with tests and broadcast.
...
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
2012-08-23 20:31:49 -07:00
Matei Zaharia
59b831b9d1
Fixed test failures due to broadcast not stopping correctly
2012-08-23 19:59:55 -07:00
Matei Zaharia
7310a6f499
Merge pull request #147 from mosharaf/dev
...
Broadcast refactoring/cleaning up
2012-08-23 19:38:28 -07:00
Matei Zaharia
25a6a39e6d
Added other SparkContext constructors to JavaSparkContext
2012-08-19 18:59:16 -07:00
Shivaram Venkataraman
0f4fbb057b
Change BlockManagerSuite test cases to use a deterministic size estimator and
...
update the results to match the new estimates
2012-08-13 13:32:23 -07:00
Shivaram Venkataraman
22ba3a3f77
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-13 13:32:10 -07:00
Shivaram Venkataraman
1f68c4b03b
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-13 13:31:54 -07:00
Shivaram Venkataraman
1ea269110c
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-13 13:31:45 -07:00
Shivaram Venkataraman
44661df9cc
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-13 13:31:39 -07:00
Shivaram Venkataraman
0dd8fe73ba
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
2012-08-13 13:31:29 -07:00
Shivaram Venkataraman
80104ce1da
Add link to Java wiki which specifies what changes with compressed oops
2012-08-13 13:31:21 -07:00
Shivaram Venkataraman
00ab5490b3
Changes to make size estimator more accurate. Fixes object size, pointer size
...
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
2012-08-13 13:31:11 -07:00
Matei Zaharia
6ae3c375a9
Renamed apply() to call() in Java API and allowed it to throw Exceptions
2012-08-12 23:10:19 +02:00
Matei Zaharia
0141879c40
Use Promises instead of having a Future wait on a thread in
...
ConnectionManager.
2012-08-12 22:16:32 +02:00
Matei Zaharia
845a870242
Return remotely fetched blocks in a pipelined fashion from BlockManager
2012-08-12 20:01:38 +02:00
Matei Zaharia
e17ed9a21d
Switch to Akka futures in connection manager.
...
It's still not good because each Future ends up waiting on a lock, but
it seems to work better than Scala Actors, and more importantly it
allows us to use onComplete and other listeners on futures.
2012-08-12 19:40:37 +02:00
Matei Zaharia
ad8a7612a4
Changed multi-get method in BlockManager to return an iterator
2012-08-12 19:18:01 +02:00
Matei Zaharia
3c94e5c188
Merge pull request #168 from shivaram/dev
...
Use JavaConversion to get a scala iterator
2012-08-10 00:57:33 -07:00
Matei Zaharia
e463e7a333
Merge pull request #167 from JoshRosen/piped-rdd-fixes
...
Detect non-zero exit status from PipedRDD process
2012-08-10 00:56:42 -07:00
Josh Rosen
59c22fb444
Print exit status in PipedRDD failure exception.
2012-08-10 00:33:56 -07:00
Shivaram Venkataraman
1803cce692
Use an implicit conversion to get the scala iterator
2012-08-08 14:31:04 -07:00
Shivaram Venkataraman
674fcf56bf
Use JavaConversion to get a scala iterator
2012-08-08 14:10:23 -07:00
Shivaram Venkataraman
f4aaec7a48
Avoid a copy in ShuffleMapTask by creating an iterator that will be used by the
...
block manager.
2012-08-08 00:47:02 -07:00
Mosharaf Chowdhury
d821dd3ccc
BroadcastManager is a class now (replaced Braodcast object)
2012-08-05 01:10:51 -07:00
Mosharaf Chowdhury
b4804119f9
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-04 20:42:12 -07:00
Matei Zaharia
88b016db2a
Merge pull request #160 from dennybritz/clusterscripts
...
Standalone cluster scripts
2012-08-04 17:45:20 -07:00
