Tathagata Das
|
e95ff45b53
|
Implemented checkpointing of StreamingContext and DStream graph.
|
2012-10-13 20:10:49 -07:00 |
|
|
Tathagata Das
|
c63a606458
|
Made NewHadoopRDD broadcast its job configuration (same as HadoopRDD).
|
2012-09-10 19:51:27 +00:00 |
|
haoyuan
|
db08a362aa
|
commit opt for grep scalibility test.
|
2012-09-07 02:17:52 +00:00 |
|
root
|
9ef90c95f4
|
Bug fix
|
2012-09-06 00:43:46 +00:00 |
|
|
root
|
2fa6d999fd
|
Tuning Akka more
|
2012-09-06 00:16:39 +00:00 |
|
|
root
|
215544820f
|
Serialize map output locations more efficiently, and only once, in MapOutputTracker
|
2012-09-05 23:54:04 +00:00 |
|
|
root
|
dc68febdce
|
User Spark's closure serializer for the ShuffleMapTask cache
|
2012-09-05 23:06:59 +00:00 |
|
|
root
|
ed937a821f
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 22:26:49 +00:00 |
|
|
root
|
1d6b36d3c3
|
Further tuning for network performance
|
2012-09-05 22:26:37 +00:00 |
|
root
|
3fa0d7f0c9
|
Serialize BlockRDD more efficiently
|
2012-09-05 08:28:15 +00:00 |
|
root
|
4a5d0d249e
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 08:23:09 +00:00 |
|
|
root
|
efc7668d16
|
Allow serializing HttpBroadcast through Kryo
|
2012-09-05 08:22:57 +00:00 |
|
|
root
|
75487b2f5a
|
Broadcast the JobConf in HadoopRDD to reduce task sizes
|
2012-09-05 08:14:50 +00:00 |
|
|
root
|
b7ad291ac5
|
Tuning Akka for more connections
|
2012-09-05 07:08:07 +00:00 |
|
|
root
|
fc186dc18a
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 05:53:18 +00:00 |
|
|
root
|
4ea032a142
|
Some changes to make important log output visible even if we set the logging to WARNING
|
2012-09-05 05:53:07 +00:00 |
|
|
Tathagata Das
|
7c09ad0e04
|
Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage.
|
2012-09-04 19:11:49 -07:00 |
|
|
Tathagata Das
|
b8e9e8ea78
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-02 02:35:32 -07:00 |
|
root
|
ceabf71257
|
tweaks
|
2012-09-01 21:52:42 +00:00 |
|
|
root
|
6025889be0
|
More raw network receiver programs
|
2012-09-01 20:51:07 +00:00 |
|
Matei Zaharia
|
f84d2bbe55
|
Bug fixes to RateLimitedOutputStream
|
2012-09-01 00:31:15 -07:00 |
|
|
Matei Zaharia
|
44758aa8e2
|
First work towards a RawInputDStream and a sender program for it.
|
2012-09-01 00:17:59 -07:00 |
|
root
|
c42e7ac282
|
More block manager fixes
|
2012-09-01 04:31:11 +00:00 |
|
root
|
113277549c
|
Really fixed the replication-3 issue. The problem was a few buffers not being rewound.
|
2012-08-31 05:39:35 +00:00 |
|
|
Matei Zaharia
|
101ae493e2
|
Replicate serialized blocks properly, without sharing a ByteBuffer.
|
2012-08-30 22:24:14 -07:00 |
|
|
Matei Zaharia
|
1b3e3352eb
|
Deserialize multi-get results in the caller's thread. This fixes an
issue with shared buffers with the KryoSerializer.
|
2012-08-30 17:59:25 -07:00 |
|
|
root
|
c4366eb764
|
Fixes to ShuffleFetcher
|
2012-08-31 00:34:24 +00:00 |
|
|
Tathagata Das
|
4db3a96766
|
Made minor changes to reduce compilation errors in Eclipse. Twirl stuff still does not compile in Eclipse.
|
2012-08-29 13:04:01 -07:00 |
|
|
Matei Zaharia
|
17af2df0cd
|
Log levels
|
2012-08-27 23:07:32 -07:00 |
|
|
Matei Zaharia
|
b4a2214218
|
More fault tolerance fixes to catch lost tasks
|
2012-08-27 22:49:29 -07:00 |
|
|
Matei Zaharia
|
b914cd0dfa
|
Serialize generation correctly in ShuffleMapTask
|
2012-08-26 20:07:59 -07:00 |
|
|
Matei Zaharia
|
69c2ab0408
|
logging
|
2012-08-26 20:00:58 -07:00 |
|
|
Matei Zaharia
|
117e3f8c86
|
Fix a bug that was causing FetchFailedException not to be thrown
|
2012-08-26 19:52:56 -07:00 |
|
|
Matei Zaharia
|
3c9c44a8d3
|
More helpful log messages
|
2012-08-26 19:37:43 -07:00 |
|
|
Matei Zaharia
|
26dfd20c9a
|
Detect disconnected slaves in StandaloneScheduler
|
2012-08-26 18:56:56 -07:00 |
|
|
Matei Zaharia
|
29e83f39e9
|
Fix replication with MEMORY_ONLY_DESER_2
|
2012-08-26 18:16:25 -07:00 |
|
|
Matei Zaharia
|
06ef7c3d1b
|
Less debug info
|
2012-08-26 16:29:20 -07:00 |
|
|
Matei Zaharia
|
741899b21e
|
Fix sendMessageReliablySync
|
2012-08-26 16:26:06 -07:00 |
|
|
Matei Zaharia
|
5a8015d2db
|
Merge remote-tracking branch 'public/dev' into dev
|
2012-08-24 16:11:44 -07:00 |
|
|
Matei Zaharia
|
2c16ae36d7
|
Set log level in tests to WARN
|
2012-08-23 20:38:14 -07:00 |
|
|
Matei Zaharia
|
deedb9e7b7
|
Fix further issues with tests and broadcast.
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
|
2012-08-23 20:31:49 -07:00 |
|
|
Matei Zaharia
|
59b831b9d1
|
Fixed test failures due to broadcast not stopping correctly
|
2012-08-23 19:59:55 -07:00 |
|
|
Matei Zaharia
|
7310a6f499
|
Merge pull request #147 from mosharaf/dev
Broadcast refactoring/cleaning up
|
2012-08-23 19:38:28 -07:00 |
|
|
Matei Zaharia
|
25a6a39e6d
|
Added other SparkContext constructors to JavaSparkContext
|
2012-08-19 18:59:16 -07:00 |
|
Shivaram Venkataraman
|
0f4fbb057b
|
Change BlockManagerSuite test cases to use a deterministic size estimator and
update the results to match the new estimates
|
2012-08-13 13:32:23 -07:00 |
|
|
Shivaram Venkataraman
|
22ba3a3f77
|
Add test-cases for 32-bit and no-compressed oops scenarios.
|
2012-08-13 13:32:10 -07:00 |
|
|
Shivaram Venkataraman
|
1f68c4b03b
|
Update test cases to match the new size estimates. Uses 64-bit and compressed
oops setting to get deterministic results
|
2012-08-13 13:31:54 -07:00 |
|
|
Shivaram Venkataraman
|
1ea269110c
|
Move object size and pointer size initialization into a function to enable unit-testing
|
2012-08-13 13:31:45 -07:00 |
|
|
Shivaram Venkataraman
|
44661df9cc
|
If spark.test.useCompressedOops is set, use that to infer compressed oops
setting. This is useful to get a deterministic test case
|
2012-08-13 13:31:39 -07:00 |
|
|
Shivaram Venkataraman
|
0dd8fe73ba
|
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
|
2012-08-13 13:31:29 -07:00 |
|