root
|
215544820f
|
Serialize map output locations more efficiently, and only once, in MapOutputTracker
|
2012-09-05 23:54:04 +00:00 |
|
|
root
|
dc68febdce
|
User Spark's closure serializer for the ShuffleMapTask cache
|
2012-09-05 23:06:59 +00:00 |
|
|
root
|
ed937a821f
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 22:26:49 +00:00 |
|
|
root
|
1d6b36d3c3
|
Further tuning for network performance
|
2012-09-05 22:26:37 +00:00 |
|
root
|
3fa0d7f0c9
|
Serialize BlockRDD more efficiently
|
2012-09-05 08:28:15 +00:00 |
|
root
|
4a5d0d249e
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 08:23:09 +00:00 |
|
|
root
|
efc7668d16
|
Allow serializing HttpBroadcast through Kryo
|
2012-09-05 08:22:57 +00:00 |
|
|
root
|
75487b2f5a
|
Broadcast the JobConf in HadoopRDD to reduce task sizes
|
2012-09-05 08:14:50 +00:00 |
|
|
root
|
b7ad291ac5
|
Tuning Akka for more connections
|
2012-09-05 07:08:07 +00:00 |
|
|
root
|
fc186dc18a
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 05:53:18 +00:00 |
|
|
root
|
4ea032a142
|
Some changes to make important log output visible even if we set the logging to WARNING
|
2012-09-05 05:53:07 +00:00 |
|
Tathagata Das
|
7c09ad0e04
|
Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage.
|
2012-09-04 19:11:49 -07:00 |
|
|
Tathagata Das
|
b8e9e8ea78
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-02 02:35:32 -07:00 |
|
root
|
ceabf71257
|
tweaks
|
2012-09-01 21:52:42 +00:00 |
|
|
root
|
6025889be0
|
More raw network receiver programs
|
2012-09-01 20:51:07 +00:00 |
|
Matei Zaharia
|
f84d2bbe55
|
Bug fixes to RateLimitedOutputStream
|
2012-09-01 00:31:15 -07:00 |
|
|
Matei Zaharia
|
44758aa8e2
|
First work towards a RawInputDStream and a sender program for it.
|
2012-09-01 00:17:59 -07:00 |
|
root
|
c42e7ac282
|
More block manager fixes
|
2012-09-01 04:31:11 +00:00 |
|
root
|
113277549c
|
Really fixed the replication-3 issue. The problem was a few buffers not being rewound.
|
2012-08-31 05:39:35 +00:00 |
|
|
Matei Zaharia
|
101ae493e2
|
Replicate serialized blocks properly, without sharing a ByteBuffer.
|
2012-08-30 22:24:14 -07:00 |
|
|
Matei Zaharia
|
1b3e3352eb
|
Deserialize multi-get results in the caller's thread. This fixes an
issue with shared buffers with the KryoSerializer.
|
2012-08-30 17:59:25 -07:00 |
|
|
root
|
c4366eb764
|
Fixes to ShuffleFetcher
|
2012-08-31 00:34:24 +00:00 |
|
|
Tathagata Das
|
4db3a96766
|
Made minor changes to reduce compilation errors in Eclipse. Twirl stuff still does not compile in Eclipse.
|
2012-08-29 13:04:01 -07:00 |
|
|
Matei Zaharia
|
17af2df0cd
|
Log levels
|
2012-08-27 23:07:32 -07:00 |
|
|
Matei Zaharia
|
b4a2214218
|
More fault tolerance fixes to catch lost tasks
|
2012-08-27 22:49:29 -07:00 |
|
|
Matei Zaharia
|
b914cd0dfa
|
Serialize generation correctly in ShuffleMapTask
|
2012-08-26 20:07:59 -07:00 |
|
|
Matei Zaharia
|
69c2ab0408
|
logging
|
2012-08-26 20:00:58 -07:00 |
|
|
Matei Zaharia
|
117e3f8c86
|
Fix a bug that was causing FetchFailedException not to be thrown
|
2012-08-26 19:52:56 -07:00 |
|
|
Matei Zaharia
|
3c9c44a8d3
|
More helpful log messages
|
2012-08-26 19:37:43 -07:00 |
|
|
Matei Zaharia
|
26dfd20c9a
|
Detect disconnected slaves in StandaloneScheduler
|
2012-08-26 18:56:56 -07:00 |
|
|
Matei Zaharia
|
29e83f39e9
|
Fix replication with MEMORY_ONLY_DESER_2
|
2012-08-26 18:16:25 -07:00 |
|
|
Matei Zaharia
|
06ef7c3d1b
|
Less debug info
|
2012-08-26 16:29:20 -07:00 |
|
|
Matei Zaharia
|
741899b21e
|
Fix sendMessageReliablySync
|
2012-08-26 16:26:06 -07:00 |
|
|
Matei Zaharia
|
5a8015d2db
|
Merge remote-tracking branch 'public/dev' into dev
|
2012-08-24 16:11:44 -07:00 |
|
|
Matei Zaharia
|
2c16ae36d7
|
Set log level in tests to WARN
|
2012-08-23 20:38:14 -07:00 |
|
|
Matei Zaharia
|
deedb9e7b7
|
Fix further issues with tests and broadcast.
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
|
2012-08-23 20:31:49 -07:00 |
|
|
Matei Zaharia
|
59b831b9d1
|
Fixed test failures due to broadcast not stopping correctly
|
2012-08-23 19:59:55 -07:00 |
|
|
Matei Zaharia
|
7310a6f499
|
Merge pull request #147 from mosharaf/dev
Broadcast refactoring/cleaning up
|
2012-08-23 19:38:28 -07:00 |
|
|
Matei Zaharia
|
25a6a39e6d
|
Added other SparkContext constructors to JavaSparkContext
|
2012-08-19 18:59:16 -07:00 |
|
Shivaram Venkataraman
|
0f4fbb057b
|
Change BlockManagerSuite test cases to use a deterministic size estimator and
update the results to match the new estimates
|
2012-08-13 13:32:23 -07:00 |
|
|
Shivaram Venkataraman
|
22ba3a3f77
|
Add test-cases for 32-bit and no-compressed oops scenarios.
|
2012-08-13 13:32:10 -07:00 |
|
|
Shivaram Venkataraman
|
1f68c4b03b
|
Update test cases to match the new size estimates. Uses 64-bit and compressed
oops setting to get deterministic results
|
2012-08-13 13:31:54 -07:00 |
|
|
Shivaram Venkataraman
|
1ea269110c
|
Move object size and pointer size initialization into a function to enable unit-testing
|
2012-08-13 13:31:45 -07:00 |
|
|
Shivaram Venkataraman
|
44661df9cc
|
If spark.test.useCompressedOops is set, use that to infer compressed oops
setting. This is useful to get a deterministic test case
|
2012-08-13 13:31:39 -07:00 |
|
|
Shivaram Venkataraman
|
0dd8fe73ba
|
Use HotSpotDiagnosticMXBean to get if CompressedOops are in use or not
|
2012-08-13 13:31:29 -07:00 |
|
|
Shivaram Venkataraman
|
80104ce1da
|
Add link to Java wiki which specifies what changes with compressed oops
|
2012-08-13 13:31:21 -07:00 |
|
|
Shivaram Venkataraman
|
00ab5490b3
|
Changes to make size estimator more accurate. Fixes object size, pointer size
according to architecture and also aligns objects and arrays when computing
instance sizes. Verified using Eclipse Memory Analysis Tool (MAT)
|
2012-08-13 13:31:11 -07:00 |
|
|
Matei Zaharia
|
6ae3c375a9
|
Renamed apply() to call() in Java API and allowed it to throw Exceptions
|
2012-08-12 23:10:19 +02:00 |
|
|
Matei Zaharia
|
0141879c40
|
Use Promises instead of having a Future wait on a thread in
ConnectionManager.
|
2012-08-12 22:16:32 +02:00 |
|
|
Matei Zaharia
|
845a870242
|
Return remotely fetched blocks in a pipelined fashion from BlockManager
|
2012-08-12 20:01:38 +02:00 |
|