Denny
4e7b264cf7
Set SPARK_LAUNCH_WITH_SCALA=0 in Executor Runner
2012-09-07 11:39:44 -07:00
root
c2da64409a
Randomize the order of block fetches in getMultiple
2012-09-06 23:16:26 +00:00
Mosharaf Chowdhury
7c6936f1bc
Merge remote-tracking branch 'upstream/dev' into dev
2012-09-06 01:46:48 -07:00
Denny
886183e591
Renamed spark-cluster to spark-local.
2012-09-05 17:10:54 -07:00
Matei Zaharia
53a5681c8a
Merge pull request #190 from rxin/dev
...
Log cache add/remove messages in block manager.
2012-09-05 16:41:52 -07:00
Reynold Xin
c308fbcb79
Removed cache add/remove log messages from CacheTracker.
...
Added log messages on BlockManagerMaster to reflect block add/remove.
Also did some minor cleanup of storage package code.
2012-09-05 15:59:48 -07:00
Denny
babbca0a2f
Fix wrong counting
2012-09-04 22:04:18 -07:00
Denny
9326509f66
Delete old DeployUtils.
2012-09-04 21:15:23 -07:00
Denny
1588d4dbe6
Renamed class.
2012-09-04 21:13:25 -07:00
Denny
22dde6e020
Start a standalone cluster locally.
2012-09-04 20:56:30 -07:00
Matei Zaharia
a842c63044
Minor formatting fixes
2012-09-03 16:24:00 -07:00
Mosharaf Chowdhury
5e8e8e4c9d
Merge remote-tracking branch 'upstream/dev' into dev
2012-09-03 16:16:56 -07:00
Matei Zaharia
2d6a629f8c
Merge pull request #182 from HarveyFeng/dev-fetch
...
Add a limit on the number of parallel fetches in the reduce stage
2012-09-03 16:14:57 -07:00
Harvey
3076b038f4
Start fetching a remote block when a received remote block has been passed
...
to the reduce function
2012-09-01 12:01:35 -07:00
Mosharaf Chowdhury
10dfee7501
Merge remote-tracking branch 'upstream/dev' into dev
2012-09-01 00:33:00 -07:00
Matei Zaharia
389fb4cc54
End runJob() with a SparkException when a task fails too many times in
...
one of the cluster schedulers.
2012-08-31 17:47:43 -07:00
Mosharaf Chowdhury
baf2a7ccd2
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-30 22:28:14 -07:00
Mosharaf Chowdhury
31ffe8d528
Synchronization bug fix in broadcast implementations
2012-08-30 22:26:43 -07:00
Mosharaf Chowdhury
3883532545
Bug fix. Fixed log messages. Updated BroadcastTest example to have iterations.
2012-08-30 21:43:00 -07:00
Matei Zaharia
a480dec6b2
Deserialize multi-get results in the caller's thread. This fixes an
...
issue with shared buffers in the KryoSerializer.
2012-08-30 20:01:06 -07:00
Mosharaf Chowdhury
8f2bd399da
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-30 15:21:08 -07:00
Matei Zaharia
bf3212615a
Merge pull request #184 from rxin/dev
...
Disable running combiners on map tasks when mergeCombiners function is not specified by the user.
2012-08-30 14:12:40 -07:00
Reynold Xin
a8a2a08a1a
Added a test for testing map-side combine on/off switch.
2012-08-30 12:34:28 -07:00
Reynold Xin
5945bcdcc5
Added a new flag in Aggregator to indicate applying map side combiners.
2012-08-29 23:32:08 -07:00
Reynold Xin
c68e820b2a
Merge branch 'dev' of github.com:mesos/spark into dev
2012-08-29 23:01:19 -07:00
Reynold Xin
940869dfda
Disable running combiners on map tasks when mergeCombiners function is
...
not specified by the user.
2012-08-29 23:00:02 -07:00
Matei Zaharia
47507d69d9
Made region used by spark-ec2 configurable.
2012-08-28 22:40:00 -07:00
Mosharaf Chowdhury
c74455f309
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-28 14:56:57 -07:00
Matei Zaharia
bf2e9cb08e
Fault tolerance and block store fixes discovered through streaming tests.
2012-08-27 23:07:50 -07:00
Matei Zaharia
291abc2c28
Merge pull request #181 from rxin/dev
...
Removed the deserialization cache for ShuffleMapTask
2012-08-27 22:38:22 -07:00
Reynold Xin
3a6a95dc24
Removed the deserialization cache for ShuffleMapTask because it was
...
causing concurrency problems (some variables in Shark get set to null).
The cost of task deserialization on slaves is trivial compared with the
execution time of the task anyway.
2012-08-27 22:33:15 -07:00
Matei Zaharia
51453eb87b
Merge pull request #179 from JoshRosen/fix/sparklr-caching
...
Cache points in SparkLR example
2012-08-26 15:32:50 -07:00
Josh Rosen
566feafe1d
Cache points in SparkLR example.
2012-08-26 15:24:43 -07:00
Mosharaf Chowdhury
edd1a740a6
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 20:43:27 -07:00
Matei Zaharia
2c16ae36d7
Set log level in tests to WARN
2012-08-23 20:38:14 -07:00
Matei Zaharia
deedb9e7b7
Fix further issues with tests and broadcast.
...
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
2012-08-23 20:31:49 -07:00
Mosharaf Chowdhury
3b1f5480a4
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 20:16:50 -07:00
Matei Zaharia
59b831b9d1
Fixed test failures due to broadcast not stopping correctly
2012-08-23 19:59:55 -07:00
Matei Zaharia
7310a6f499
Merge pull request #147 from mosharaf/dev
...
Broadcast refactoring/cleaning up
2012-08-23 19:38:28 -07:00
Mosharaf Chowdhury
995ad6ba36
Merge remote-tracking branch 'upstream/dev' into dev
2012-08-23 09:51:38 -07:00
Matei Zaharia
25a6a39e6d
Added other SparkContext constructors to JavaSparkContext
2012-08-19 18:59:16 -07:00
Matei Zaharia
9a0c128fec
Merge pull request #172 from dennybritz/dev
...
Rsync root directory in EC2 script
2012-08-14 13:05:22 -07:00
Denny
8dc7242544
Use root login in standalone AMI
2012-08-14 10:18:24 -07:00
Denny
7152c7c12d
rsync root directory in EC2 script
2012-08-14 09:26:47 -07:00
Matei Zaharia
942e604c62
Merge pull request #171 from shivaram/for-size-estimator-pull
...
Size estimator changes for dev
2012-08-13 15:29:40 -07:00
Shivaram Venkataraman
0f4fbb057b
Change BlockManagerSuite test cases to use a deterministic size estimator and
...
update the results to match the new estimates
2012-08-13 13:32:23 -07:00
Shivaram Venkataraman
22ba3a3f77
Add test-cases for 32-bit and no-compressed oops scenarios.
2012-08-13 13:32:10 -07:00
Shivaram Venkataraman
1f68c4b03b
Update test cases to match the new size estimates. Uses 64-bit and compressed
...
oops setting to get deterministic results
2012-08-13 13:31:54 -07:00
Shivaram Venkataraman
1ea269110c
Move object size and pointer size initialization into a function to enable unit-testing
2012-08-13 13:31:45 -07:00
Shivaram Venkataraman
44661df9cc
If spark.test.useCompressedOops is set, use that to infer compressed oops
...
setting. This is useful to get a deterministic test case
2012-08-13 13:31:39 -07:00