Thomas Graves 5b08ee6396 [SPARK-15671] performance regression CoalesceRDD.pickBin with large #… ... I was running a 15TB join job with 202000 partitions. It looks like the changes I made to CoalesceRDD in pickBin() are really slow with that large of partitions. The array filter with that many elements just takes to long. It took about an hour for it to pickBins for all the partitions. original change: 83ee92f603 Just reverting the pickBin code back to get currpreflocs fixes the issue After reverting the pickBin code the coalesce takes about 10 seconds so for now it makes sense to revert those changes and we can look at further optimizations later. Tested this via RDDSuite unit test and manually testing the very large job. Author: Thomas Graves <tgraves@prevailsail.corp.gq1.yahoo.com> Closes #13443 from tgravescs/SPARK-15671.		2016-06-01 13:21:40 -07:00
..
main	[SPARK-15671] performance regression CoalesceRDD.pickBin with large #…	2016-06-01 13:21:40 -07:00
test	[SPARK-15601][CORE] CircularBuffer's toString() to print only the contents written if buffer isn't full	2016-05-31 19:52:22 -05:00