Sital Kedia c7ac027d5f [SPARK-17839][CORE] Use Nio's directbuffer instead of BufferedInputStream in order to avoid additional copy from os buffer cache to user buffer ... ## What changes were proposed in this pull request? Currently we use BufferedInputStream to read the shuffle file which copies the file content from os buffer cache to the user buffer. This adds additional latency in reading the spill files. We made a change to use java nio's direct buffer to read the spill files and for certain pipelines spilling significant amount of data, we see up to 7% speedup for the entire pipeline. ## How was this patch tested? Tested by running the job in the cluster and observed up to 7% speedup. Author: Sital Kedia <skedia@fb.com> Closes #15408 from sitalkedia/skedia/nio_spill_read.		2016-10-17 11:03:04 -07:00
..
java/org/apache/spark/sql	[SPARK-17830][SQL] Annotate remaining SQL APIs with InterfaceStability	2016-10-13 11:12:30 -07:00
resources	[SPARK-16031] Add debug-only socket source in Structured Streaming	2016-06-19 21:27:04 -07:00
scala/org/apache/spark/sql	[SPARK-17839][CORE] Use Nio's directbuffer instead of BufferedInputStream in order to avoid additional copy from os buffer cache to user buffer	2016-10-17 11:03:04 -07:00