Sahil Takiar e36ec38d89 [SPARK-20466][CORE] HadoopRDD#addLocalConfiguration throws NPE ... ## What changes were proposed in this pull request? Fix for SPARK-20466, full description of the issue in the JIRA. To summarize, `HadoopRDD` uses a metadata cache to cache `JobConf` objects. The cache uses soft-references, which means the JVM can delete entries from the cache whenever there is GC pressure. `HadoopRDD#getJobConf` had a bug where it would check if the cache contained the `JobConf`, if it did it would get the `JobConf` from the cache and return it. This doesn't work when soft-references are used as the JVM can delete the entry between the existence check and the get call. ## How was this patch tested? Haven't thought of a good way to test this yet given the issue only occurs sometimes, and happens during high GC pressure. Was thinking of using mocks to verify `#getJobConf` is doing the right thing. I deleted the method `HadoopRDD#containsCachedMetadata` so that we don't hit this issue again. Author: Sahil Takiar <stakiar@cloudera.com> Closes #19413 from sahilTakiar/master.		2017-10-03 16:53:32 -07:00
..
src	[SPARK-20466][CORE] HadoopRDD#addLocalConfiguration throws NPE	2017-10-03 16:53:32 -07:00
pom.xml	[SPARK-22167][R][BUILD] sparkr packaging issue allow zinc	2017-10-02 11:46:51 -07:00