Gabor Somogyi 579edf4728 [SPARK-28335][DSTREAMS][TEST] DirectKafkaStreamSuite wait for Kafka async commit ... ## What changes were proposed in this pull request? `DirectKafkaStreamSuite.offset recovery from kafka` commits offsets to Kafka with `Consumer.commitAsync` API (and then reads it back). Since this API is asynchronous it may send notifications late(or not at all). The actual test makes the assumption if the data sent and collected then the offset must be committed as well. This is not true. In this PR I've made the following modifications: * Wait for async offset commit before context stopped * Added commit succeed log to see whether it arrived at all * Using `ConcurrentHashMap` for committed offsets because 2 threads are using the variable (`JobGenerator` and `ScalaTest...`) ## How was this patch tested? Existing unit test in a loop + jenkins runs. Closes #25100 from gaborgsomogyi/SPARK-28335. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>		2019-07-10 09:35:39 -07:00
..
src	[SPARK-28335][DSTREAMS][TEST] DirectKafkaStreamSuite wait for Kafka async commit	2019-07-10 09:35:39 -07:00
pom.xml	[SPARK-27294][SS] Add multi-cluster Kafka delegation token	2019-05-07 11:40:43 -07:00