2f55809425
## What changes were proposed in this pull request? The actual implementation doesn't support multi-cluster Kafka connection with delegation token. In this PR I've added this functionality. What this PR contains: * New way of configuration * Multiple delegation token obtain/store/use functionality * Documentation * The change works on DStreams also ## How was this patch tested? Existing + additional unit tests. Additionally tested on cluster. Test scenario: * 2 * 4 node clusters * The 4-4 nodes are in different kerberos realms * Cross-Realm trust between the 2 realms * Yarn * Kafka broker version 2.1.0 * security.protocol = SASL_SSL * sasl.mechanism = SCRAM-SHA-512 * Artificial exceptions during processing * Source reads from realm1 sink writes to realm2 Kafka broker settings: * delegation.token.expiry.time.ms=600000 (10 min) * delegation.token.max.lifetime.ms=1200000 (20 min) * delegation.token.expiry.check.interval.ms=300000 (5 min) Closes #24305 from gaborgsomogyi/SPARK-27294. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> |
||
|---|---|---|
| .. | ||
| src | ||
| pom.xml | ||