1db7179fae
Recently, PySpark ML streaming tests have been flaky, most likely because of the batches not being processed in time. Proposal: Replace the use of _ssc_wait (which waits for a fixed amount of time) with a method which waits for a fixed amount of time but can terminate early based on a termination condition method. With this, we can extend the waiting period (to make tests less flaky) but also stop early when possible (making tests faster on average, which I verified locally). CC: mengxr tdas freeman-lab Author: Joseph K. Bradley <joseph@databricks.com> Closes #8087 from jkbradley/streaming-ml-tests. |
||
|---|---|---|
| .. | ||
| linalg | ||
| stat | ||
| __init__.py | ||
| classification.py | ||
| clustering.py | ||
| common.py | ||
| evaluation.py | ||
| feature.py | ||
| fpm.py | ||
| random.py | ||
| recommendation.py | ||
| regression.py | ||
| tests.py | ||
| tree.py | ||
| util.py | ||