534f5d409a
### What changes were proposed in this pull request? This PR aims to increase the timeout of `StreamingLogisticRegressionWithSGDTests.test_parameter_accuracy` from 30s (default) to 60s. In this PR, before increasing the timeout, 1. I verified that this is not a JDK11 environmental issue by repeating 3 times first. 2. I reproduced the accuracy failure by reducing the timeout in Jenkins (https://github.com/apache/spark/pull/27424#issuecomment-580981262) Then, the final commit passed the Jenkins. ### Why are the changes needed? This seems to happen when Jenkins environment has congestion and the jobs are slowdown. The streaming job seems to be unable to repeat the designed iteration `numIteration=25` in 30 seconds. Since the error is decreasing at each iteration, the failure occurs. By reducing the timeout, we can reproduce the similar issue locally like Jenkins. ```python - eventually(condition, catch_assertions=True) + eventually(condition, timeout=10.0, catch_assertions=True) ``` ``` $ python/run-tests --testname 'pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_parameter_accuracy' --python-executables=python ... ====================================================================== FAIL: test_parameter_accuracy (pyspark.mllib.tests.test_streaming_algorithms.StreamingLogisticRegressionWithSGDTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dongjoon/PRS/SPARK-TEST/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 229, in test_parameter_accuracy eventually(condition, timeout=10.0, catch_assertions=True) File "/Users/dongjoon/PRS/SPARK-TEST/python/pyspark/testing/utils.py", line 86, in eventually raise lastValue Reproduce the error File "/Users/dongjoon/PRS/SPARK-TEST/python/pyspark/testing/utils.py", line 77, in eventually lastValue = condition() File "/Users/dongjoon/PRS/SPARK-TEST/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 226, in condition self.assertAlmostEqual(rel, 0.1, 1) AssertionError: 0.25749106949322637 != 0.1 within 1 places (0.15749106949322636 difference) ---------------------------------------------------------------------- Ran 1 test in 14.814s FAILED (failures=1) ``` ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins (and manual check by reducing the timeout). Since this is a flakiness issue depending on the Jenkins job situation, it's difficult to reproduce there. Closes #27424 from dongjoon-hyun/SPARK-TEST. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> |
||
---|---|---|
.. | ||
linalg | ||
stat | ||
tests | ||
__init__.py | ||
classification.py | ||
clustering.py | ||
common.py | ||
evaluation.py | ||
feature.py | ||
fpm.py | ||
random.py | ||
recommendation.py | ||
regression.py | ||
tree.py | ||
util.py |