[SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker
### What changes were proposed in this pull request? This log should at least be WARN not INFO (in org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala ) "Expected $numSubmittedFiles files, but only saw $numFiles." ### Why are the changes needed? INFO logs don't indicate possible issue but WARN logs should ### Does this PR introduce any user-facing change? Yes, Log level changed. ### How was this patch tested? manual, trivial change Closes #33332 from tooptoop4/warn. Authored-by: tooptoop4 <33283496+tooptoop4@users.noreply.github.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
parent
f2e593bcf1
commit
2fc9c0bfb5
|
@ -169,7 +169,7 @@ class BasicWriteTaskStatsTracker(
|
|||
}
|
||||
|
||||
if (numSubmittedFiles != numFiles) {
|
||||
logInfo(s"Expected $numSubmittedFiles files, but only saw $numFiles. " +
|
||||
logWarning(s"Expected $numSubmittedFiles files, but only saw $numFiles. " +
|
||||
"This could be due to the output format not writing empty files, " +
|
||||
"or files being not immediately visible in the filesystem.")
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue