ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker

Browse source 
### What changes were proposed in this pull request?

This log should at least be WARN not INFO (in org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala )
"Expected $numSubmittedFiles files, but only saw $numFiles."

### Why are the changes needed?

INFO logs don't indicate possible issue but WARN logs should

### Does this PR introduce any user-facing change?

Yes, Log level changed.

### How was this patch tested?

manual, trivial change

Closes #33332 from tooptoop4/warn.

Authored-by: tooptoop4 <33283496+tooptoop4@users.noreply.github.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
tooptoop4 2021-08-19 10:31:10 +09:00 committed by Hyukjin Kwon
parent f2e593bcf1
commit 2fc9c0bfb5
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
View file

@ -169,7 +169,7 @@ class BasicWriteTaskStatsTracker(
} }
if (numSubmittedFiles != numFiles) { if (numSubmittedFiles != numFiles) {
logInfo(s"Expected $numSubmittedFiles files, but only saw $numFiles. " + logWarning(s"Expected $numSubmittedFiles files, but only saw $numFiles. " +
"This could be due to the output format not writing empty files, " + "This could be due to the output format not writing empty files, " +
"or files being not immediately visible in the filesystem.") "or files being not immediately visible in the filesystem.")
} }