Liwei Lin
35d9c8aa69
[SPARK-14747][SQL] Add assertStreaming/assertNoneStreaming checks in DataFrameWriter
## Problem
If an end user happens to write code mixed with continuous-query-oriented methods and non-continuous-query-oriented methods:
```scala
ctx.read
.format("text")
.stream("...") // continuous query
.write
.text("...") // non-continuous query; should be startStream() here
```
He/she would get this somehow confusing exception:
>
Exception in thread "main" java.lang.AssertionError: assertion failed: No plan for FileSource[./continuous_query_test_input]
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at ...
## What changes were proposed in this pull request?
This PR adds checks for continuous-query-oriented methods and non-continuous-query-oriented methods in `DataFrameWriter`:
<table>
<tr>
<td align="center"></td>
<td align="center"><strong>can be called on continuous query?</strong></td>
<td align="center"><strong>can be called on non-continuous query?</strong></td>
</tr>
<tr>
<td align="center">mode</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">trigger</td>
<td align="center">yes</td>
<td align="center"></td>
</tr>
<tr>
<td align="center">format</td>
<td align="center">yes</td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">option/options</td>
<td align="center">yes</td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">partitionBy</td>
<td align="center">yes</td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">bucketBy</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">sortBy</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">save</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">queryName</td>
<td align="center">yes</td>
<td align="center"></td>
</tr>
<tr>
<td align="center">startStream</td>
<td align="center">yes</td>
<td align="center"></td>
</tr>
<tr>
<td align="center">insertInto</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">saveAsTable</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">jdbc</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">json</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">parquet</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">orc</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">text</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
<tr>
<td align="center">csv</td>
<td align="center"></td>
<td align="center">yes</td>
</tr>
</table>
After this PR's change, the friendly exception would be:
>
Exception in thread "main" org.apache.spark.sql.AnalysisException: text() can only be called on non-continuous queries;
at org.apache.spark.sql.DataFrameWriter.assertNotStreaming(DataFrameWriter.scala:678)
at org.apache.spark.sql.DataFrameWriter.text(DataFrameWriter.scala:629)
at ss.SSDemo$.main(SSDemo.scala:47)
## How was this patch tested?
dedicated unit tests were added
Author: Liwei Lin <lwlin7@gmail.com>
Closes #12521 from lw-lin/dataframe-writer-check.