Hopefully, this suite will not be flaky anymore.
Author: Yin Huai <yhuai@databricks.com>
Closes#7027 from yhuai/SPARK-8567 and squashes the following commits:
c0167e2 [Yin Huai] Add sc.stop().
(cherry picked from commit fbf75738fe)
Signed-off-by: Andrew Or <andrew@databricks.com>
As per the unit test log in https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35754/
```
15/06/24 23:09:10.210 Thread-3495 INFO ReceiverTracker: Starting 1 receivers
15/06/24 23:09:10.270 Thread-3495 INFO SparkContext: Starting job: apply at Transformer.scala:22
...
15/06/24 23:09:14.259 ForkJoinPool-4-worker-29 INFO StreamingListenerSuiteReceiver: Started receiver and sleeping
15/06/24 23:09:14.270 ForkJoinPool-4-worker-29 INFO StreamingListenerSuiteReceiver: Reporting error and sleeping
```
it needs at least 4 seconds to receive all receiver events in this slow machine, but `timeout` for `eventually` is only 2 seconds.
This PR increases `timeout` to make this test stable.
Author: zsxwing <zsxwing@gmail.com>
Closes#7017 from zsxwing/SPARK-8634 and squashes the following commits:
719cae4 [zsxwing] Fix flaky test StreamingListenerSuite "receiver info reporting"
(cherry picked from commit cec98525fd)
Signed-off-by: Andrew Or <andrew@databricks.com>
jira: https://issues.apache.org/jira/browse/SPARK-8710
Author: Yin Huai <yhuai@databricks.com>
Closes#7094 from yhuai/SPARK-8710 and squashes the following commits:
c854baa [Yin Huai] Change ScalaReflection.mirror from a val to a def.
(cherry picked from commit 4b497a724a)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Author: Michael Armbrust <michael@databricks.com>
Closes#7102 from marmbrus/fixStyle and squashes the following commits:
8c08124 [Michael Armbrust] [HOTFIX] Fix whitespace style error
Cherry-pick f9b397f54d to branch 1.4.
Author: Yin Huai <yhuai@databricks.com>
Closes#7092 from yhuai/SPARK-8567-1.4 and squashes the following commits:
0ae2e14 [Yin Huai] [SPARK-8567] [SQL] Add logs to record the progress of HiveSparkSubmitSuite.
Method "_load_from_socket" in rdd.py cannot load data from jvm socket when ipv6 is used. The current method only works well with ipv4. New modification should work around both two protocols.
Author: Ai He <ai.he@ussuning.com>
Author: AiHe <ai.he@ussuning.com>
Closes#6338 from AiHe/pyspark-networking-issue and squashes the following commits:
d4fc9c4 [Ai He] handle code review 2
e75c5c8 [Ai He] handle code review
5644953 [AiHe] solve python rdd socket connection problem to jvm
(cherry picked from commit ecd3aacf28)
Signed-off-by: Davies Liu <davies@databricks.com>
Author: scwf <wangfei1@huawei.com>
Closes#6145 from scwf/InConversion and squashes the following commits:
5c8ac6b [scwf] minir fix for InConversion
(cherry picked from commit edf09ea1bd)
Signed-off-by: Cheng Lian <lian@databricks.com>
I specifically randomized the test. What crosstab does is equivalent to a countByKey, therefore if this test fails again for any reason, we will know that we hit a corner case or something.
cc rxin marmbrus
Author: Burak Yavuz <brkyvz@gmail.com>
Closes#7060 from brkyvz/crosstab-fixes and squashes the following commits:
0a65234 [Burak Yavuz] addressed comments v1
d96da7e [Burak Yavuz] fixed wrong ordering of columns in crosstab
(cherry picked from commit be7ef06762)
Signed-off-by: Reynold Xin <rxin@databricks.com>
In DataFrame.scala, there are examples like as follows.
```
* // The following are equivalent:
* peopleDf.filter($"age" > 15)
* peopleDf.where($"age" > 15)
* peopleDf($"age" > 15)
```
But, I think the last example doesn't work.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes#6977 from sarutak/fix-dataframe-example and squashes the following commits:
46efbd7 [Kousuke Saruta] Removed wrong example
(cherry picked from commit 94e040d059)
Signed-off-by: Reynold Xin <rxin@databricks.com>
I compared PySpark DataFrameReader/Writer against Scala ones. `Option` function is missing in both reader and writer, but the rest seems to all match.
I added `Option` to reader and writer and updated the `pyspark-sql` test.
Author: Cheolsoo Park <cheolsoop@netflix.com>
Closes#7078 from piaozhexiu/SPARK-8355 and squashes the following commits:
c63d419 [Cheolsoo Park] Fix version
524e0aa [Cheolsoo Park] Add option function to df reader and writer
(cherry picked from commit ac2e17b01c)
Signed-off-by: Reynold Xin <rxin@databricks.com>
If `RDD.getPreferredLocations()` throws an exception it may crash the DAGScheduler and SparkContext. This patch addresses this by adding a try-catch block.
Author: Josh Rosen <joshrosen@databricks.com>
Closes#7023 from JoshRosen/SPARK-8606 and squashes the following commits:
770b169 [Josh Rosen] Fix getPreferredLocations() DAGScheduler crash with try block.
44a9b55 [Josh Rosen] Add test of a buggy getPartitions() method
19aa9f7 [Josh Rosen] Add (failing) regression test for getPreferredLocations() DAGScheduler crash
(cherry picked from commit 0b5abbf5f9)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
As per the description in the JIRA, I moved the contents of the page and added a few additional content.
Author: Neelesh Srinivas Salian <nsalian@cloudera.com>
Closes#6924 from nssalian/SPARK-3629 and squashes the following commits:
944b7a0 [Neelesh Srinivas Salian] Changed the lines about deploy-mode and added backticks to all parameters
40dbc0b [Neelesh Srinivas Salian] Changed dfs to HDFS, deploy-mode in backticks and updated the master yarn line
9cbc072 [Neelesh Srinivas Salian] Updated a few lines in the Launching Spark on YARN Section
8e8db7f [Neelesh Srinivas Salian] Removed the changes in this commit to help clearly distinguish movement from update
151c298 [Neelesh Srinivas Salian] SPARK-3629: Improvement of the Spark on YARN document
(cherry picked from commit d48e78934a)
Signed-off-by: Sean Owen <sowen@cloudera.com>
Ticket: [SPARK-8639](https://issues.apache.org/jira/browse/SPARK-8639)
fixed minor typos in docs/README.md and docs/api.md
Author: Rosstin <asterazul@gmail.com>
Closes#7046 from Rosstin/SPARK-8639 and squashes the following commits:
6c18058 [Rosstin] fixed minor typos in docs/README.md and docs/api.md
(cherry picked from commit b5a6663da2)
Signed-off-by: Sean Owen <sowen@cloudera.com>
Add `getStaticClass` method in SparkR's `RBackendHandler`
This is a fix for the problem referenced in [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185).
cc shivaram
Author: cafreeman <cfreeman@alteryx.com>
Closes#7001 from cafreeman/branch-1.4 and squashes the following commits:
8f81194 [cafreeman] Add missing license
31aedcf [cafreeman] Refactor test to call an external R script
2c22073 [cafreeman] Merge branch 'branch-1.4' of github.com:apache/spark into branch-1.4
0bea809 [cafreeman] Fixed relative path issue and added smaller JAR
ee25e60 [cafreeman] Merge branch 'branch-1.4' of github.com:apache/spark into branch-1.4
9a5c362 [cafreeman] test for including JAR when launching sparkContext
9101223 [cafreeman] Merge branch 'branch-1.4' of github.com:apache/spark into branch-1.4
5a80844 [cafreeman] Fix style nits
7c6bd0c [cafreeman] [SPARK-8607] SparkR
Test `infer_type` using a more fine-grained approach rather than comparing environments. Since `all.equal`'s behavior has changed in R 3.2, the test became unpassable.
JIRA here:
https://issues.apache.org/jira/browse/SPARK-8662
Author: cafreeman <cfreeman@alteryx.com>
Closes#7045 from cafreeman/R32_Test and squashes the following commits:
b97cc52 [cafreeman] Add `checkStructField` utility
3381e5c [cafreeman] Update SparkSQL Test
cc cafreeman
Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Closes#7022 from shivaram/sparkr-init-hotfix and squashes the following commits:
9178d15 [Shivaram Venkataraman] Fix packages argument, sparkSubmitBinName
(cherry picked from commit c392a9efab)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
…rget versions.
I basically copied the compatibility rules from the top level pom.xml into here. Someone more familiar with all the options in the top level pom may want to make sure nothing else should be copied on down.
With this is allows me to build with jdk8 and run with lower versions. Source shows compiled for jdk6 as its supposed to.
Author: Tom Graves <tgraves@yahoo-inc.com>
Author: Thomas Graves <tgraves@staydecay.corp.gq1.yahoo.com>
Closes#6989 from tgravescs/SPARK-8574 and squashes the following commits:
e1ea2d4 [Thomas Graves] Change to use combine.children="append"
150d645 [Tom Graves] [SPARK-8574] org/apache/spark/unsafe doesn't honor the java source/target versions
(cherry picked from commit e988adb58f)
Signed-off-by: Tom Graves <tgraves@yahoo-inc.com>
Fix for incorrect memory in Spark UI as per SPARK-5768
Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>
Closes#6972 from rekhajoshm/SPARK-5768 and squashes the following commits:
b678a91 [Joshi] Fix for incorrect memory in Spark UI
2fe53d9 [Joshi] Fix for incorrect memory in Spark UI
eb823b8 [Joshi] SPARK-5768: Fix for incorrect memory in Spark UI
0be142d [Rekha Joshi] Merge pull request #3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master
(cherry picked from commit 085a7216bf)
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
`HadoopFsRelation` subclasses, especially `ParquetRelation2` should set its own output format class, so that the default output committer can be setup correctly when doing appending (where we ignore user defined output committers).
Author: Cheng Lian <lian@databricks.com>
Closes#6998 from liancheng/spark-8604 and squashes the following commits:
9be51d1 [Cheng Lian] Adds more comments
6db1368 [Cheng Lian] HadoopFsRelation subclasses should set their output format class
(cherry picked from commit c337844ed7)
Signed-off-by: Cheng Lian <lian@databricks.com>
https://issues.apache.org/jira/browse/SPARK-8567
Author: Yin Huai <yhuai@databricks.com>
Closes#6957 from yhuai/SPARK-8567 and squashes the following commits:
62dff5b [Yin Huai] Increase the timeout.
Conflicts:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
Moved where the X axis' name (#batches) is written in histograms in the spark streaming web ui so the histograms and the axis' name do not overlap.
Author: BenFradet <benjamin.fradet@gmail.com>
Closes#6845 from BenFradet/SPARK-8399 and squashes the following commits:
b63695f [BenFradet] adjusted inner histograms
eb610ee [BenFradet] readjusted #batches on the x axis
dd46f98 [BenFradet] aligned all unit labels and ticks
0564b62 [BenFradet] readjusted #batches placement
edd0936 [BenFradet] moved where the X axis' name (#batches) is written in histograms in the spark streaming web ui
(cherry picked from commit 1173483f3f)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Author: Holden Karau <holden@pigscanfly.ca>
Closes#6928 from holdenk/SPARK-8506-sparkr-does-not-provide-an-easy-way-to-depend-on-spark-packages-when-performing-init-from-inside-of-r and squashes the following commits:
b60dd63 [Holden Karau] Add an example with the spark-csv package
fa8bc92 [Holden Karau] typo: sparm -> spark
865a90c [Holden Karau] strip spaces for comparision
c7a4471 [Holden Karau] Add some documentation
c1a9233 [Holden Karau] refactor for testing
c818556 [Holden Karau] Add pakages to R
(cherry picked from commit 43e66192f4)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
This is https://github.com/apache/spark/pull/6964 for branch 1.4.
Author: Yin Huai <yhuai@databricks.com>
Closes#6966 from yhuai/SPARK-8578-branch-1.4 and squashes the following commits:
9c3947b [Yin Huai] Do not use a custom output commiter when appendiing data.
It's a common mistake that user will put Column in a boolean expression (together with `and` , `or`), which does not work as expected, we should raise a exception in that case, and suggest user to use `&`, `|` instead.
Author: Davies Liu <davies@databricks.com>
Closes#6961 from davies/column_bool and squashes the following commits:
9f19beb [Davies Liu] update message
af74bd6 [Davies Liu] fix tests
07dff84 [Davies Liu] address comments, fix tests
f70c08e [Davies Liu] raise Exception if column is used in booelan expression
(cherry picked from commit 7fb5ae5024)
Signed-off-by: Davies Liu <davies@databricks.com>
fix LabeledPoint parser when there is a whitespace between label and features vector, e.g.
(y, [x1, x2, x3])
Author: Oleksiy Dyagilev <oleksiy_dyagilev@epam.com>
Closes#6954 from fe2s/SPARK-8525 and squashes the following commits:
0755b9d [Oleksiy Dyagilev] [SPARK-8525][MLLIB] addressing comment, removing dep on commons-lang
c1abc2b [Oleksiy Dyagilev] [SPARK-8525][MLLIB] fix LabeledPoint parser when there is a whitespace on specific position
(cherry picked from commit a8031183af)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
the syntax was incorrect in the example in explode
Author: lockwobr <lockwobr@gmail.com>
Closes#6943 from lockwobr/master and squashes the following commits:
3d864d1 [lockwobr] updated the documentation for explode
(cherry picked from commit 4f7fbefb8d)
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
This reverts commit 3348245055.
Reverting because `catch (Exception e) ... throw e` doesn't compile under
Java 6 unless the method declares that it throws Exception.
Author: Holden Karau <holden@pigscanfly.ca>
Closes#6918 from holdenk/SPARK-8498-fix-npe-in-errorhandling-path-in-unsafeshuffle-writer and squashes the following commits:
f807832 [Holden Karau] Log error if we can't throw it
855f9aa [Holden Karau] Spelling - not my strongest suite. Fix Propegates to Propagates.
039d620 [Holden Karau] Add missing closeandwriteoutput
30e558d [Holden Karau] go back to try/finally
e503b8c [Holden Karau] Improve the test to ensure we aren't masking the underlying exception
ae0b7a7 [Holden Karau] Fix the test
2e6abf7 [Holden Karau] Be more cautious when cleaning up during failed write and re-throw user exceptions
(cherry picked from commit 0f92be5b5f)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Author: Hari Shreedharan <hshreedharan@apache.org>
Closes#6910 from harishreedharan/remove-commons-lang3 and squashes the following commits:
9875f7d [Hari Shreedharan] Revert back to Flume 1.4.0
ca35eb0 [Hari Shreedharan] [SPARK-8483][Streaming] Remove commons-lang3 dependency from Flume Sink. Also bump Flume version to 1.6.0
A minor change but one which is (presumably) visible on the public api docs webpage.
Author: Scott Taylor <github@megatron.me.uk>
Closes#6942 from megatron-me-uk/patch-3 and squashes the following commits:
fbed000 [Scott Taylor] test the absolute error in approx doctests
(cherry picked from commit f0dcbe8a7c)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
Author: Holden Karau <holden@pigscanfly.ca>
Closes#6331 from holdenk/SPARK-7781-GradientBoostedTrees.trainRegressor-missing-max-bins and squashes the following commits:
2894695 [Holden Karau] remove extra blank line
2573e8d [Holden Karau] Update the scala side of the pythonmllibapi and make the test a bit nicer too
3a09170 [Holden Karau] add maxBins to to the train method as well
af7f274 [Holden Karau] Add maxBins to GradientBoostedTrees.trainRegressor and correctly mention the default of 32 in other places where it mentioned 100
(cherry picked from commit 164fe2aa44)
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
[[SPARK-8548] Remove the trailing whitespaces from the SparkR files - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-8548)
- This is the result of `lint-r`
https://gist.github.com/yu-iskw/0019b37a2c1167f33986
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes#6945 from yu-iskw/SPARK-8548 and squashes the following commits:
0bd567a [Yu ISHIKAWA] [SPARK-8548][SparkR] Remove the trailing whitespaces from the SparkR files
(cherry picked from commit 44fa7df64d)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
To reproduce that:
```
JAVA_HOME=/home/hcheng/Java/jdk1.8.0_45 | build/sbt -Phadoop-2.3 -Phive 'test-only org.apache.spark.sql.hive.execution.HiveWindowFunctionQueryWithoutCodeGenSuite'
```
A simple workaround to fix that is update the original query, for getting the output size instead of the exact elements of the array (output by collect_set())
Author: Cheng Hao <hao.cheng@intel.com>
Closes#6402 from chenghao-intel/windowing and squashes the following commits:
99312ad [Cheng Hao] add order by for the select clause
edf8ce3 [Cheng Hao] update the code as suggested
7062da7 [Cheng Hao] fix the collect_set() behaviour differences under different versions of JDK
(cherry picked from commit 13321e6555)
Signed-off-by: Yin Huai <yhuai@databricks.com>
https://issues.apache.org/jira/browse/SPARK-8532
This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`.
Author: Yin Huai <yhuai@databricks.com>
Closes#6937 from yhuai/SPARK-8532 and squashes the following commits:
f972d5d [Yin Huai] davies's comment.
d37abd2 [Yin Huai] style.
d21290a [Yin Huai] Python doc.
889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet.
7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value.
d696dff [Yin Huai] Python style.
88eb6c4 [Yin Huai] If mode is "error", do not call mode method.
c40c461 [Yin Huai] Regression test.
(cherry picked from commit 5ab9fcfb01)
Signed-off-by: Yin Huai <yhuai@databricks.com>
[[SPARK-8511] Modify a test to remove a saved model in `regression.py` - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-8511)
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes#6926 from yu-iskw/SPARK-8511 and squashes the following commits:
7cd0948 [Yu ISHIKAWA] Use `shutil.rmtree()` to temporary directories for saving model testings, instead of `os.removedirs()`
4a01c9e [Yu ISHIKAWA] [SPARK-8511][pyspark] Modify a test to remove a saved model in `regression.py`
(cherry picked from commit 5d89d9f00b)
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
Conflicts:
python/pyspark/mllib/tests.py
This is branch 1.4 backport of https://github.com/apache/spark/pull/6888.
Below is the original description.
In earlier versions of Spark SQL we casted `TimestampType` and `DataType` to `StringType` when it was involved in a binary comparison with a `StringType`. This allowed comparing a timestamp with a partial date as a user would expect.
- `time > "2014-06-10"`
- `time > "2014"`
In 1.4.0 we tried to cast the String instead into a Timestamp. However, since partial dates are not a valid complete timestamp this results in `null` which results in the tuple being filtered.
This PR restores the earlier behavior. Note that we still special case equality so that these comparisons are not affected by not printing zeros for subsecond precision.
Author: Michael Armbrust <michaeldatabricks.com>
Closes#6888 from marmbrus/timeCompareString and squashes the following commits:
bdef29c [Michael Armbrust] test partial date
1f09adf [Michael Armbrust] special handling of equality
1172c60 [Michael Armbrust] more test fixing
4dfc412 [Michael Armbrust] fix tests
aaa9508 [Michael Armbrust] newline
04d908f [Michael Armbrust] [SPARK-8420][SQL] Fix comparision of timestamps/dates with strings
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
Author: Michael Armbrust <michael@databricks.com>
Closes#6914 from yhuai/timeCompareString-1.4 and squashes the following commits:
9882915 [Michael Armbrust] [SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings
Author: Cheng Lian <lian@databricks.com>
Closes#6932 from liancheng/spark-8406-for-1.4 and squashes the following commits:
a0168fe [Cheng Lian] Backports SPARK-8406 and PR #6864 to branch-1.4
Ref. #6905
ping yhuai
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes#6929 from viirya/hot_fix_cv_test and squashes the following commits:
b1aec53 [Liang-Chi Hsieh] Hotfix branch-1.4 by removing avgMetrics in CrossValidatorSuite.
Reorganized docs a bit. Added migration guides.
**Q**: Do we want to say more for the 1.3 -> 1.4 migration guide for ```spark.ml```? It would be a lot.
CC: mengxr
Author: Joseph K. Bradley <joseph@databricks.com>
Closes#6897 from jkbradley/ml-guide-1.4 and squashes the following commits:
4bf26d6 [Joseph K. Bradley] tiny fix
8085067 [Joseph K. Bradley] fixed spacing/layout issues in ml guide from previous commit in this PR
6cd5c78 [Joseph K. Bradley] Updated MLlib programming guide for release 1.4
(cherry picked from commit a1894422ad)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
The issue link [SPARK-8379](https://issues.apache.org/jira/browse/SPARK-8379)
Currently,when we insert data to the dynamic partition with speculative tasks we will get the Exception
```
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
Lease mismatch on /tmp/hive-jeanlyn/hive_2015-06-15_15-20-44_734_8801220787219172413-1/-ext-10000/ds=2015-06-15/type=2/part-00301.lzo
owned by DFSClient_attempt_201506031520_0011_m_000189_0_-1513487243_53
but is accessed by DFSClient_attempt_201506031520_0011_m_000042_0_-1275047721_57
```
This pr try to write the data to temporary dir when using dynamic parition avoid the speculative tasks writing the same file
Author: jeanlyn <jeanlyn92@gmail.com>
Closes#6833 from jeanlyn/speculation and squashes the following commits:
64bbfab [jeanlyn] use FileOutputFormat.getTaskOutputPath to get the path
8860af0 [jeanlyn] remove the never using code
e19a3bd [jeanlyn] avoid speculative tasks write same file
(cherry picked from commit a1e3649c87)
Signed-off-by: Cheng Lian <lian@databricks.com>
JIRA: https://issues.apache.org/jira/browse/SPARK-8468
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes#6905 from viirya/cv_min and squashes the following commits:
930d3db [Liang-Chi Hsieh] Fix python unit test and add document.
d632135 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into cv_min
16e3b2c [Liang-Chi Hsieh] Take the negative instead of reciprocal.
c3dd8d9 [Liang-Chi Hsieh] For comments.
b5f52c1 [Liang-Chi Hsieh] Add param to CrossValidator for choosing whether to maximize evaulation value.
(cherry picked from commit 0b8995168f)
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
Author: cody koeninger <cody@koeninger.org>
Closes#6863 from koeninger/SPARK-8390 and squashes the following commits:
26a06bd [cody koeninger] Merge branch 'master' into SPARK-8390
3744492 [cody koeninger] [Streaming][Kafka][SPARK-8390] doc changes per TD, test to make sure approach shown in docs actually compiles + runs
b108c9d [cody koeninger] [Streaming][Kafka][SPARK-8390] further doc fixes, clean up spacing
bb4336b [cody koeninger] [Streaming][Kafka][SPARK-8390] fix docs related to HasOffsetRanges, cleanup
3f3c57a [cody koeninger] [Streaming][Kafka][SPARK-8389] Example of getting offset ranges out of the existing java direct stream api
…f the existing java direct stream api
Author: cody koeninger <cody@koeninger.org>
Closes#6846 from koeninger/SPARK-8389 and squashes the following commits:
3f3c57a [cody koeninger] [Streaming][Kafka][SPARK-8389] Example of getting offset ranges out of the existing java direct stream api
