spark-instrumented-optimizer/python/pyspark/sql
anabranch 7a7ce272fe [SPARK-16609] Add to_date/to_timestamp with format functions
## What changes were proposed in this pull request?

This pull request adds two new user facing functions:
- `to_date` which accepts an expression and a format and returns a date.
- `to_timestamp` which accepts an expression and a format and returns a timestamp.

For example, Given a date in format: `2016-21-05`. (YYYY-dd-MM)

### Date Function
*Previously*
```
to_date(unix_timestamp(lit("2016-21-05"), "yyyy-dd-MM").cast("timestamp"))
```
*Current*
```
to_date(lit("2016-21-05"), "yyyy-dd-MM")
```

### Timestamp Function
*Previously*
```
unix_timestamp(lit("2016-21-05"), "yyyy-dd-MM").cast("timestamp")
```
*Current*
```
to_timestamp(lit("2016-21-05"), "yyyy-dd-MM")
```
### Tasks

- [X] Add `to_date` to Scala Functions
- [x] Add `to_date` to Python Functions
- [x] Add `to_date` to SQL Functions
- [X] Add `to_timestamp` to Scala Functions
- [x] Add `to_timestamp` to Python Functions
- [x] Add `to_timestamp` to SQL Functions
- [x] Add function to R

## How was this patch tested?

- [x] Add Functions to `DateFunctionsSuite`
- Test new `ParseToTimestamp` Expression (*not necessary*)
- Test new `ParseToDate` Expression (*not necessary*)
- [x] Add test for R
- [x] Add test for Python in test.py

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: anabranch <wac.chambers@gmail.com>
Author: Bill Chambers <bill@databricks.com>
Author: anabranch <bill@databricks.com>

Closes #16138 from anabranch/SPARK-16609.
2017-02-07 15:50:30 +01:00
..
__init__.py [SPARK-16772][PYTHON][DOCS] Fix API doc references to UDFRegistration + Update "important classes" 2016-08-06 05:02:59 +01:00
catalog.py [SPARK-19148][SQL] do not expose the external table concept in Catalog 2017-01-17 12:54:50 +08:00
column.py [SPARK-19403][PYTHON][SQL] Correct pyspark.sql.column.__all__ list. 2017-01-30 18:01:02 +01:00
conf.py [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code 2016-05-23 18:14:48 -07:00
context.py [SPARK-18687][PYSPARK][SQL] Backward compatibility - creating a Dataframe on a new SQLContext object fails with a Derby error 2017-01-13 18:35:51 +08:00
dataframe.py [SPARK-14352][SQL] approxQuantile should support multi columns 2017-02-01 14:11:28 -08:00
functions.py [SPARK-16609] Add to_date/to_timestamp with format functions 2017-02-07 15:50:30 +01:00
group.py [MINOR][PYSPARK][DOC] Fix wrongly formatted examples in PySpark documentation 2016-07-06 10:45:51 -07:00
readwriter.py [SPARK-19239][PYSPARK] Check parameters whether equals None when specify the column in jdbc API 2017-01-17 10:37:29 -08:00
session.py [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped 2017-01-12 20:53:31 +08:00
streaming.py [SPARK-19140][SS] Allow update mode for non-aggregation streaming queries 2017-01-10 17:58:11 -08:00
tests.py [SPARK-16609] Add to_date/to_timestamp with format functions 2017-02-07 15:50:30 +01:00
types.py [SPARK-13748][PYSPARK][DOC] Add the description for explictly setting None for a named argument for a Row 2017-01-07 12:52:41 +00:00
utils.py [MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo 2017-01-04 15:07:29 +00:00
window.py [SPARK-18690][PYTHON][SQL] Backward compatibility of unbounded frames 2016-12-02 17:39:28 -08:00