ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark
History
Davies Liu 6183b5e2ca [SPARK-6911] [SQL] improve accessor for nested types 
Support access columns by index in Python:
```
>>> df[df[0] > 3].collect()
[Row(age=5, name=u'Bob')]
```

Access items in ArrayType or MapType
```
>>> df.select(df.l.getItem(0), df.d.getItem("key")).show()
>>> df.select(df.l[0], df.d["key"]).show()
```

Access field in StructType
```
>>> df.select(df.r.getField("b")).show()
>>> df.select(df.r.a).show()
```

Author: Davies Liu <davies@databricks.com>

Closes #5513 from davies/access and squashes the following commits:

e04d5a0 [Davies Liu] Update run-tests-jenkins
7ada9eb [Davies Liu] update timeout
d125ac4 [Davies Liu] check column name, improve scala tests
6b62540 [Davies Liu] fix test
db15b42 [Davies Liu] Merge branch 'master' of github.com:apache/spark into access
6c32e79 [Davies Liu] add scala tests
11f1df3 [Davies Liu] improve accessor for nested types
2015-04-16 17:33:57 -07:00
..
ml [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
mllib [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
sql [SPARK-6911] [SQL] improve accessor for nested types 2015-04-16 17:33:57 -07:00
streaming [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
__init__.py [SPARK-4172] [PySpark] Progress API in Python 2015-02-17 13:36:43 -08:00
accumulators.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
broadcast.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
cloudpickle.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
conf.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
context.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
daemon.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
files.py [SPARK-3309] [PySpark] Put all public API in __all__ 2014-09-03 11:49:45 -07:00
heapq3.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
java_gateway.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
join.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
profiler.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
rdd.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
rddsampler.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
resultiterable.py [SPARK-3074] [PySpark] support groupByKey() with single huge key 2015-04-09 17:07:23 -07:00
serializers.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
shell.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
shuffle.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
statcounter.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
status.py [SPARK-4172] [PySpark] Progress API in Python 2015-02-17 13:36:43 -08:00
storagelevel.py [SPARK-3417] Use new-style classes in PySpark 2014-09-08 15:45:36 -07:00
tests.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00
traceback_utils.py [SPARK-1087] Move python traceback utilities into new traceback_utils.py file. 2014-09-15 19:28:17 -07:00
worker.py [SPARK-4897] [PySpark] Python 3 support 2015-04-16 16:20:57 -07:00