Don't Wrangle, Guess
Exploring Online Data Interactions
Don't Wrangle, Guess Instead
with
| Students | |||||
|---|---|---|---|---|---|
Poonam |
Will |
Aaron |
Shivang |
Lisa |
Olivia |
| Alumni | ||
|---|---|---|
Ying |
Niccolò |
Arindam |
| Dev |
|---|
Mike |
| External Collaborators | |||
|---|---|---|---|
|
Dieter Gawlick (Oracle) |
Zhen Hua Liu (Oracle) |
Ronny Fehling (Airbus) |
Beda Hammerschmidt (Oracle) |
|
Boris Glavic (IIT) |
Su Feng (IIT) |
Juliana Freire (NYU) |
Wolfgang Gatterbauer (NEU) |
Heiko Mueller (NYU) |
Remi Rampin (NYU) |
A Big Data Fairy Tale
Meet Alice
Alice has a Store
→
Alice's store collects sales data
+
=
Alice wants to use her sales data to run a promotion
→
So Alice loads up her sales data in her trusty database/hadoop/spark/etc... server.
+ ?
... asks her question ...
+ ? →
... and basks in the limitless possibilities of big data.
Why is this a fairy tale?
→
It's never this easy...
CSV Import
Run a SELECT on a raw CSV File
- File may not have column headers
- CSV does not provide "types"
- Lines may be missing fields
- Fields may be mistyped (typo, missing comma)
- Comment text can be inlined into the file
State of the art: External Table Defn + "Manually" edit CSV
Merge Two Datasets
UNION two data sources
- Schema matching
- Deduplication
- Format alignment (GIS coordinates, $ vs €)
- Precision alignment (State vs County)
State of the art: Manually map schema
JSON Shredding
Run a SELECT on JSON or a Doc Store
- Separating fields and record sets:
(e.g.,{ A: "Bob", B: "Alice" }) - Missing fields (Records with no 'address')
- Type alignment (Records with 'address' as an array)
- Schema matching$^2$
State of the art: DataGuide, Wrangler, etc...
Loading requires curation...
Data Curation is Hard!
- It's not the data that's uncertain, it's the interpretation.
- Tagged best-guess evaluation is faster and easier to understand.
- Not committing to one representation allows faster query processing.
Thanks!