+
+ 
+
+
+ 
+ VizierDB
+
+ +
Your notebook is not crumby enough, REPLace it
++
Michael Brachmann, + William Spoth, + Oliver Kennedy, + Boris Glavic, + Heiko Mueller, + Sonia Castelo, + Carlos Bautista, + Juliana Freire
+ +Demo
+
+
+ VizierDB
+
+
-
+
- Automatic Refresh & Dependency Management +
- Caveats: Tracking Errors and Assumptions +
- Hybrid Notebook/Spreadsheet +
- History & Version Management +
- Polyglot & Multimodal +
Data Errors Suck
+
+
+
+ 
+ →
+
+ 
+ →
+ 
+
+ ↓
+ ↓
+
+ Assumption
+ ≠
+ Assumption
+
Communicating Assumptions
+-
+
- README.txt +
- Wikis +
- Notebooks +
+
+ How to Deal With Errors? (Idealized)
++
-
+
- Detect Data Errors + +
- Repair Data Errors + +
- Live happily ever after! + +
+ How to Deal With Errors? (Actually)
+-
+
- Detect Data Errors + +
- false positives and negatives! +
- errors may be subjective! +
- not all errors affect analysis! +
- Repair Data Errors + +
- gold standard unknown! +
- correctness of repair often not verifiable! +
-
+
-
+
Takeaway: Need to embrace uncertainty!
+-
+
- However, currently user has to manually: +
- Document errors and repairs +
- Document assumptions +
- Determine impact on analysis results +
-
+
What is a Caveat?
+ +Annotation on data value(s) that indicates a concern, e.g.,
+-
+
- The value is assumed to be erroneous +
- The value is result of a heuristic repair +
- The value is an outlier +
Caveats consist of ...
+ +-
+
- A reference to the operation that detected/caused the caveat (e.g., a repair) +
- Documentation (free text or structured) +
- A set of possible alternative values (optional) +
+
+
+
+ caveat(race_ethnicity,
+ 'Unexpected race_ethnicity: ' & race_ethnicity)
+
+
+
+
+
+
+
+ CASE WHEN race_ethnicity NOT IN ('Black Non-Hispanic', /* ... */)
+
+ THEN caveat(race_ethnicity,
+ 'Unexpected race_ethnicity: ' & race_ethnicity)
+
+ ELSE race_ethnicity
+
+
+
+
+ SELECT
+ CASE WHEN race_ethnicity NOT IN ('Black Non-Hispanic', /* ... */)
+
+ THEN caveat(race_ethnicity,
+ 'Unexpected race_ethnicity: ' & race_ethnicity)
+
+ ELSE race_ethnicity
+
+ END, /* ... */
+ FROM R
+ |
+
+ →
+ 
+ →
+ |
+  |
+
| ↑ | ++ | ↑ | +
| Caveats | +→ | +Caveats | +
Propagation
+Can twiddling the caveatted value change the output?
+ +Caveats on
Some conditions may apply
+Sloooow!
+
+ +
+
+ Is a value caveatted?
+≡ Certain answers in incomplete databases
+(coNP-complete)
+Conservative Approximation
+ +
+ Correctness of SQL Queries on Databases with Nulls.
+ Paolo Guagliardo, Leonid Libkin
+
+ Uncertainty Annotated Databases - A Lightweight Approach for Approximating Certain Answers
+ Su Feng, Aaron Huber, Boris Glavic, Oliver Kennedy
+
-
+
- Unmarked rows are guaranteed to be caveat-free. +
- Marked rows might not be caveatted. +
+ Uncertainty Annotated Databases - A Lightweight Approach for Approximating Certain Answers
+ Su Feng, Aaron Huber, Boris Glavic, Oliver Kennedy
+
Add and maintain a binary "has caveat"
column for each row/column.
Enumerating Caveats
+ +-
+
- Static Analysis +
- Which caveats could possibly affect the element? + +
- Dynamic Analysis +
- Which specific caveats affect the element? +
-
+
- Oliver +
- I have this great tool for tracking assumptions! +
- Data Scientist +
- Super! How does it work? +
- Oliver +
- Well you just write a SQL query... +
- Data Scientist +
- ... +
+ Vizual
+Spreadsheet Operations → SQL DDL / SQL DML
+-
+
- Edit Cell A3 to 'foo' +
- UPDATE R SET A = 'foo' WHERE ROWID = 3; +
- Insert Row +
- INSERT INTO R() VALUES (); +
- Insert Column `bar` +
- ALTER TABLE R ADD COLUMN `bar`; +
This gives us an edit history in DDL/DML.
+Caveats on DDL/DML
+DML → SQL
+
+ Using Reenactment to Retroactively Capture Provenance for Transactions
+ Bahareh Sadat Arab, Dieter Gawlick, Vasudha Krishnaswamy, Venkatesh Radhakrishnan, Boris Glavic
+
DDL → SQL
+
+ Graceful database schema evolution: the PRISM workbench
+ Carlo Curino, Hyun Jin Moon, Carlo Zaniolo
+
+ UPDATE R SET A = 'foo' WHERE ROWID = 3;
+
+ SELECT CASE ROWID
+ WHEN 3 THEN 'foo'
+ ELSE A END AS A,
+ B, C, /* ... */
+ FROM R
+
+ INSERT INTO R() VALUES ();
+
+ SELECT * FROM R
+ UNION ALL
+ SELECT NULL AS A, NULL AS B,
+ NULL AS C, /* ... */
+
+ ALTER TABLE R ADD COLUMN `bar`;
+
+ SELECT *, NULL as `bar` FROM R;
+
+
+ https://vizierdb.info
+
+
+ $> pip3 install --user vizier-webapi
+ $> vizier
+
+
+
+ [https://]VizierDB[.info]
+
+ 
+
+ +
+ Michael Brachmann, + William Spoth, + Oliver Kennedy, + Boris Glavic, + Heiko Mueller, + Sonia Castelo, + Carlos Bautista, + Juliana Freire
++ +
+ Ying Yang, + Su Feng, + Poonam Kumari, + Aaron Huber, + Niccolò Meneghetti, + Arindam Nandi, + Shivang Agarwal, + Olivia Alphonse, + Lisa Lu, + Gourab Malhotra, + Remi Rampin
++
Vizier is supported by NSF Awards ACI-1640864 and IIS-1750460 and gifts from Oracle
+ + + +Bonus Slides
+
+ CREATE VIEW survey_responses AS
+ SELECT language,
+ CASE WHEN CAST(salary AS float) IS NULL THEN
+ caveat(NULL, 'Could not cast [ '&salary&' ] to float.')
+ ELSE CAST(salary AS float) END AS salary
+ FROM raw_csv_data;
+
+ CREATE VIEW survey_responses AS
+ SELECT language, CAST(salary AS float) AS salary,
+ FALSE AS _caveat_field_language,
+ CAST(salary as float) IS NULL AS _caveat_field_salary
+ FALSE AS _caveat_row
+ FROM raw_csv_data;
+
+ SELECT salary
+ FROM survey_responses
+ WHERE language = 'Scala'
+
+ SELECT salary,
+ _caveat_field_salary AS _caveat_field_salary,
+ _caveat_row AND _caveat_field_language AS _caveat_row
+ FROM survey_responses
+ WHERE language = 'Scala'
+
+ SELECT AVG(salary) AS salary
+ FROM survey_responses
+
+ SELECT AVG(salary),
+ GROUP_OR(_caveat_field_salary
+ OR _caveat_row) AS _caveat_field_salary,
+ FALSE AS _caveat_row
+ FROM survey_responses
+
+ SELECT language, AVG(salary) AS salary
+ FROM survey_responses
+ GROUP BY language
+
+ SELECT GROUP_OR(_caveat_field_language)
+ FROM survey_responses
+ Can often be evaluated statically.
+If GROUP BY has caveats
+ +
+ SELECT language, AVG(salary) AS salary
+ FALSE AS _caveat_field_language
+ TRUE AS _caveat_field_salary
+ GROUP_AND(_caveat_field_language OR
+ _caveat_row) AS _caveat_row
+ FROM by_language
+ GROUP BY language
+ If no GROUP BY caveats
+ +
+ SELECT language, AVG(salary) AS salary
+ FALSE AS _caveat_field_language
+ GROUP_OR(_caveat_field_salary,
+ _caveat_row) AS _caveat_field_salary
+ GROUP_AND(_caveat_row) AS _caveat_row
+ FROM by_language
+ GROUP BY language
+