@@ -467,12 +462,6 @@ end
-
-
<%=
notebook() do
@@ -489,8 +478,17 @@ end
-
+
+
+
@@ -522,14 +520,14 @@ end
Active if: $\forall (x \rightarrow \textbf{@i}) \in \texttt{DynamicReads} : \texttt{InState}[x] = \textbf{@i}$
$\texttt{OutState} = \texttt{InState} + \{\;x \rightarrow \textbf{@i}\;|\;\forall (x \rightarrow \textbf{@i}) \in \texttt{DynamicWrites}\;\}$
- Stale
- Active if: first run or $\exists (x \rightarrow \textbf{@i}) \in \texttt{DynamicReads} : \texttt{InState}[x] \neq \textbf{@i}$
- $\texttt{OutState} = \texttt{InState} + \{\;x \rightarrow \textbf{???}\;|\;\forall x \in \texttt{StaticWrites}\;\}$
-
Runnable
Active if: $\forall x \in \texttt{StaticReads} : \texttt{InState}[x] \neq \textbf{???}$
$\texttt{OutState} = \texttt{InState} + \{\;x \rightarrow \textbf{???}\;|\;\forall x \in \texttt{StaticWrites}\;\}$
+ Stale
+ Active if: first run or $\exists (x \rightarrow \textbf{@i}) \in \texttt{DynamicReads} : \texttt{InState}[x] \neq \textbf{@i}$
+ $\texttt{OutState} = \texttt{InState} + \{\;x \rightarrow \textbf{???}\;|\;\forall x \in \texttt{StaticWrites}\;\}$
+
Unknown
Active otherwise.
$\texttt{OutState} = \texttt{InState} + \{\;x \rightarrow \textbf{???}\;|\;\forall x \in \texttt{StaticWrites}\;\}$
@@ -537,19 +535,19 @@ end
-
- "The Right Tool for the Job: Data-Centric Workflows in Vizier" (Kennedy et. al.; IEEE DEB 2022)
-
-
-
- "Runtime Provenance Refinement for Notebooks" (Deo et. al.; TaPP 2022)
+ 
https://openclipart.com
@@ -580,6 +578,15 @@ end
+
+
+
+
+ "The Right Tool for the Job: Data-Centric Workflows in Vizier" (Kennedy et. al.; IEEE DEB 2022)
+
+
@@ -608,48 +615,156 @@ end
- https://openclipart.com
-
+
+
+
+
+
+
+
+
+
+
+
+
- Naive approach: Pickle
+
- Interop: Define standards
+
-
+
+
+
+
+
+
+
+
-<%#
+
- %>
diff --git a/src/talks/graphics/2024-04-12/DataframeAbstraction.svg b/src/talks/graphics/2024-04-12/DataframeAbstraction.svg
new file mode 100644
index 00000000..f1cda281
--- /dev/null
+++ b/src/talks/graphics/2024-04-12/DataframeAbstraction.svg
@@ -0,0 +1,280 @@
+
+
+
+
diff --git a/src/talks/graphics/2024-04-12/Dependencies-State.svg b/src/talks/graphics/2024-04-12/Dependencies-State.svg
new file mode 100644
index 00000000..07b36255
--- /dev/null
+++ b/src/talks/graphics/2024-04-12/Dependencies-State.svg
@@ -0,0 +1,229 @@
+
+
+
+
diff --git a/src/talks/graphics/2024-04-12/NotebookExtensions.svg b/src/talks/graphics/2024-04-12/NotebookExtensions.svg
new file mode 100644
index 00000000..63f00831
--- /dev/null
+++ b/src/talks/graphics/2024-04-12/NotebookExtensions.svg
@@ -0,0 +1,261 @@
+
+
+
+
diff --git a/src/talks/graphics/logos/breadcrumb.png b/src/talks/graphics/logos/breadcrumb.png
new file mode 100644
index 00000000..86de55c3
Binary files /dev/null and b/src/talks/graphics/logos/breadcrumb.png differ
We need to be able to recover the kernel to any state.
-Why have only one kernel?
- -🤷
-When is parallelism allowed?
-When is a cell runnable?
+Why have only one kernel?
+ +🤷
+Parallelism
+-
+
- When is parallelism allowed? +
- When is a cell runnable? +
Serial
- -Parallel
- +
+
+ Serial
+ +
+
Parallel
+ +Microkernel Notebooks
🤷
+Vizier Demo
+Repeatable Spreadsheet Dataframe Editing
@@ -608,48 +615,156 @@ end
- ... but this requires migrating state.
+... but this requires migrating state... across languages.
State Management
+ +Approach 1: Pickle
+ +Python's native serialization support.
+ +-
+
- The Good +
- Easy +
- The Bad +
- Not everything is serializable†+
- Limited compatibility with ¬Python +
- Expensive for e.g., dataframes +
+
+
+
+ Approach 2: Json
+ +Standard data interchange format.
+ +-
+
- The Good +
- Easy +
- Near universal platform compatibility +
- The Bad +
- Even less state is supported +
- Even more expensive for e.g., dataframes +
- Limited support for nuanced types (e.g., dates) +
+
+
+
+ Approach 3: Arrow, Shapefile, Parquet, NPY
+ +Specialized formats for specific datatypes.
+-
+
- The Good +
- High Performance +
- Precise, Well Typed +
- The Bad +
- Only one type of state is supported +
+
+
+
+ Vizier (Now)
+ +Vizier-level Typing.
-
-
- State needs to be checkpointed out of the process that created it. -
- State needs to be restored into the cell that is about to consume it. +
- Simple Data: JSON +
- Typed Data: Standard JSON Encoding +
- Special Data: 'Active' Data +
- Fallback: Pickle
Active Data
- ... but pickle doesn't allow interop - ... but pickle doesn't always work (e.g., for 'File' objects) +Datasets, Functions/Classes, etc...
+-
+
- One concept, Many physical representations (Arrow, Parquet, CSV).
+
-
+
- A cell interpreter may not support a representation. +
- Generating a standard representation can be expensive. +
+ - State (e.g., Datasets) can get big.
+
-
+
- An interpreter may not want/need to load the entire state. +
- Versioning all checkpoints becomes infeasible. +
+
Desiderata
- - Primitive Values (int, float, date, etc...) - - Collection Types (map, list, etc...) - - Libraries - - Function [Challenge: Chained Dependencies] - - Dataframe/Series [Challenge: These are BIG] +
+
+ An abstraction that...
+-
+
- ... represents the concept. +
- ... allows on-demand conversion between representations. +
- ... allows partial in-store interactions. +
- ... allows incremental changes. +
Vizier's artifact store provides a thin wrapper around standards compliant libraries (e.g., Apache Spark).
"Active" Data
+... but it's a lot of special case code.
+Generalizing Active Data
+(future work)
+ +-
+
- What's the right abstraction? +
- Efficient type coercion (without $N^2$) +
- Microservice RPCs +
- Caching Strategies +
Questions?
+
https://vizierdb.info
-Mike Brachmann, Boris Glavic, Nachiket Deo, Juliana Freire, Heiko Mueller, Sonia Castello, Munaf Arshad Qazi, William Spoth, Poonam Kumari, Soham Patel, and more...
+Mike Brachmann, Boris Glavic, Nachiket Deo, Juliana Freire, Heiko Mueller, Sonia Castello, Munaf Arshad Qazi, William Spoth, Poonam Kumari, Nicholas Brown, Soham Patel, Thomas Slowe, and more...
+ + +
+ Supported by:
+ 
+ 
+
+
+