26 lines
2.2 KiB
TeX
26 lines
2.2 KiB
TeX
%!TEX root=../main.tex
|
|
|
|
Workflow systems \OK{cite several systems: Vistrails, etc...} help users to break complex tasks like ETL processes, model-fitting, and more, into a series of smaller steps.
|
|
Users explicitly declare dependencies between steps, permitting parallel execution of mutually independent steps.
|
|
A recent trend in industry has been to instead encode such tasks through computational notebooks like Jupyter or Zeppelin.
|
|
Notebooks likewise allow users to declare tasks as a series of steps, but do not require the user to explicitly declare dependencies.
|
|
The entire notebook executes serially, in effect, as a single python script.
|
|
|
|
\begin{figure}
|
|
FIGURE
|
|
\caption{The number of python cells in a notebooks scraped from github~\cite{DBLP:journals/ese/PimentelMBF21} against the number of sequential steps required}
|
|
\label{fig:parallelismSurvey}
|
|
\end{figure}
|
|
|
|
In this paper, we present a novel \emph{coarse-grained} dataflow provenance model --- a hybrid of classical workflow and dataflow provenance --- that permits not only parallel execution, but also incremental re-execution of computational notebooks.
|
|
We outline the implementation of this provenance model into an existing workflow system named Vizier~\cite{brachmann:2020:cidr:your,brachmann:2019:sigmod:data}, and address several of the challenges that arise when parallelizing notebooks.
|
|
|
|
\subsection{Potential for Improvement}
|
|
To assess the potential for improvement, we conducted a preliminary survey on an archive of Jupyter notebooks scraped from Github by Pimentel et. al.~\cite{DBLP:journals/ese/PimentelMBF21}.
|
|
Our survey included only notebooks using a python kernel and known to execute successfully; A total of 800\OK{fill in the exact number} notebooks met these criteria.
|
|
We used the python \texttt{ast} module to construct an inter-cell dataflow graph (e.g., using the methodology of \OK{citations}).
|
|
As a proxy measure for potential speedup, we considered the depth of this graph in relation to the total number of python cells in the notebook.
|
|
\Cref{fig:parallelismSurvey} relates these measures in a XXX.
|
|
Although XXX percent of the notebooks do require sequential execution, as many as XXX percent can XXX.
|
|
|