233 lines
15 KiB
TeX
233 lines
15 KiB
TeX
%!TEX root=../main.tex
|
|
\section{Evaluating \systemlang}
|
|
\label{sec:queryEvaluation}
|
|
|
|
A typical optimizer is typically defined as a collection of rules of the form $\tuple{\matcher_i, \expression_i} \in \matcherdom \times \expressiondom$\footnote{
|
|
A typical optimizer has many `batches' of such rules, applied in sequence.
|
|
As the generalization to multiple batches is straightforward, we assume only one batch.
|
|
}.
|
|
To perform one rewrite step, the optimizer searches for the first subtree with a successful match from $\bigcup_i \query{\matcher_i}(\db)$.
|
|
Given a subtree $\constant$ matched by $\matcher_i$ and the resulting scope $\scope$, the step results in an updated AST $\db' = \db[\constant \backslash \expression_i(\scope)]$ (where $\db[\constant \backslash \constant']$ indicates $\db$ with subtree $\constant$ replaced by subtree $\constant'$).
|
|
% \DB{$\db[\constant \backslash \expression_i(\scope)]$ this supports match pattern that have an experssion evaluation right? Not all match patterns.}
|
|
The optimizer repeats this process until $\db$ converges to a fixed point, or a timeout or threshold number of steps is reached.
|
|
|
|
In this section, we focus on optimizing the search for matched subtrees, and return to the rewrite step in \Cref{sec:updates}.
|
|
We first outline a baseline query evaluation strategy, and then explore techniques for optimized evaluation.
|
|
|
|
To begin, we note that while $\rewritematcher{\var}{\cdot}$ emits a wide (`flat') join, many of the relation atoms being joined mimic the behavior of other relational operators, per the following \textbf{join-elimination rewrites}:
|
|
\begin{align*}
|
|
Q \bowtie \inbrackets{\expression}
|
|
& \equiv \sigma_{\expression}(Q)
|
|
& \textbf{if } \schemaOf{Q} \subseteq \varsOf{\expression}\\
|
|
%%%%%%%%%%%%%%%%%%%%
|
|
Q \bowtie \inbrackets{\var = \expression}
|
|
& \equiv \pi_{\schemaOf{Q}, \var \leftarrow \expression}(Q)
|
|
& \textbf{if } \schemaOf{Q} \subseteq \varsOf{\expression}\\
|
|
&& \textbf{and } \var \not\in \schemaOf{Q}\\
|
|
%%%%%%%%%%%%%%%%%%%%
|
|
Q \bowtie [\var = \nodelabel(\var_1, \ldots, \var_n)]
|
|
& \equiv \expandop_{\var \rightarrow \nodelabel(\var_1, \ldots, \var_n)}(Q)
|
|
& \textbf{if } \inset{\var, \var_1, \ldots \var_n} \hspace{7mm}\\
|
|
&&\cap\; \schemaOf{Q} = \inset{\var}
|
|
\end{align*}
|
|
We introduce the new operator $\expandop$ in the third equivalence shortly.
|
|
The first two equivalences follow from the fact that $\varsOf{\expression}$ is a key for both $\inbrackets{e}$ and $\inbrackets{\var = \expression}$; the join is a foreign-key join.
|
|
In the former case, tuples that fail the predicate have no join partner and are filtered out, like a selection.
|
|
In the latter, the join is guaranteed to find a foreign key, and the resulting schema is extended by $\var$, like projection.
|
|
The constraints on $\schemaOf{Q}$ enforce query safety.
|
|
|
|
Match atoms similarly act as a foreign key join, but simultaneously filter (like select) and extend the schema (like project): (i) only tuples where $\var$ is a $\nodelabel$-typed AST-node find join partners, but (ii) the resulting schema is extended by $\inset{\var_1, \ldots, \var_n}$.
|
|
We capture this behavior in a new operator named \textbf{expand} (denoted $\expandop$);
|
|
The expand operator is similar to the Unnest operator in nested relational algebra~\cite{DBLP:conf/pods/JaeschkeS82}; but never emits more than one tuple per input.
|
|
|
|
\begin{figure}
|
|
\begin{tikzpicture}[node distance=3mm]
|
|
\node (source) {$\db(r)$};
|
|
\node (filter) [above=of source] {$\expandop_{r\rightarrow{\textbf{Filter}(\texttt{cond}, b)}}$};
|
|
\node (project) [above left=of filter] {$\expandop_{b\rightarrow{\textbf{Project}(\texttt{tgt}, \texttt{child})}}$};
|
|
\node (check1) [above=of project] {$\sigma_{\textsc{det}(\texttt{tgt})}$};
|
|
\node (check2) [above=of filter] {$\sigma_{\texttt{cond} = true}$};
|
|
\node (join) [above right=of filter] {$\expandop_{b\rightarrow \textbf{Join}(c, d)}$};
|
|
\node (exp) [above=of join] {$\pi_{*, v \gets \textsc{refs}(\texttt{cond})}$};
|
|
\node (leftj) [above left=of exp] {$\sigma_{\textsc{outputs}(c) \subseteq v}$};
|
|
\node (rightj) [above=of exp] {$\sigma_{\textsc{outputs}(d) \subseteq v}$};
|
|
\draw[very thick,draw=red!50!black] (source) -> (filter);
|
|
\draw[very thick,draw=red!50!black] (filter) -> (project);
|
|
\draw[very thick,draw=red!50!black] (project) -> (check1);
|
|
\draw (filter) -> (check2);
|
|
\draw (filter) -> (join);
|
|
\draw (join) -> (exp);
|
|
\draw (exp) -> (leftj);
|
|
\draw (exp) -> (rightj);
|
|
\end{tikzpicture}
|
|
\caption{Example rewrite execution plan. Thicker, red lines are for the running example.}
|
|
\label{fig:executionPlan}
|
|
\end{figure}
|
|
|
|
\begin{example}
|
|
Continuing \Cref{ex:rewrite}, $\db(r) \bowtie \inbrackets{r = \textbf{Filter}(a, b)}$ meets the constraints of the first join elimination rewrite and can be rewritten to
|
|
$\expandop_{r\rightarrow\textbf{Filter}(a, b)}(\db(r))$.
|
|
As before, one result is produced for every \textbf{Filter}-typed AST subtree of $\db$.
|
|
\end{example}
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\subsection{Selecting an Execution Plan}
|
|
% \DB{Should we call this section differently? Evaluation Workflow or Rewrite Workflow? In my head its conflicting with the physical plan thats evaluated. Ignore if its just me.}
|
|
Starting with a join over atomic relations allows us to explore the space of evaluation plans by leveraging the associativity and commutativity of join.
|
|
\Cref{alg:makePlan} outlines a simple, greedy strategy for eliminating joins.
|
|
Specifically, given a query of the form: $\atom_1 \bowtie \ldots \bowtie \atom_n = \rewritematcher{\var}{\matcher}$, the return value of $\textsc{MakePlan}(\inset{\atom_1, \ldots, \atom_n}, \db(\var))$ is equivalent to $\query{\matcher}(\db)$\footnote{
|
|
Matchers with disjunctions rewrite to queries with unions; A more robust optimization strategy is likely possible, but for the purposes of this paper we rely on distributivity to produce a set of union-free queries, each individually passed to \textsc{MakePlan}.
|
|
}
|
|
$\textsc{MakePlan}$ proceeds in three steps:
|
|
(i) $\textsc{EnumerateCandidates}$ selects atoms that are candidates for a join elimination rewrite according to the relevant constraint on $\schemaOf{Q}$ (listed with the rewrites).
|
|
(ii) $\textsc{PickCandidate}$ selects one candidate from the set of rewrites. We discuss this step in greater depth below
|
|
(iii) $\textsc{Rewrite}$ applies the selected join elimination rewrite.
|
|
In summary, the key challenge of selecting an execution plan is (greedily) selecting an order in which to resolve the atoms.
|
|
|
|
\begin{algorithm}
|
|
\caption{\textsc{MakePlan}$(A, Q)$}
|
|
\label{alg:makePlan}
|
|
\begin{algorithmic}
|
|
\Require $A = \inset{\atom_1, \ldots, \atom_n} \subseteq \atomdom$: A set of target atoms
|
|
\Require $Q$: A query
|
|
\Ensure $Q': $ A query equivalent to $Q \bowtie \atom_1 \bowtie \ldots \bowtie \atom_n$
|
|
\State \textbf{if} {$|A| = 0$} \textbf{then} \Return Q
|
|
\State $\texttt{candidates} = \textsc{EnumerateCandidates}(Q, A)$
|
|
\State $\atom = \textsc{PickCandidate}(\texttt{candidates})$
|
|
\State \Return \textsc{MakePlan}$(\;A - \inset{\atom},\; \textsc{Rewrite}(Q \bowtie \atom)\;)$
|
|
\end{algorithmic}
|
|
\end{algorithm}
|
|
|
|
The optimal strategy for \textsc{PickCandidate} mirrors the selection predicate ordering problem in a typical database:
|
|
We want to prioritize atoms that have a low selectivity and a low cost.
|
|
We leave a more thorough cost-based optimizer to future work, and for now adopt the following heuristic order over atoms:
|
|
(i) match atoms (which are inexpensive), (ii) test atoms (which have nonzero selectivity), (iii) binding atoms (which are neither).
|
|
% We note that universal and empty atoms may be optimized away, and AST atoms only appear at the query root.
|
|
|
|
\begin{example}
|
|
Consider the augmented version of our example pattern from \Cref{ex:sideEffects}.
|
|
Schema constraints enforce a specific execution plan --- \textsc{EnumerateCandidates} only returns a single candidate for each call.
|
|
The output of \textsc{MakePlan} on this pattern is illustrated in \Cref{fig:executionPlan} (bold, red lines).
|
|
\end{example}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\subsection{Merging Plans}
|
|
|
|
An optimizer is simultaneously interested in matching multiple patterns.
|
|
We find an appropriate optimization opportunity in stream processing systems (e.g., Aurora/Borealis\cite{DBLP:conf/cidr/AbadiABCCHLMRRTXZ05}), where multiple simultaneous streams are rewritten to share overlapping computations~\cite{DBLP:journals/ieeecc/KremienKM93}.
|
|
\Cref{alg:makeSharedPlan} generalizes \Cref{alg:makePlan} to detect and leverage such opportunities, rewriting multiple atom sets in parallel.
|
|
|
|
\begin{algorithm}
|
|
\caption{$\textsc{MakeSharedPlan}(Q, \vec A)$}
|
|
\label{alg:makeSharedPlan}
|
|
\begin{algorithmic}
|
|
\Require $Q$: A query
|
|
\Require $\vec A$: A vector of target atom sets $A_i$ as in \Cref{alg:makePlan}.
|
|
\Ensure $\vec Q'$: A vector of queries, where $Q_i'$ is as in \Cref{alg:makePlan}.
|
|
\State \textbf{if } {$|\vec A| = 0$} \textbf{then} \Return $[]$
|
|
\For{$A_i \in \vec A$}
|
|
\State $C_i = \textsc{EnumerateCandidates}(Q, A_i)$
|
|
\EndFor
|
|
\State $\atom = \textsc{PickCandidate}(\vec C)$
|
|
\State $Q' = \textsc{Rewrite}(Q \bowtie \atom)$
|
|
\State \Return $
|
|
\textsc{MakeSharedPlan}(Q', \inbrackets{\;
|
|
A_i \;\left|\; A_i \in \vec A \wedge \atom \in A_i\right.
|
|
})$ \\
|
|
\hspace{15mm} $\cup \;
|
|
\textsc{MakeSharedPlan}(Q, \inbrackets{\;
|
|
A_i \;\left|\; A_i \in \vec A \wedge \atom \not \in A_i\right.
|
|
})$
|
|
\end{algorithmic}
|
|
\end{algorithm}
|
|
|
|
As before, the key challenge is abstracted by $\textsc{PickCandidate}$, which now selects from a vector of candidate sets.
|
|
Our preliminary implementation buckets candidate atoms into the same three priority groups, and selects an the atom that appears the most frequently in the highest priority bucket.
|
|
|
|
\begin{example}
|
|
Consider the following additional match patterns
|
|
{\footnotesize \begin{align*}
|
|
&\textbf{Filter}(cond, child) \wedge (cond = true)\\
|
|
&\textbf{Filter}(cond, Join(lhs, rhs)) \wedge (v \leftarrow \textsc{refs}(\texttt{cond})) \wedge (\textsc{outputs}(lhs) \subseteq v)\\
|
|
&\textbf{Filter}(cond, Join(lhs, rhs)) \wedge (v \leftarrow \textsc{refs}(\texttt{cond})) \wedge (\textsc{outputs}(rhs) \subseteq v)
|
|
\end{align*}}
|
|
These test, respectively, for tautological filters and joins that admit filter push-down.
|
|
The rewrite into \systemlang, and subsequent variable-elimination optimizations are left as an exercise for the reader, but \Cref{fig:executionPlan} shows the final work-shared plan.
|
|
\end{example}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\subsection{Limit-1 Execution}
|
|
|
|
Naively evaluating the query entails computing the union of all match queries in full, while we only want one tuple at a time.
|
|
Volcano style query execution is limited in its ability to share work from common subplans without expensive materialization of intermediate results.
|
|
Meanwhile, `push'-based streaming style query execution requires asynchronicity.
|
|
Here, we develop a non-asynchronous push-based evaluation strategy that can efficiently shortcut execution once a single result is produced.
|
|
|
|
We assume, as per above, that all match pattern queries have been rewritten into a join-free DAG of project, select, and expand operators, rooted at a single AST atom.
|
|
We write $\parentsOf{Q}$ to denote the set of subplans in the DAG with $Q$ as a child.
|
|
|
|
\begin{algorithm}
|
|
\caption{$\textsc{PushParents}(Q, t)$}
|
|
\label{alg:pushParents}
|
|
\begin{algorithmic}
|
|
\Require $Q$: A query subplan
|
|
\Require $t$: A tuple
|
|
\Ensure A matched tuple or $\nullresult$
|
|
\State \textbf{if} $\parentsOf{Q} = \emptyset$ \textbf{then return} $t$
|
|
\Comment{Match found, terminate}
|
|
\For{$Q' \in \parentsOf{Q}$}
|
|
\State $t' \gets \textsc{PushPlan}(Q, t)$
|
|
\State \textbf{if} {$t' \neq \nullresult$} \textbf{then} \Return $t'$
|
|
\EndFor
|
|
\State \Return $\nullresult$
|
|
\end{algorithmic}
|
|
\end{algorithm}
|
|
|
|
\begin{algorithm}
|
|
\caption{$\textsc{PushPlan}(Q, t)$}
|
|
\label{alg:pushPlan}
|
|
\begin{algorithmic}
|
|
\If{$Q$ \textbf{is} $\sigma_{\expression}$}
|
|
\State \textbf{if} $\expression(t) = \top$ \textbf{then return} $\textsc{PushParents}(Q, t)$
|
|
\State \textbf{else return} $\nullresult$
|
|
\ElsIf{$Q$ \textbf{is} $\pi_{\var_1 \gets \expression_1, \ldots, \var_n \gets \expression_n}$}
|
|
\State \Return $\textsc{PushParents}(Q, \tuple{\var_1: \expression_1(t),\; \ldots,\; \var_n: \expression_n(t)})$
|
|
\ElsIf{$Q$ \textbf{is} $\expandop_{\var \rightarrow \nodelabel(\var_1, \ldots, \var_n)}$}
|
|
\If{$t[\var]$ \textbf{is} $\nodelabel(\constant_1, \ldots, \constant_n)$}
|
|
\State \Return $\textsc{PushParents}(Q, t \cup \tuple{\var_1:\constant_1,\;\ldots,\;\var_n:\constant_n})$
|
|
\EndIf
|
|
\State \textbf{else return} $\nullresult$
|
|
\EndIf
|
|
\end{algorithmic}
|
|
\end{algorithm}
|
|
|
|
\Cref{alg:pushParents,alg:pushPlan} define a mutually recursive execution strategy that terminates when it reaches the top of a plan.
|
|
\textsc{PushParents} iteratively explores each ancestor of $Q$ until it finds a match, while \textsc{PushPlan} defines semantics for each operator, returning when an operator filters out a tuple.
|
|
To find matches, we invoke $\textsc{PushParents}(\db(\var), \tuple{\var: \constant})$ for each $\constant \in \subtreesOf{\db}$ until a match is discovered.
|
|
If no match is found, the optimizer has reached a fixed point.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\subsection{Storage and Updates}
|
|
\label{sec:updates}
|
|
|
|
Having established a language for querying the subtrees of $\db$, we next turn to the question of how these subtrees are stored.
|
|
The optimizer interacts with $\db$ through three access patterns:
|
|
(i) Full subtree scan ($\db(\var)$),
|
|
(ii) Node expansion ($\expandop_{\var \rightarrow \ell(\var_1, \ldots, \var_n)}$), and
|
|
(iii) Subtree replacement $\db' = \db[\constant \backslash \constant]$.
|
|
|
|
In \cite{balakrishnan:2021:sigmod:treetoaster}, we contrasted two different storage layouts for ASTs.
|
|
In one, each node of a subtree was stored as a tuple in a relation, with one relation per AST node type (i.e., $\nodelabel$).
|
|
In the other, we superimposed relational semantics over an existing AST.
|
|
Both approaches had competitive performance, but the relational representation had significant storage overheads.
|
|
|
|
We implement the latter approach for \systemlang, and summarize key features of it here.
|
|
A naive realization of this representation stores AST nodes and variable-sized constants as as heap-allocated objects.
|
|
AST nodes are stored as a tuple of their type and the fields, with fields containing fixed-size constants inlined, and child AST nodes stored by reference.
|
|
|
|
Enumerating the subtrees is trivial on this structure, requiring only a series of pointer traversals.
|
|
Node expansion is likewise viable on this naive representation, as variables holding AST nodes necessarily store references to the node and its fields on the heap.
|
|
Subtree replacement requires only maintaining a lookup table of parent nodes, and in the case of Spark, an indirection layer~\cite{balakrishnan:2019:dbpl:fluid} to work around immutability constraints.
|