pass through S5

circuits-model-runtime.tex

@@ -5,31 +5,33 @@ In this section, we consider a couple of generalizations/corollaries of our resu
 \subsection{Lineage circuits}
 \label{sec:circuits}
 
-In~\Cref{sec:semnx-as-repr}, we switched to thinking of our query results as polynomials and indeed pretty much of the rest of the paper has focused on thinking of our input as  a polynomial. In particular, starting with~\Cref{sec:expression-trees} with considered these polynomials to be represented as an expression tree. However, these do not capture many of the compressed polynomial representations that we can get from query processing algorithms on bags including the recent work on worst-case optimal join algorithms~\cite{ngo-survey,skew}, factorized databases~\cite{factorized-db} and FAQ~\cite{DBLP:conf/pods/KhamisNR16}. Intuitively the main reason is that an expression tree does not allow for `storing' any intermediate results, which is crucial for these algorithms (and other query processing results as well).
+In~\Cref{sec:semnx-as-repr}, we switched to thinking of our query results as polynomials and indeed pretty much of the rest of the paper has focused on thinking of our input as  a polynomial. In particular, starting with~\Cref{sec:expression-trees} we considered these polynomials to be represented as an expression tree. However, these do not capture many of the compressed polynomial representations that we can get from query processing algorithms on bags, including the recent work on worst-case optimal join algorithms~\cite{ngo-survey,skew}, factorized databases~\cite{factorized-db}, and FAQ~\cite{DBLP:conf/pods/KhamisNR16}. Intuitively, the main reason is that an expression tree does not allow for `storing' any intermediate results, which is crucial for these algorithms (and other query processing results as well).
 
-In this section, we represent query polynomials via {\em arithmetic circuits}~\cite{arith-complexity}, which are a standard way to represent polynomials over fields (and is standard in the field of algebraic complexity), though in our case we use them for polynomials over $\mathbb N$ in the obvious way. We present a formal treatment of {\em lineage circuit} in~\Cref{sec:circuits-formal}, but only a quick overview here. A lineage circuit is represented by DAG, where each source node corresponds to either one of the input variables or a constant and the sinks correspond to the output. Every other node has at most two incoming edges (and is labeled as either an addition or a multiplication node) but there is no limit on the outdegree of such nodes. We note that if we restricted the outdegree to be one, then we get back expression trees.
+In this section, we represent query polynomials via {\em arithmetic circuits}~\cite{arith-complexity}, which are a standard way to represent polynomials over fields (and is standard in the field of algebraic complexity), though in our case we use them for polynomials over $\mathbb N$ in the obvious way. We present a formal treatment of {\em lineage circuit}s in~\Cref{sec:circuits-formal}, with only a quick overview to start. A lineage circuit is represented by a DAG, where each source node corresponds to either one of the input variables or a constant, and the sinks correspond to the output. Every other node has at most two incoming edges (and is labeled as either an addition or a multiplication node), but there is no limit on the outdegree of such nodes. We note that if we restricted the outdegree to be one, then we get back expression trees.
 
 
-In~\Cref{sec:results-circuits} we argue why our results from earlier sections also hold for lineage circuits and then argue why lineage circuits do indeed capture the notion of runtime of some well-known query processing algorithms in~\Cref{sec:circuit-runtime} (and we formally define our cost model to capture the runtime of algorithms in~\Cref{sec:cost-model}).
+In~\Cref{sec:results-circuits} we argue why our results from earlier sections also hold for lineage circuits and then argue why lineage circuits do indeed capture the notion of runtime of some well-known query processing algorithms in~\Cref{sec:circuit-runtime} (We formally define the corresponding cost model in~\Cref{sec:cost-model}).
 
 \subsubsection{Extending our results to lineage circuits}
 \label{sec:results-circuits}
 
-We first note that since expression trees are a special case of lineage circuits, all of our hardness results in~\Cref{sec:hard} are still valid for lineage circuits.
+We first note that since expression trees are a special case of them, all of our hardness results in~\Cref{sec:hard} are still valid for lineage circuits.
 
-For the approximation algorithm in~\Cref{sec:algo} we note that \textsc{Approx}\textsc{imate}$\rpoly$ (\Cref{alg:mon-sam}) works for lineage circuits as long as $\onepass$ and $\sampmon$ have the same guarantees (\Cref{lem:one-pass} and \Cref{lem:sample} respectively) hold for lineage circuits as well. It turns out that both $\onepass$ and $\sampmon$ work for lineage circuits as well simply because the only property these use for expression trees is that each node has two children and this is still valid of lineage circuits (where for each non-source node the children correspond to the two nodes that have incoming edges to the given node). Put another way, our argument never used the fact that in an expression tree, each node has at most one parent.
+For the approximation algorithm in~\Cref{sec:algo} we note that \textsc{Approx}\textsc{imate}$\rpoly$ (\Cref{alg:mon-sam}) works for lineage circuits as long as the same guarantees on $\onepass$ and $\sampmon$ (\Cref{lem:one-pass} and \Cref{lem:sample} respectively) hold for lineage circuits as well. It turns out that both $\onepass$ and $\sampmon$ work for lineage circuits as well, simply because the only property these use for expression trees is that each node has two children. This is still valid of lineage circuits where for each non-source node the children correspond to the two nodes that have incoming edges to the given node. Put another way, our argument never used the fact that in an expression tree, each node has at most one parent.
 
-More specifically consider $\onepass$. The algorithm (as well as its analysis) basically uses the fact that one can compute the corresponding polynomial at all $1$s input with a simple recursive formula (\cref{eq:T-all-ones}) and that we can compute a probability distribution based on these weights (as in~\cref{eq:T-weights}). It can be verified that all the arguments go through if we replace $\etree_\lchild$ and $\etree_\rchild$ for expression tree $\etree$ with the two incoming nodes of the sink for the given lineage circuit. Another way to look at this is we could `unroll' the recursion in $\onepass$ and think of the algorithm as doing the evaluation at each node bottom up from leaves to the root in the expression tree. For lineage circuits, we start from the source nodes and do the computation in the topological order till we reach the sink(s).
+More specifically consider $\onepass$. The algorithm (as well as its analysis) basically uses the fact that one can compute the corresponding polynomial at all $1$s input with a simple recursive formula (\cref{eq:T-all-ones}), and that we can compute a probability distribution based on these weights (as in~\cref{eq:T-weights}). It can be verified that all the arguments go through if we replace $\etree_\lchild$ and $\etree_\rchild$ for expression tree $\etree$ with the two incoming nodes of the sink for the given lineage circuit. Another way to look at this is we could `unroll' the recursion in $\onepass$ and think of the algorithm as doing the evaluation at each node bottom up from leaves to the root in the expression tree. For lineage circuits, we start from the source nodes and do the computation in the topological order till we reach the sink(s).
 
-The argument for $\sampmon$ is similar. Since we argued that $\onepass$ works as intended for lineage circuits since~\Cref{alg:one-pass} only recurses on children of the current node in the expression tree and we can generalize it to lineage circuits by recursing to the two children of the current node in the lineage circuit. Alternatively, as we have already used in the proof of~\Cref{lem:sample}, we can think of the sampling algorithm sampling a sub-graph of the expression tree. For lineage circuits, we can think of $\sampmon$ as sampling the same sub-graph. Alternatively, one can implicitly expand the circuit lineage into a (larger but) equivalent expression tree. Since $\sampmon$ only explores one sub-graph during its run we can think of its run on a lineage circuit as being done on the implicit equivalent expression tree. Hence, all of the results on $\sampmon$  on expression trees carry over to lineage circuits.
+The argument for $\sampmon$ is similar. Since we argued that $\onepass$ works as intended for lineage circuits since~\Cref{alg:one-pass} only recurses on children of the current node in the expression tree and we can generalize it to lineage circuits by recursing to the two children of the current node in the lineage circuit. Alternatively, as we have already used in the proof of~\Cref{lem:sample}, we can think of the sampling algorithm sampling a sub-graph of the expression tree. For lineage circuits, we can think of $\sampmon$ as sampling the same sub-graph. Alternatively, one can implicitly expand the circuit lineage into a (larger but) equivalent expression tree. Since $\sampmon$ only explores one sub-graph during its run we can think of its run on a lineage circuit as being done on the implicit equivalent expression tree\footnote{
+  Recall that $\sampmon$ scales only in the depth of the expression and its polynomial degree ($k$). There exist polynomials that can be encoded in size $\Omega(\log k)$, but we follow convention in assuming that the circuit size is asymptotically larger than $k$ and thus treat the degree (i.e., join width) as a constant.
+}. Hence, all of the results on $\sampmon$  on expression trees carry over to lineage circuits.
 
 Thus, we have argued that~\Cref{lem:approx-alg} also holds if we use a lineage circuit instead of an expression tree as the input to our approximation algorithm.
 
 \subsubsection{The cost model}
 \label{sec:cost-model}
 Thus far, our analysis of the runtime of $\onepass$ has been in terms of the size of the compressed lineage polynomial. 
-We now show that this models the behavior of a deterministic database by proving that for any union of conjunctive query, we can construct a compressed lineage polynomial with the same complexity as it would take to evaluate the query on a deterministic \emph{bag-relational} database.
-We adopt a minimalistic model of query evaluation focusing on the size of intermediate materialized states.
+We now show that this model corresponds to the behavior of a deterministic database by proving that for any union of conjunctive query, we can construct a compressed lineage polynomial with the same complexity as it would take to evaluate the query on a deterministic \emph{bag-relational} database.
+We adopt a minimalistic compute-bound model of query evaluation drawn from worst-case optimal joins~\cite{skew,ngo-survey}.
 \newcommand{\qruntime}[1]{\textbf{cost}(#1)}
 \begin{align*}
 \qruntime{Q} & = |Q|\\
@@ -41,7 +43,7 @@ We adopt a minimalistic model of query evaluation focusing on the size of interm
 Under this model the query plan $Q(D)$ has runtime $O(\qruntime{Q(D)})$.
 Base relations assume that a full table scan is required; We model index scans by treating an index scan query $\sigma_\theta(R)$ as a single base relation.
 
-It can be verified that the worst-case join algorithms~\cite{skew,ngo-survey} as well as query evaluation via factorized databases~\cite{factorized-db} (and work on FAQs~\cite{DBLP:conf/pods/KhamisNR16}) can be modeled as select-union-project-join queries (though these queries can be data dependent).\footnote{This claim can be verified by e.g. simply looking at the {\em Generic-Join} algorithm in~\cite{skew} and {\em factorize} algorithm in~\cite{factorized-db}.} Further, it can be verified that the above cost model on the corresponding SUPJ join queries correctly captures their runtime.
+It can be verified that the worst-case join algorithms~\cite{skew,ngo-survey}, as well as query evaluation via factorized databases~\cite{factorized-db} (and work on FAQs~\cite{DBLP:conf/pods/KhamisNR16}) can be modeled as select-union-project-join queries (though these queries can be data dependent).\footnote{This claim can be verified by e.g. simply looking at the {\em Generic-Join} algorithm in~\cite{skew} and {\em factorize} algorithm in~\cite{factorized-db}.} Further, it can be verified that the above cost model on the corresponding SUPJ join queries correctly captures their runtime.
 \AH{I am used to folks using the order SPJU, is this ordering of operations a `standard' that we should follow?}
 \AR{Am not sure if we need to motivate the cost model more.} 
 %We now make a simple observation on the above cost model:
@@ -57,7 +59,7 @@ We now define a lineage circuit more formally and also show how to construct a l
 
 As mentioned earlier, we represent lineage polynomials with arithmetic circuits over $\mathbb N$ with $+$, $\times$.  
 A circuit for query $Q$ is a directed acyclic graph $\tuple{V_Q, E_Q, \phi_Q, \ell_Q}$ with vertices $V_Q$ and directed edges $E_Q \subset V_Q^2$.  
-A sink function $\phi_Q : Q \rightarrow V_Q$ maps the tuples of the relation to vertices in the graph.  
+A sink function $\phi_Q : \udom^n \rightarrow V_Q$ is a partial function that maps the tuples of the $n$-ary relation defined by $Q$ to vertices in the graph.  
 We require that $\phi_Q$'s range be limited to sink vertices (i.e., vertices with out-degree 0).
 %We call a sink vertex not in the range of $\phi_R$ a \emph{dead sink}.
 A function $\ell_Q : V_Q \rightarrow \{\;+,\times\;\}\cup \mathbb N \cup \vct X$ assigns a label to each node: Source nodes (i.e., vertices with in-degree 0) are labeled with constants or variables (i.e., $\mathbb N \cup \vct X$), while the remaining nodes are labeled with the symbol $+$ or $\times$.
@@ -85,27 +87,26 @@ We re-use the circuit for $Q_1$. %, but define a new distinguished node $v_0$ wi
 Formally, let $V_Q = V_{Q_1}$, let $\ell_Q(v_0) = 0$, and let $\ell_Q(v) = \ell_{Q_1}(v)$ for any $v \in V_{Q_1}$.  Let $E_Q = E_{Q_1}$, and define
 $$\phi_Q(t) =
 \phi_{Q_1}(t)  \text{ for } t \text{ s.t.}\; \theta(t).$$
-
-\AH{While not explicit, I assume a reviewer would know that the notation above discards tuples/vertices not satisfying the selection predicate.}
+Dead sinks are iteratively removed, and so 
+%\AH{While not explicit, I assume a reviewer would know that the notation above discards tuples/vertices not satisfying the selection predicate.}
 %v_0 & \textbf{otherwise}
 %\end{cases}$$
-This circuit has $|V_{Q_1}|$ vertices.
+this circuit has at most $|V_{Q_1}|$ vertices.
 
 \caseheading{Projection}
 Let $Q = \pi_{\vct A} {Q_1}$.
 We extend the circuit for ${Q_1}$ with a new set of sum vertices (i.e., vertices with label $+$) for each tuple in $Q$, and connect them to the corresponding sink nodes of the circuit for ${Q_1}$.
 Naively, let $V_Q = V_{Q_1} \cup \comprehension{v_t}{t \in \pi_{\vct A} {Q_1}}$, let $\phi_Q(t) = v_t$, and let $\ell_Q(v_t) = +$.  Finally let 
 $$E_Q = E_{Q_1} \cup \comprehension{(\phi_{Q_1}(t'), v_t)}{t = \pi_{\vct A} t', t' \in {Q_1}, t \in \pi_{\vct A} {Q_1}}$$
-This formulation will produce vertices with an in-degree greater than two, a problem that we correct by replacing every vertex with an in-degree over two by an equivalent fan-in tree.  The resulting structure has at most $|{Q_1}|-1$ additional vertices.
-\AH{Is the rightmost operator \emph{supposed} to be a $-$?  In the beginning we add $|\pi_{\vct A}{Q_1}|$ vertices.}
-The corrected circuit thus has at most $|V_{Q_1}|+ |{Q_1}|-|\pi_{\vct A} {Q_1}|$ vertices.
+This formulation will produce vertices with an in-degree greater than two, a problem that we correct by replacing every vertex with an in-degree over two by an equivalent fan-in tree.  The resulting structure has at most $|{Q_1}|-1$ new vertices.
+% \AH{Is the rightmost operator \emph{supposed} to be a $-$?  In the beginning we add $|\pi_{\vct A}{Q_1}|$ vertices.}
+The corrected circuit thus has at most $|V_{Q_1}|+|{Q_1}|$ vertices.
 
 \caseheading{Union}
 Let $Q = {Q_1} \cup {Q_2}$.
 We merge graphs and produce a sum vertex for all tuples in both sides of the union.
-Formally, let $V_Q = V_{Q_1} \cup V_{Q_2} \cup \comprehension{v_t}{t \in {Q_1} \cap {Q_2}}$, let 
-$$E_Q = E_{Q_1} \cup E_{Q_2} \cup \comprehension{(\phi_{Q_1}(t), v_t), (\phi_{Q_2}(t), v_t)}{t \in {Q_1} \cap {Q_2}}$$, 
-let $\ell_Q(v_t) = +$, and let 
+Formally, let $V_Q = V_{Q_1} \cup V_{Q_2} \cup \comprehension{v_t}{t \in {Q_1} \cap {Q_2}}$, let $\ell_Q(v_t) = +$, and let 
+$$E_Q = E_{Q_1} \cup E_{Q_2} \cup \comprehension{(\phi_{Q_1}(t), v_t), (\phi_{Q_2}(t), v_t)}{t \in {Q_1} \cap {Q_2}}$$
 $$\phi_Q(t) = \begin{cases}
 v_t & \textbf{if } t \in {Q_1} \cap {Q_1}\\
 \phi_{Q_1}(t) & \textbf{if } t \not \in {Q_2}\\
@@ -119,8 +120,11 @@ We merge graphs and produce a multiplication vertex for all tuples resulting fro
 Naively, let $V_Q = V_{Q_1} \cup \ldots \cup V_{Q_k} \cup \comprehension{v_t}{t \in {Q_1} \bowtie \ldots \bowtie {Q_k}}$, let 
 {\small
 \begin{multline*}
-E_Q = E_{Q_1} \cup \ldots \cup E_{Q_k} \cup \\
-\comprehension{(\phi_{Q_1}(\pi_{\sch({Q_1})}t), v_t), \ldots, (\phi_{Q_k}(\pi_{\sch({Q_k})}t), v_t)}{t \in {Q_1} \bowtie \ldots \bowtie {Q_k}}
+E_Q = E_{Q_1} \cup \ldots \cup E_{Q_k} \cup 
+\left\{\;
+(\phi_{Q_1}(\pi_{\sch({Q_1})}t), v_t), \right.\\
+\ldots, (\phi_{Q_k}(\pi_{\sch({Q_k})}t), v_t)
+\;\left|\;t \in {Q_1} \bowtie \ldots \bowtie {Q_k}\;\right\}
 \end{multline*}
 }
 Let $\ell_Q(v_t) = \times$, and let $\phi_Q(t) = v_t$
@@ -138,19 +142,19 @@ The runtime of any query plan $Q$ has the same or better complexity as the linea
 \end{lemma}
 \begin{proof}
 Proof by induction.  The base case is a base relation: $Q = R$ and is trivially true since $|V_R| = |R|$.
-For the inductive step, we assume that we have circuits for subplans $Q_1, \ldots, Q_n$ such that $|V_{Q_i}| \leq a_i\qruntime{Q_i} + b_i$.
+For the inductive step, we assume that we have circuits for subplans $Q_1, \ldots, Q_n$ such that $|V_{Q_i}| \leq (k_i-1)\qruntime{Q_i}$ where $k_i$ is the degree of $Q_i$.
 
 \caseheading{Selection}
 Assume that $Q = \sigma_\theta(Q_1)$.
 In the circuit for $Q$, $|V_Q| = |V_{Q_1}|$ vertices, so from the inductive assumption and $\qruntime{Q} = \qruntime{Q_1}$ by definition, we have $|V_Q| \leq (k-1) \qruntime{Q} $.
-\AH{Technically, $\kElem$ is the degree of $\poly_1$, but I guess this is a moot point since one can argue that $\kElem$ is also the degree of $\poly$.}
+% \AH{Technically, $\kElem$ is the degree of $\poly_1$, but I guess this is a moot point since one can argue that $\kElem$ is also the degree of $\poly$.}
+% OK: Correct
 \caseheading{Projection}
 Assume that $Q = \pi_{\vct A}(Q_1)$.
-The circuit for $Q$ has at most $|V_{Q_1}|+|\pi_{\vct A} {Q_1}| + |{Q_1}|-\abs{\pi_AQ}$ vertices.
-\AH{The combination of terms above doesn't follow the details for projection above.}
+The circuit for $Q$ has at most $|V_{Q_1}|+|{Q_1}|$ vertices.
+% \AH{The combination of terms above doesn't follow the details for projection above.}
 \begin{align*}
-|V_{Q}| & \leq |V_{Q_1}|+ \abs{Q_1} -|\pi_{\vct A} {Q_1}| \\
-& \leq |V_{Q_1}| + |Q_1|\\
+|V_{Q}| & \leq |V_{Q_1}| + |Q_1|\\
 %\intertext{By \Cref{prop:queries-need-to-output-tuples} $\qruntime{Q_1} \geq |Q_1|$}
 %& \leq |V_{Q_1}| + 2 \qruntime{Q_1}\\
 \intertext{(From the inductive assumption)}
@@ -178,7 +182,7 @@ Assume that $Q = Q_1 \bowtie \ldots \bowtie Q_k$.
 The circuit for $Q$ has $|V_{Q_1}|+\ldots+|V_{Q_k}|+(k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|$ vertices.
 \begin{align*}
 |V_{Q}| & = |V_{Q_1}|+\ldots+|V_{Q_k}|+(k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|\\
-\intertext{From the inductive assumption}
+\intertext{From the inductive assumption and noting $\forall i: k_i \leq k-1$}
 & \leq (k-1)\qruntime{Q_1}+\ldots+(k-1)\qruntime{Q_k}+\\
 &\;\;\; (k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|\\
 & \leq (k-1)(\qruntime{Q_1}+\ldots+\qruntime{Q_k}+\\

intro.tex

@@ -28,7 +28,7 @@ The initial picture is not good.
 In this paper, we prove that computing expected counts is \emph{not} linear in the size of a compressed --- specifically a factorized~\cite{10.1145/3003665.3003667} --- lineage polynomial by reduction to counting 3-matchings.
 Thus, even bag PDBs do not enjoy the same computational complexity as deterministic databases.
 This motivates our second goal, a linear time approximation algorithm for computing expected counts in a bag database, with complexity linear in the size of a factorized lineage formula.
-As we show (\Cref{prop:queries-need-to-output-tuples}), the size of the factorized 
+As we will show, the size of the factorized 
 lineage formula for a query --- and by extension, our approximation algorithm --- is proportional to the complexity of evaluating the same query on a comparable deterministic database instance~\cite{DBLP:conf/pods/KhamisNR16,10.1145/3003665.3003667}.
 In other words, our approximation algorithm can estimate expected multiplicities for tuples in the result of an SPJU query with a complexity comparable to deterministic query-processing.
 

main.tex

@@ -87,7 +87,8 @@ sensitive=true
 %%%%%%%%%%%%%%%%%%%%
 % \textbullet Modelling Uncertainty as Attribute-level Taints and its Relationship to Provenance}
 
-\title{Standard Operating Procedure in PDBs Considered Harmful for Bags}
+\title{Standard Operating Procedure in PDBs Considered Harmful}
+\subtitle{(for bags)}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%