Merge branch 'master' of gitlab.odin.cse.buffalo.edu:ahuber/SketchingWorlds
This commit is contained in:
Boris Glavic
commit
dcbbafa4e9
|
|
@ -9,40 +9,22 @@ In~\cref{sec:hard}, we showed that computing the expected multiplicity of a comp
|
|||
First, let us introduce some useful definitions and notation related to polynomials and their representations. For illustrative purposes in the definitions below, we will use the following {\em bivariate} polynomial:
|
||||
\begin{equation}
|
||||
\label{eq:poly-eg}
|
||||
\poly(x,y) = 2x^2 + 3xy - 2y^2.
|
||||
\poly(X, Y) = 2X^2 + 3XY - 2Y^2.
|
||||
\end{equation}
|
||||
|
||||
\AR{The definition from this and my next comments are "new"-- they might be better off in the prelims section and moved to later in this section. Am keeping all of them in one place for easy lookup for now.}
|
||||
|
||||
\begin{Definition}[Variables in a monomial]\label{def:vars}
|
||||
Given a monomial $v$, we use $\var(v)$ to denote the set of variables in $v$.
|
||||
\end{Definition}
|
||||
For example the monomial $3xy$ in the polynomial in~\cref{eq:poly-eg} has $\var(3xy)=\inset{x,y}$.
|
||||
For example the monomial $XY$ has $\var(XY)=\inset{X,Y}$.
|
||||
|
||||
%\AH{@atri, just a heads up. I had defined \emph{monomial} as the variables exclusively, i.e., without the coefficient. However, it appears that @lordpretzel removed that definition, but I just wanted to mention this so that \emph{hopefully} we are consistent in our language as to what we mean by the term monomial.}
|
||||
%\AR{I think its OK: it not a big difference and I don't think the readers will get confused-- if we are worried we can always add a disclaimer saying we might include the coefficient or nont dependning on the context}.
|
||||
|
||||
\begin{definition}[Modding with a set]\label{def:mod-set}
|
||||
Let $S$ be a {\em set} of polynomials over $\vct{X}$. Then $\poly(\vct{X})\mod{S}$ is the polynomial obtained by taking the mod of $\poly(\vct{X})$ over {\em all} polynomials in $S$ (the order does not matter).
|
||||
\end{definition}
|
||||
For example when $S_0=\inset{x^2-x,y^2-y}$, taking the polynomial in~\cref{eq:poly-eg} mod $S_0$, we get $2x+3xy-2y$.
|
||||
|
||||
\begin{Definition}\label{def:mod-set-polys}
|
||||
Given the set of BIDB variables $\inset{X_{b,i}}$, define
|
||||
\[\mathcal{B}=\inset{X_{b,i}\cdot X_{b,j}|\text{ for every block } b \text{and } i\ne j},\]
|
||||
\[\mathcal{T}=\inset{X_{b,i}^2-X_{b,i}|\text{ for every block } b \text{and } i}.\]
|
||||
\end{Definition}
|
||||
%\begin{Definition}[Expression Tree]\label{def:express-tree}
|
||||
%An expression tree $\etree$ is a binary %an ADT logically viewed as an n-ary
|
||||
%tree, whose internal nodes are from the set $\{+, \times\}$, with leaf nodes being either from the set $\mathbb{R}$ $(\tnum)$ or from the set of monomials $(\var)$. The members of $\etree$ are \type, \val, \vari{partial}, \vari{children}, and \vari{weight}, where \type is the type of value stored in the node $\etree$ (i.e. one of $\{+, \times, \var, \tnum\}$, \val is the value stored, and \vari{children} is the list of $\etree$'s children where $\etree_\lchild$ is the left child and $\etree_\rchild$ the right child. Remaining fields hold values whose semantics we will fix later. When $\etree$ is used as input of ~\Cref{alg:mon-sam} and ~\Cref{alg:one-pass}, the values of \vari{partial} and \vari{weight} will not be set. %SEMANTICS FOR \etree: \vari{partial} is the sum of $\etree$'s coefficients , n, and \vari{weight} is the probability of $\etree$ being sampled.
|
||||
%\end{Definition}
|
||||
|
||||
\AR{Something to check/square out: we have been using both $X_{b,j}$ and $X_1,\dots,X_n$ for vars in BIDB-- I think this is OK as long as we explicitly talk about these two notations and how we might switch between them. Or we decide not to...}
|
||||
|
||||
\AR{Some of these definitions have been pulled to the prelims section. Another pass is needed to sync up these occurrences. Leaving them in for now.}
|
||||
|
||||
\begin{Definition}[Expression Tree]\label{def:express-tree}
|
||||
An expression tree $\etree$ is a binary %an ADT logically viewed as an n-ary
|
||||
tree, whose internal nodes are from the set $\{+, \times\}$, with leaf nodes being either from the set $\mathbb{R}$ $(\tnum)$ or from the set of monomials $(\var)$. The members of $\etree$ are \type, \val, \vari{partial}, \vari{children}, and \vari{weight}, where \type is the type of value stored in the node $\etree$ (i.e. one of $\{+, \times, \var, \tnum\}$, \val is the value stored, and \vari{children} is the list of $\etree$'s children where $\etree_\lchild$ is the left child and $\etree_\rchild$ the right child. Remaining fields hold values whose semantics we will fix later. When $\etree$ is used as input of ~\Cref{alg:mon-sam} and ~\Cref{alg:one-pass}, the values of \vari{partial} and \vari{weight} will not be set. %SEMANTICS FOR \etree: \vari{partial} is the sum of $\etree$'s coefficients , n, and \vari{weight} is the probability of $\etree$ being sampled.
|
||||
\end{Definition}
|
||||
|
||||
Note that $\etree$ need not encode an expression in the standard monomial basis. For instance, $\etree$ could represent a compressed form of the polynomial in~\cref{eq:poly-eg}, such as $(x + 2y)(2x - y)$.
|
||||
%Note that $\etree$ need not encode an expression in the standard monomial basis. For instance, $\etree$ could represent a compressed form of the polynomial in~\cref{eq:poly-eg}, such as $(x + 2y)(2x - y)$.
|
||||
|
||||
\begin{Definition}[$\polyf(\cdot)$]\label{def:poly-func}
|
||||
Denote $\polyf(\etree)$ to be the function that takes as input expression tree $\etree$ and outputs its corresponding polynomial. $poly(\cdot)$ is recursively defined on $\etree$ as follows, where $\etree_\lchild$ and $\etree_\rchild$ denote the left and right child of $\etree$ respectively.
|
||||
|
|
@ -66,10 +48,10 @@ Denote $\polyf(\etree)$ to be the function that takes as input expression tree $
|
|||
Note that addition and multiplication above follow the standard interpretation over polynomials.
|
||||
%Specifically, when adding two monomials whose variables and respective exponents agree, the coefficients corresponding to the monomials are added and their sum is multiplied to the monomial. Multiplication here is denoted by concatenation of the monomial and coefficient. When two monomials are multiplied, the product of each corresponding coefficient is computed, and the variables in each monomial are multiplied, i.e., the exponents of like variables are added. Again we notate this by the direct product of coefficient product and all disitinct variables in the two monomials, with newly computed exponents.
|
||||
|
||||
\begin{Definition}[Expression Tree Set]\label{def:express-tree-set}$\etreeset{\smb}$ is the set of all possible expression trees $\etree$, such that $poly(\etree) = \poly(\vct{X})$.
|
||||
\end{Definition}
|
||||
|
||||
For the polynomial in~\cref{eq:poly-eg}, $\etreeset{\smb}$ would include the following (represented as their corresponding expression trees): $2x^2 + 3xy - 2y^2, (x + 2y)(2x - y), x(2x - y) + 2y(2x - y), 2x(x + 2y) - y(x + 2y)$. Note that \cref{def:express-tree-set} implies that for any expression tree $\etree$, we have $\etree \in \etreeset{poly(\etree)}$.
|
||||
%\begin{Definition}[Expression Tree Set]\label{def:express-tree-set}$\etreeset{\smb}$ is the set of all possible expression trees $\etree$, such that $poly(\etree) = \poly(\vct{X})$.
|
||||
%\end{Definition}
|
||||
%
|
||||
%For the polynomial in~\cref{eq:poly-eg}, $\etreeset{\smb}$ would include the following (represented as their corresponding expression trees): $2x^2 + 3xy - 2y^2, (x + 2y)(2x - y), x(2x - y) + 2y(2x - y), 2x(x + 2y) - y(x + 2y)$. Note that \cref{def:express-tree-set} implies that for any expression tree $\etree$, we have $\etree \in \etreeset{poly(\etree)}$.
|
||||
|
||||
|
||||
\begin{Definition}[Expanded T]\label{def:expand-tree}
|
||||
|
|
@ -97,7 +79,7 @@ $\expandtree{\etree}$ is the (pure) sum of products expansion of $\etree$, which
|
|||
|
||||
|
||||
\begin{Example}\label{example:expr-tree-T}
|
||||
Consider the factorized representation $(x + 2y)(2x - y)$ of the polynomial in~\cref{eq:poly-eg}. Its expression tree $\etree$ is illustrated in Figure ~\ref{fig:expr-tree-T}. The pure expansion of the product is $2x^2 - xy + 4xy - 2y^2$ and the $\expandtree{\etree}$ is $[(2, x^2), (-1, xy), (4, xy), (-2, y^2)]$.
|
||||
Consider the factorized representation $(X+ 2Y)(2X - Y)$ of the polynomial in~\cref{eq:poly-eg}. Its expression tree $\etree$ is illustrated in Figure ~\ref{fig:expr-tree-T}. The pure expansion of the product is $2X^2 - XY + 4XY - 2Y^2$ and the $\expandtree{\etree}$ is $[(2, X^2), (-1, XY), (4, XY), (-2, Y^2)]$.
|
||||
\end{Example}
|
||||
|
||||
|
||||
|
|
@ -144,7 +126,7 @@ For any expression tree $\etree$, the corresponding
|
|||
{\em positive tree}, denoted $\abs{\etree}$ obtained from $\etree$ as follows. For each leaf node $\ell$ of $\etree$ where $\ell.\type$ is $\tnum$, update $\ell.\vari{value}$ to $|\ell.\vari{value}|$. %value $\coef$ of each coefficient leaf node in $\etree$ is set to %$\coef_i$ in $\etree$ is exchanged with its absolute value$|\coef|$.
|
||||
\end{Definition}
|
||||
|
||||
Using the same factorization from ~\cref{example:expr-tree-T}, $poly(\abs{\etree}) = (x + 2y)(2x + y) = 2x^2 +xy +4xy + 2y^2 = 2x^2 + 5xy + 2y^2$. Note that this \textit{is not} the same as the polynomial from~\cref{eq:poly-eg}.
|
||||
Using the same factorization from ~\cref{example:expr-tree-T}, $poly(\abs{\etree}) = (X + 2Y)(2X + Y) = 2X^2 +XY +4XY + 2Y^2 = 2X^2 + 5XY + 2Y^2$. Note that this \textit{is not} the same as the polynomial from~\cref{eq:poly-eg}.
|
||||
|
||||
\begin{Definition}[Evaluation]\label{def:exp-poly-eval}
|
||||
Given an expression tree $\etree$ and $\vct{v} \in \mathbb{R}^\numvar$, $\etree(\vct{v}) = poly(\etree)(\vct{v})$.
|
||||
|
|
|
|||
|
|
@ -1,35 +1,38 @@
|
|||
%!TEX root=./main.tex
|
||||
\section{Generalizations}
|
||||
\label{sec:gen}
|
||||
In this section, we consider a couple of generalizations/corollaries of our results so far. In particular, in~\Cref{sec:circuits} we first consider the case when the compressed polynomial is represented by a Directed Acyclic Graph (DAG) instead of the earlier (expression) tree (\Cref{def:express-tree}) and we observe that all of our results carry over to the DAG representation. Then we formalize our claim in~\Cref{sec:intro} that a linear runtime algorithm for our problem would imply that we can process PDBs in the same time as deterministic query processing. Finally, in~\Cref{sec:momemts}, we make some simple observations on how our results can be used to estimate moments beyond the expectation of a lineage polynomial.
|
||||
|
||||
\subsection{Lineage circuits}
|
||||
\label{sec:circuits}
|
||||
|
||||
In~\Cref{sec:semnx-as-repr}, we switched to thinking of our query results as polynomials and indeed pretty much of the rest of the paper has focused on thinking of our input as a polynomial. In particular, starting with~\Cref{sec:expression-trees} with considered these polynomials to be represented as an expression tree. However, these do not capture many of the compressed polynomial representations that we can get from query processing algorithms on bags including the recent work on worst-case optimal join algorithms~\cite{ngo-survey,skew}, factorized databases~\cite{factorized-db} and FAQ~\cite{DBLP:conf/pods/KhamisNR16}. Intuitively the main reason is that an expression tree does not allow for `storing' any intermediate results, which is crucial for these algorithms (and other query processing results as well).
|
||||
In~\Cref{sec:semnx-as-repr}, we switched to thinking of our query results as polynomials and indeed pretty much of the rest of the paper has focused on thinking of our input as a polynomial. In particular, starting with~\Cref{sec:expression-trees} we considered these polynomials to be represented as an expression tree. However, these do not capture many of the compressed polynomial representations that we can get from query processing algorithms on bags, including the recent work on worst-case optimal join algorithms~\cite{ngo-survey,skew}, factorized databases~\cite{factorized-db}, and FAQ~\cite{DBLP:conf/pods/KhamisNR16}. Intuitively, the main reason is that an expression tree does not allow for `storing' any intermediate results, which is crucial for these algorithms (and other query processing results as well).
|
||||
|
||||
In this section, we represent query polynomials via {\em arithmetic circuits}~\cite{arith-complexity}, which are a standard way to represent polynomials over fields (and is standard in the field of algebraic complexity), though in our case we use them for polynomials over $\mathbb N$ in the obvious way. We present a formal treatment of {\em lineage circuit} in~\Cref{sec:circuits-formal}, but only a quick overview here. A lineage circuit is represented by DAG, where each source node corresponds to either one of the input variables or a constant and the sinks correspond to the output. Every other node has at most two incoming edges (and is labeled as either an addition or a multiplication node) but there is no limit on the outdegree of such nodes. We note that if we restricted the outdegree to be one, then we get back expression trees.
|
||||
In this section, we represent query polynomials via {\em arithmetic circuits}~\cite{arith-complexity}, which are a standard way to represent polynomials over fields (and is standard in the field of algebraic complexity), though in our case we use them for polynomials over $\mathbb N$ in the obvious way. We present a formal treatment of {\em lineage circuit}s in~\Cref{sec:circuits-formal}, with only a quick overview to start. A lineage circuit is represented by a DAG, where each source node corresponds to either one of the input variables or a constant, and the sinks correspond to the output. Every other node has at most two incoming edges (and is labeled as either an addition or a multiplication node), but there is no limit on the outdegree of such nodes. We note that if we restricted the outdegree to be one, then we get back expression trees.
|
||||
|
||||
|
||||
In~\Cref{sec:results-circuits} we argue why our results from earlier sections also hold for lineage circuits and then argue why lineage circuits do indeed capture the notion of runtime of some well-known query processing algorithms in~\Cref{sec:circuit-runtime} (and we formally define our cost model to capture the runtime of algorithms in~\Cref{sec:cost-model}).
|
||||
In~\Cref{sec:results-circuits} we argue why our results from earlier sections also hold for lineage circuits and then argue why lineage circuits do indeed capture the notion of runtime of some well-known query processing algorithms in~\Cref{sec:circuit-runtime} (We formally define the corresponding cost model in~\Cref{sec:cost-model}).
|
||||
|
||||
\subsubsection{Extending our results to lineage circuits}
|
||||
\label{sec:results-circuits}
|
||||
|
||||
We first note that since expression trees are a special case of lineage circuits, all of our hardness results in~\Cref{sec:hard} are still valid for lineage circuits.
|
||||
We first note that since expression trees are a special case of them, all of our hardness results in~\Cref{sec:hard} are still valid for lineage circuits.
|
||||
|
||||
For the approximation algorithm in~\Cref{sec:algo} we note that \textsc{Approx}\textsc{imate}$\rpoly$ (\Cref{alg:mon-sam}) works for lineage circuits as long as $\onepass$ and $\sampmon$ have the same guarantees (\Cref{lem:one-pass} and \Cref{lem:sample} respectively) hold for lineage circuits as well. It turns out that both $\onepass$ and $\sampmon$ work for lineage circuits as well simply because the only property these use for expression trees is that each node has two children and this is still valid of lineage circuits (where for each non-source node the children correspond to the two nodes that have incoming edges to the given node). Put another way, our argument never used the fact that in an expression tree, each node has at most one parent.
|
||||
For the approximation algorithm in~\Cref{sec:algo} we note that \textsc{Approx}\textsc{imate}$\rpoly$ (\Cref{alg:mon-sam}) works for lineage circuits as long as the same guarantees on $\onepass$ and $\sampmon$ (\Cref{lem:one-pass} and \Cref{lem:sample} respectively) hold for lineage circuits as well. It turns out that both $\onepass$ and $\sampmon$ work for lineage circuits as well, simply because the only property these use for expression trees is that each node has two children. This is still valid of lineage circuits where for each non-source node the children correspond to the two nodes that have incoming edges to the given node. Put another way, our argument never used the fact that in an expression tree, each node has at most one parent.
|
||||
|
||||
More specifically consider $\onepass$. The algorithm (as well as its analysis) basically uses the fact that one can compute the corresponding polynomial at all $1$s input with a simple recursive formula (\cref{eq:T-all-ones}) and that we can compute a probability distribution based on these weights (as in~\cref{eq:T-weights}). It can be verified that all the arguments go through if we replace $\etree_\lchild$ and $\etree_\rchild$ for expression tree $\etree$ with the two incoming nodes of the sink for the given lineage circuit. Another way to look at this is we could `unroll' the recursion in $\onepass$ and think of the algorithm as doing the evaluation at each node bottom up from leaves to the root in the expression tree. For lineage circuits, we start from the source nodes and do the computation in the topological order till we reach the sink(s).
|
||||
More specifically consider $\onepass$. The algorithm (as well as its analysis) basically uses the fact that one can compute the corresponding polynomial at all $1$s input with a simple recursive formula (\cref{eq:T-all-ones}), and that we can compute a probability distribution based on these weights (as in~\cref{eq:T-weights}). It can be verified that all the arguments go through if we replace $\etree_\lchild$ and $\etree_\rchild$ for expression tree $\etree$ with the two incoming nodes of the sink for the given lineage circuit. Another way to look at this is we could `unroll' the recursion in $\onepass$ and think of the algorithm as doing the evaluation at each node bottom up from leaves to the root in the expression tree. For lineage circuits, we start from the source nodes and do the computation in the topological order till we reach the sink(s).
|
||||
|
||||
The argument for $\sampmon$ is similar. Since we argued that $\onepass$ works as intended for lineage circuits since~\Cref{alg:one-pass} only recurses on children of the current node in the expression tree and we can generalize it to lineage circuits by recursing to the two children of the current node in the lineage circuit. Alternatively, as we have already used in the proof of~\Cref{lem:sample}, we can think of the sampling algorithm sampling a sub-graph of the expression tree. For lineage circuits, we can think of $\sampmon$ as sampling the same sub-graph. Alternatively, one can implicitly expand the circuit lineage into a (larger but) equivalent expression tree. Since $\sampmon$ only explores one sub-graph during its run we can think of its run on a lineage circuit as being done on the implicit equivalent expression tree. Hence, all of the results on $\sampmon$ on expression trees carry over to lineage circuits.
|
||||
The argument for $\sampmon$ is similar. Since we argued that $\onepass$ works as intended for lineage circuits since~\Cref{alg:one-pass} only recurses on children of the current node in the expression tree and we can generalize it to lineage circuits by recursing to the two children of the current node in the lineage circuit. Alternatively, as we have already used in the proof of~\Cref{lem:sample}, we can think of the sampling algorithm sampling a sub-graph of the expression tree. For lineage circuits, we can think of $\sampmon$ as sampling the same sub-graph. Alternatively, one can implicitly expand the circuit lineage into a (larger but) equivalent expression tree. Since $\sampmon$ only explores one sub-graph during its run we can think of its run on a lineage circuit as being done on the implicit equivalent expression tree\footnote{
|
||||
Recall that $\sampmon$ scales only in the depth of the expression and its polynomial degree ($k$). There exist polynomials that can be encoded in size $\Omega(\log k)$, but we follow convention in assuming that the circuit size is asymptotically larger than $k$ and thus treat the degree (i.e., join width) as a constant.
|
||||
}. Hence, all of the results on $\sampmon$ on expression trees carry over to lineage circuits.
|
||||
|
||||
Thus, we have argued that~\Cref{lem:approx-alg} also holds if we use a lineage circuit instead of an expression tree as the input to our approximation algorithm.
|
||||
|
||||
\subsubsection{The cost model}
|
||||
\label{sec:cost-model}
|
||||
Thus far, our analysis of the runtime of $\onepass$ has been in terms of the size of the compressed lineage polynomial.
|
||||
We now show that this models the behavior of a deterministic database by proving that for any union of conjunctive query, we can construct a compressed lineage polynomial with the same complexity as it would take to evaluate the query on a deterministic \emph{bag-relational} database.
|
||||
We adopt a minimalistic model of query evaluation focusing on the size of intermediate materialized states.
|
||||
We now show that this model corresponds to the behavior of a deterministic database by proving that for any union of conjunctive query, we can construct a compressed lineage polynomial with the same complexity as it would take to evaluate the query on a deterministic \emph{bag-relational} database.
|
||||
We adopt a minimalistic compute-bound model of query evaluation drawn from worst-case optimal joins~\cite{skew,ngo-survey}.
|
||||
\newcommand{\qruntime}[1]{\textbf{cost}(#1)}
|
||||
\begin{align*}
|
||||
\qruntime{Q} & = |Q|\\
|
||||
|
|
@ -41,7 +44,7 @@ We adopt a minimalistic model of query evaluation focusing on the size of interm
|
|||
Under this model the query plan $Q(D)$ has runtime $O(\qruntime{Q(D)})$.
|
||||
Base relations assume that a full table scan is required; We model index scans by treating an index scan query $\sigma_\theta(R)$ as a single base relation.
|
||||
|
||||
It can be verified that the worst-case join algorithms~\cite{skew,ngo-survey} as well as query evaluation via factorized databases~\cite{factorized-db} (and work on FAQs~\cite{DBLP:conf/pods/KhamisNR16}) can be modeled as select-union-project-join queries (though these queries can be data dependent).\footnote{This claim can be verified by e.g. simply looking at the {\em Generic-Join} algorithm in~\cite{skew} and {\em factorize} algorithm in~\cite{factorized-db}.} Further, it can be verified that the above cost model on the corresponding SUPJ join queries correctly captures their runtime.
|
||||
It can be verified that the worst-case join algorithms~\cite{skew,ngo-survey}, as well as query evaluation via factorized databases~\cite{factorized-db} (and work on FAQs~\cite{DBLP:conf/pods/KhamisNR16}) can be modeled as select-union-project-join queries (though these queries can be data dependent).\footnote{This claim can be verified by e.g. simply looking at the {\em Generic-Join} algorithm in~\cite{skew} and {\em factorize} algorithm in~\cite{factorized-db}.} Further, it can be verified that the above cost model on the corresponding SUPJ join queries correctly captures their runtime.
|
||||
\AH{I am used to folks using the order SPJU, is this ordering of operations a `standard' that we should follow?}
|
||||
\AR{Am not sure if we need to motivate the cost model more.}
|
||||
%We now make a simple observation on the above cost model:
|
||||
|
|
@ -57,7 +60,7 @@ We now define a lineage circuit more formally and also show how to construct a l
|
|||
|
||||
As mentioned earlier, we represent lineage polynomials with arithmetic circuits over $\mathbb N$ with $+$, $\times$.
|
||||
A circuit for query $Q$ is a directed acyclic graph $\tuple{V_Q, E_Q, \phi_Q, \ell_Q}$ with vertices $V_Q$ and directed edges $E_Q \subset V_Q^2$.
|
||||
A sink function $\phi_Q : Q \rightarrow V_Q$ maps the tuples of the relation to vertices in the graph.
|
||||
A sink function $\phi_Q : \udom^n \rightarrow V_Q$ is a partial function that maps the tuples of the $n$-ary relation defined by $Q$ to vertices in the graph.
|
||||
We require that $\phi_Q$'s range be limited to sink vertices (i.e., vertices with out-degree 0).
|
||||
%We call a sink vertex not in the range of $\phi_R$ a \emph{dead sink}.
|
||||
A function $\ell_Q : V_Q \rightarrow \{\;+,\times\;\}\cup \mathbb N \cup \vct X$ assigns a label to each node: Source nodes (i.e., vertices with in-degree 0) are labeled with constants or variables (i.e., $\mathbb N \cup \vct X$), while the remaining nodes are labeled with the symbol $+$ or $\times$.
|
||||
|
|
@ -85,27 +88,26 @@ We re-use the circuit for $Q_1$. %, but define a new distinguished node $v_0$ wi
|
|||
Formally, let $V_Q = V_{Q_1}$, let $\ell_Q(v_0) = 0$, and let $\ell_Q(v) = \ell_{Q_1}(v)$ for any $v \in V_{Q_1}$. Let $E_Q = E_{Q_1}$, and define
|
||||
$$\phi_Q(t) =
|
||||
\phi_{Q_1}(t) \text{ for } t \text{ s.t.}\; \theta(t).$$
|
||||
|
||||
\AH{While not explicit, I assume a reviewer would know that the notation above discards tuples/vertices not satisfying the selection predicate.}
|
||||
Dead sinks are iteratively removed, and so
|
||||
%\AH{While not explicit, I assume a reviewer would know that the notation above discards tuples/vertices not satisfying the selection predicate.}
|
||||
%v_0 & \textbf{otherwise}
|
||||
%\end{cases}$$
|
||||
This circuit has $|V_{Q_1}|$ vertices.
|
||||
this circuit has at most $|V_{Q_1}|$ vertices.
|
||||
|
||||
\caseheading{Projection}
|
||||
Let $Q = \pi_{\vct A} {Q_1}$.
|
||||
We extend the circuit for ${Q_1}$ with a new set of sum vertices (i.e., vertices with label $+$) for each tuple in $Q$, and connect them to the corresponding sink nodes of the circuit for ${Q_1}$.
|
||||
Naively, let $V_Q = V_{Q_1} \cup \comprehension{v_t}{t \in \pi_{\vct A} {Q_1}}$, let $\phi_Q(t) = v_t$, and let $\ell_Q(v_t) = +$. Finally let
|
||||
$$E_Q = E_{Q_1} \cup \comprehension{(\phi_{Q_1}(t'), v_t)}{t = \pi_{\vct A} t', t' \in {Q_1}, t \in \pi_{\vct A} {Q_1}}$$
|
||||
This formulation will produce vertices with an in-degree greater than two, a problem that we correct by replacing every vertex with an in-degree over two by an equivalent fan-in tree. The resulting structure has at most $|{Q_1}|-1$ additional vertices.
|
||||
\AH{Is the rightmost operator \emph{supposed} to be a $-$? In the beginning we add $|\pi_{\vct A}{Q_1}|$ vertices.}
|
||||
The corrected circuit thus has at most $|V_{Q_1}|+ |{Q_1}|-|\pi_{\vct A} {Q_1}|$ vertices.
|
||||
This formulation will produce vertices with an in-degree greater than two, a problem that we correct by replacing every vertex with an in-degree over two by an equivalent fan-in tree. The resulting structure has at most $|{Q_1}|-1$ new vertices.
|
||||
% \AH{Is the rightmost operator \emph{supposed} to be a $-$? In the beginning we add $|\pi_{\vct A}{Q_1}|$ vertices.}
|
||||
The corrected circuit thus has at most $|V_{Q_1}|+|{Q_1}|$ vertices.
|
||||
|
||||
\caseheading{Union}
|
||||
Let $Q = {Q_1} \cup {Q_2}$.
|
||||
We merge graphs and produce a sum vertex for all tuples in both sides of the union.
|
||||
Formally, let $V_Q = V_{Q_1} \cup V_{Q_2} \cup \comprehension{v_t}{t \in {Q_1} \cap {Q_2}}$, let
|
||||
$$E_Q = E_{Q_1} \cup E_{Q_2} \cup \comprehension{(\phi_{Q_1}(t), v_t), (\phi_{Q_2}(t), v_t)}{t \in {Q_1} \cap {Q_2}}$$,
|
||||
let $\ell_Q(v_t) = +$, and let
|
||||
Formally, let $V_Q = V_{Q_1} \cup V_{Q_2} \cup \comprehension{v_t}{t \in {Q_1} \cap {Q_2}}$, let $\ell_Q(v_t) = +$, and let
|
||||
$$E_Q = E_{Q_1} \cup E_{Q_2} \cup \comprehension{(\phi_{Q_1}(t), v_t), (\phi_{Q_2}(t), v_t)}{t \in {Q_1} \cap {Q_2}}$$
|
||||
$$\phi_Q(t) = \begin{cases}
|
||||
v_t & \textbf{if } t \in {Q_1} \cap {Q_1}\\
|
||||
\phi_{Q_1}(t) & \textbf{if } t \not \in {Q_2}\\
|
||||
|
|
@ -119,8 +121,11 @@ We merge graphs and produce a multiplication vertex for all tuples resulting fro
|
|||
Naively, let $V_Q = V_{Q_1} \cup \ldots \cup V_{Q_k} \cup \comprehension{v_t}{t \in {Q_1} \bowtie \ldots \bowtie {Q_k}}$, let
|
||||
{\small
|
||||
\begin{multline*}
|
||||
E_Q = E_{Q_1} \cup \ldots \cup E_{Q_k} \cup \\
|
||||
\comprehension{(\phi_{Q_1}(\pi_{\sch({Q_1})}t), v_t), \ldots, (\phi_{Q_k}(\pi_{\sch({Q_k})}t), v_t)}{t \in {Q_1} \bowtie \ldots \bowtie {Q_k}}
|
||||
E_Q = E_{Q_1} \cup \ldots \cup E_{Q_k} \cup
|
||||
\left\{\;
|
||||
(\phi_{Q_1}(\pi_{\sch({Q_1})}t), v_t), \right.\\
|
||||
\ldots, (\phi_{Q_k}(\pi_{\sch({Q_k})}t), v_t)
|
||||
\;\left|\;t \in {Q_1} \bowtie \ldots \bowtie {Q_k}\;\right\}
|
||||
\end{multline*}
|
||||
}
|
||||
Let $\ell_Q(v_t) = \times$, and let $\phi_Q(t) = v_t$
|
||||
|
|
@ -138,19 +143,19 @@ The runtime of any query plan $Q$ has the same or better complexity as the linea
|
|||
\end{lemma}
|
||||
\begin{proof}
|
||||
Proof by induction. The base case is a base relation: $Q = R$ and is trivially true since $|V_R| = |R|$.
|
||||
For the inductive step, we assume that we have circuits for subplans $Q_1, \ldots, Q_n$ such that $|V_{Q_i}| \leq a_i\qruntime{Q_i} + b_i$.
|
||||
For the inductive step, we assume that we have circuits for subplans $Q_1, \ldots, Q_n$ such that $|V_{Q_i}| \leq (k_i-1)\qruntime{Q_i}$ where $k_i$ is the degree of $Q_i$.
|
||||
|
||||
\caseheading{Selection}
|
||||
Assume that $Q = \sigma_\theta(Q_1)$.
|
||||
In the circuit for $Q$, $|V_Q| = |V_{Q_1}|$ vertices, so from the inductive assumption and $\qruntime{Q} = \qruntime{Q_1}$ by definition, we have $|V_Q| \leq (k-1) \qruntime{Q} $.
|
||||
\AH{Technically, $\kElem$ is the degree of $\poly_1$, but I guess this is a moot point since one can argue that $\kElem$ is also the degree of $\poly$.}
|
||||
% \AH{Technically, $\kElem$ is the degree of $\poly_1$, but I guess this is a moot point since one can argue that $\kElem$ is also the degree of $\poly$.}
|
||||
% OK: Correct
|
||||
\caseheading{Projection}
|
||||
Assume that $Q = \pi_{\vct A}(Q_1)$.
|
||||
The circuit for $Q$ has at most $|V_{Q_1}|+|\pi_{\vct A} {Q_1}| + |{Q_1}|-\abs{\pi_AQ}$ vertices.
|
||||
\AH{The combination of terms above doesn't follow the details for projection above.}
|
||||
The circuit for $Q$ has at most $|V_{Q_1}|+|{Q_1}|$ vertices.
|
||||
% \AH{The combination of terms above doesn't follow the details for projection above.}
|
||||
\begin{align*}
|
||||
|V_{Q}| & \leq |V_{Q_1}|+ \abs{Q_1} -|\pi_{\vct A} {Q_1}| \\
|
||||
& \leq |V_{Q_1}| + |Q_1|\\
|
||||
|V_{Q}| & \leq |V_{Q_1}| + |Q_1|\\
|
||||
%\intertext{By \Cref{prop:queries-need-to-output-tuples} $\qruntime{Q_1} \geq |Q_1|$}
|
||||
%& \leq |V_{Q_1}| + 2 \qruntime{Q_1}\\
|
||||
\intertext{(From the inductive assumption)}
|
||||
|
|
@ -178,7 +183,7 @@ Assume that $Q = Q_1 \bowtie \ldots \bowtie Q_k$.
|
|||
The circuit for $Q$ has $|V_{Q_1}|+\ldots+|V_{Q_k}|+(k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|$ vertices.
|
||||
\begin{align*}
|
||||
|V_{Q}| & = |V_{Q_1}|+\ldots+|V_{Q_k}|+(k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|\\
|
||||
\intertext{From the inductive assumption}
|
||||
\intertext{From the inductive assumption and noting $\forall i: k_i \leq k-1$}
|
||||
& \leq (k-1)\qruntime{Q_1}+\ldots+(k-1)\qruntime{Q_k}+\\
|
||||
&\;\;\; (k-1)|{Q_1} \bowtie \ldots \bowtie {Q_k}|\\
|
||||
& \leq (k-1)(\qruntime{Q_1}+\ldots+\qruntime{Q_k}+\\
|
||||
|
|
|
|||
|
|
@ -1,21 +1,23 @@
|
|||
% root: main.tex
|
||||
We ran our experiments using Windows 10 WSL Operating System on a machine with an Intel Core i7 2.40GHz processor with 16GB RAM. All experiments used the PostgreSQL 13.0 database system.
|
||||
|
||||
The intention of the experiments was to determine whether queries over $\bi$ instances in practice generate a lot of cancellations or not. Recall that by definition of $\bi$, a query result cannot be derived by a self-join between non-identical tuples belonging to the same block.
|
||||
|
||||
For this purpose we used the MayBMS data generator~\cite{pdbench} tool to randomly generate uncertain versions of TPCH tables. We then ran $\poly_1$, $\poly_2$, and $\poly_3$ from~\cite{Antova_fastand}, all of which are modified versions of TPC-H queries $\poly_3$, $\poly_6$, and $\poly_7$ where all aggregations have been dropped.
|
||||
Recall that by definition of $\bi$, a query result cannot be derived by a self-join between non-identical tuples belonging to the same block. Note, that by~\Cref{cor:approx-algo-const-p}, $\gamma$ must be a constant in order for~\Cref{alg:mon-sam} to acheive linear time. We would like to determine experimentally whether queries over $\bi$ instances in practice generate a constant number of cancellations or not. Such an experiment would ideally use a database instance with queries both considered to be typical representations of what is seen in practice.
|
||||
|
||||
As written, the queries disallow $\bi$ cross terms. We ran all queries, and then rewrote the queries so as not to filter out the cross terms. The results show that in practice, there are little to no cancelling terms, as shown in \Cref{fig:experiment-bidb-cancel}. The columns of the table in~\Cref{fig:experiment-bidb-cancel} show the number of result tuples returned when the query filters out tuples that are cancelled by $\bi$ constraints, the number of output tuples when the cancelled tuples are included in the result, and the difference between the two. The experiments show a range between $[0, 0.1]\%$ of tuples are cancelled tuples across the queries, suggesting that only a negligible amount of tuples are cancelled in practice when running queries over a typical $\bi$ instance. Interestingly, only one of the three queries had tuples that violated the $\bi$ constraint.
|
||||
We ran our experiments using Windows 10 WSL Operating System with an Intel Core i7 2.40GHz processor and 16GB RAM. All experiments used the PostgreSQL 13.0 database system.
|
||||
|
||||
For the data we used the MayBMS data generator~\cite{pdbench} tool to randomly generate uncertain versions of TPCH tables. The queries computed over the database instance are $\poly_1$, $\poly_2$, and $\poly_3$ from~\cite{Antova_fastand}, all of which are modified versions of TPC-H queries $\poly_3$, $\poly_6$, and $\poly_7$ where all aggregations have been dropped.
|
||||
|
||||
As written, the queries disallow $\bi$ cross terms. We first ran all queries, noting the result size for each. Next the queries were rewritten so as not to filter out the cross terms. The comparison of the sizes of both result sets should then suggest in one way or another whether or not there exist many cross terms in practice. As seen, the experimental query results contain little to no cancelling terms. \Cref{fig:experiment-bidb-cancel} shows the result sizes of the queries, where column CF is the result size when all cross terms are filtered out, column CI shows the number of output tuples when the cancelled tuples are included in the result, and the last column is the value of $\gamma$. The experiments show $\gamma$ to be in a range between $[0, 0.1]\%$, indicating that only a negligible or constant (compare the result sizes of $\poly_1 < \poly_2$ and their respective $\gamma$ values) amount of tuples are cancelled in practice when running queries over a typical $\bi$ instance. Interestingly, only one of the three queries had tuples that violated the $\bi$ constraint.
|
||||
|
||||
\begin{figure}[ht]
|
||||
\begin{tabular}{ c | c c c}\label{tbl:cancel}
|
||||
Query & Cancellations Filtered & Cancellations Included & Difference\\
|
||||
Query & CF & CI & $\gamma$\\
|
||||
\hline
|
||||
$\poly_1$ & $46,714$ & $46,768$ & $54$\\
|
||||
$\poly_2$ & $179.917$ & $179,917$ & $0$\\
|
||||
$\poly_3$ & $11,535$ & $11,535$ & $0$\\
|
||||
$\poly_1$ & $46,714$ & $46,768$ & $0.1\%$\\
|
||||
$\poly_2$ & $179.917$ & $179,917$ & $0\%$\\
|
||||
$\poly_3$ & $11,535$ & $11,535$ & $0\%$\\
|
||||
\end{tabular}
|
||||
\caption{Number of Cancellations for Queries Over $\bi$.}
|
||||
\label{fig:experiment-bidb-cancel}
|
||||
\end{figure}
|
||||
\AR{Experimental stuff about BIDB should go in here}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
\section{Missing details from Section~\ref{sec:background}}\label{sec:proofs-background}
|
||||
|
||||
\AH{I made small changes to the proof, noteably the summation, the variable definition and the world subscript, the latter of which I am not sure if it is the best notation or not.}
|
||||
\subsection{Proof of~\Cref{prop:semnx-pdbs-are-a-}}
|
||||
\AH{I made small changes to the proof, noteably the summation, the variable definition and the world subscript, the latter of which I am not sure if it is the best notation or not.}
|
||||
|
||||
To prove that $\semNX$-PDBs are complete consider the following construction that for any $\semN$-PDB $\pdb = (\idb, \pd)$ produces an $\semNX$-PDB $\pxdb = (\db, \pd')$ such that $\rmod(\pxdb) = \pdb$. Let $\idb = \{D_1, \ldots, D_{\abs{\idb}}\}$ and let $max(D_i)$ denote $max_{\tup} D_i(\tup)$. For each world $D_i$ we create a corresponding variable $X_i$.
|
||||
%variables $X_{i1}$, \ldots, $X_{im}$ where $m = max(D_i)$.
|
||||
In $\db$ we assign each tuple $\tup$ the polynomial:
|
||||
|
|
|
|||
60
intro.tex
60
intro.tex
|
|
@ -4,13 +4,15 @@
|
|||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
\AR{\textbf{Oliver/Boris:} What is missing from the intro is why would someone care about bag-PDBs in {\em practice}? This is kinda obliquely referred to in the first para but it would be good to motivate this more. The intro (rightly) focuses on the theoretical reasons to study bag PDBs but what (if any) are the practical significance of getting bag PDBs done in linear-time? Would this lead to much faster real-life PDB systems?}
|
||||
|
||||
Modern production databases like Postgres and Oracle use bag semantics, while research on probabilistic databases (PDBs)~\cite{DBLP:series/synthesis/2011Suciu,DBLP:conf/sigmod/BoulosDMMRS05,DBLP:conf/icde/AntovaKO07a,DBLP:conf/sigmod/SinghMMPHS08} focuseses predominantly on query evaluation under set semantics.
|
||||
This is not surprising, as the conventional strategy for encoding the lineage of a query result --- a key component of query evaluation in PDBs --- makes computing typical statistics like marginal probabilities or moments easy (at worst linear in the size of the lineage) for bags, but hard (at worst exponential in the size of the lineage) for sets.
|
||||
This is not surprising, as the conventional strategy for encoding the lineage of a query result --- a key component of query evaluation in PDBs --- makes computing typical statistics like marginal probabilities or moments easy (at worst linear in the size of the lineage) for bags and hence, perhaps not worthy of research attention, but hard (at worst exponential in the size of the lineage) for sets and hence, interesting from a research perspective.
|
||||
However, conventional encodings of a result's lineage are typically large, and even for Bag-PDBs, computing such statistics from lineage formulas still has a higher complexity than answering queries in a deterministic (i.e., non-probabilistic) database.
|
||||
In this paper, we formally prove this limitation of PDBs, and address it by proposing an approximation algorithm that, to the best of our knowledge, is the first $(1-\epsilon)$-approximation for expectations of counts to have a runtime within a constant factor of deterministic query processing.
|
||||
|
||||
Consider the dominant problem in Set-PDBs: Computing marginal probabilities, and the corresponding problem in Bag-PDBs: computing expectations of counts.
|
||||
In work that addresses the former problem~\cite{DBLP:series/synthesis/2011Suciu}, the lineage of a query result tuple is a boolean formula over random variables that captures the conditions under which the tuple appears in the result.
|
||||
In work that addresses the former problem~\cite{DBLP:series/synthesis/2011Suciu}, the lineage of a query result tuple is a Boolean formula over random variables that captures the conditions under which the tuple appears in the result.
|
||||
Computing the probability of the tuple appearing in the result is thus analogous to weighted model counting (a known \sharpphard problem).
|
||||
In the corresponding problem for Bag-PDBs~\cite{kennedy:2010:icde:pip,DBLP:conf/vldb/AgrawalBSHNSW06,feng:2019:sigmod:uncertainty}, lineage is a polynomial over random variables that captures the multiplicity of the output tuple.
|
||||
Thus, the expectation of the multiplicity is the expectation of this polynomial.
|
||||
|
|
@ -18,17 +20,17 @@ Thus, the expectation of the multiplicity is the expectation of this polynomial.
|
|||
Lineage in Set-PDBs is typically encoded in disjunctive normal form.
|
||||
This representation is significantly larger than the query result sans lineage.
|
||||
However, even with alternative encodings~\cite{DBLP:journals/vldb/FinkHO13}, the limiting factor in computing marginal probabilities remains the probability computation itself, and not the lineage formula.
|
||||
The corresponding lineage encoding for Bag-PDBs is a polynomial in sum of products (SOP) form --- a sum of clauses, each of which is the product of a set of integer or variable atoms.
|
||||
The corresponding lineage encoding for Bag-PDBs is a polynomial in sum of products (SOP) form --- a sum of `clauses', each of which is the product of a set of integer or variable atoms.
|
||||
Thanks to linearity of expectation, computing the expectation of a count query is linear in the number of clauses in the SOP polynomial.
|
||||
Unlike Set-PDBs, however, when we consider compressed representations of this polynomial, the complexity landscape becomes much more nuanced and is \textit{not} linear in general.
|
||||
Such compressed representations like Factorized Databases~\cite{10.1145/3003665.3003667,DBLP:conf/tapp/Zavodny11} or Polynomial Circuits\todo[noinline]{cite}, are analogous to deterministic query optimizations (e.g. pushing down projections)~\cite{DBLP:conf/pods/KhamisNR16,10.1145/3003665.3003667}.
|
||||
Such compressed representations like Factorized Databases~\cite{10.1145/3003665.3003667,DBLP:conf/tapp/Zavodny11} or Arithmetic/Polynomial Circuits~\cite{arith-complexity}, are analogous to deterministic query optimizations (e.g. pushing down projections)~\cite{DBLP:conf/pods/KhamisNR16,10.1145/3003665.3003667}.
|
||||
Thus, measuring the performance of a PDB algorithm in terms of the size of the \emph{compressed} lineage formula allows us to more closely relate the algorithm's performance to the complexity of query evaluation in a deterministic database.
|
||||
|
||||
The initial picture is not good.
|
||||
In this paper, we prove that computing expected counts is \emph{not} linear in the size of a compressed --- specifically a factorized~\cite{10.1145/3003665.3003667} --- lineage polynomial by reduction to counting 3-matchings.
|
||||
In this paper, we prove that computing expected counts is \emph{not} linear in the size of a compressed --- specifically a factorized~\cite{10.1145/3003665.3003667} --- lineage polynomial by reduction from counting $k$-matchings.
|
||||
Thus, even bag PDBs do not enjoy the same computational complexity as deterministic databases.
|
||||
This motivates our second goal, a linear time approximation algorithm for computing expected counts in a bag database, with complexity linear in the size of a factorized lineage formula.
|
||||
As we show (\Cref{prop:queries-need-to-output-tuples}), the size of the factorized
|
||||
As we will show, the size of the factorized
|
||||
lineage formula for a query --- and by extension, our approximation algorithm --- is proportional to the complexity of evaluating the same query on a comparable deterministic database instance~\cite{DBLP:conf/pods/KhamisNR16,10.1145/3003665.3003667}.
|
||||
In other words, our approximation algorithm can estimate expected multiplicities for tuples in the result of an SPJU query with a complexity comparable to deterministic query-processing.
|
||||
|
||||
|
|
@ -97,23 +99,23 @@ In other words, our approximation algorithm can estimate expected multiplicities
|
|||
|
||||
\begin{Example}\label{ex:intro}
|
||||
Consider the Tuple Independent ($\ti$) Set-PDB\footnote{Our work also handles Block Independent Disjoint Databases ($\bi$)~\cite{DBLP:conf/sigmod/BoulosDMMRS05,DBLP:series/synthesis/2011Suciu}, we return to this model later.} given in \Cref{fig:intro-ex} with two input relations $R$ and $E$.
|
||||
Each input tuple is assigned an annotation (attribute $\Phi$): an independent random boolean variable ($W_i$) or the constant $\top$.
|
||||
Each input tuple is assigned an annotation (attribute $\Phi$): an independent random Boolean variable ($W_i$) or the constant $\top$.
|
||||
Each assignment of values to variables ($\{\;W_a,W_b,W_c\;\}\mapsto \{\;\top,\bot\;\}$) \SF{Do we need to state the meaning of $\top$ and $\bot$? Also do we want to add bag annotation to Figure 1 too since we are discussing both sets and bags later?} identifies one \emph{possible world}, a deterministic database instance that contains exactly the tuples annotated by the constant $\top$ or by a variable assigned to $\top$.
|
||||
The probability of this world is the joint probability of the corresponding assignments.
|
||||
For example, let $P[W_a] = P[W_b] = P[W_c] = p$ and consider the possible world where $R = \{\;\tuple{a}, \tuple{b}\;\}$.
|
||||
The corresponding variable assignment is $\{\;W_a \mapsto \top, W_b \mapsto \top, W_c \mapsto \bot\;\}$, and the probability of this world is $P[W_a]\cdot P[W_b] \cdot P[\neg W_c] = p^2-p^3$
|
||||
The corresponding variable assignment is $\{\;W_a \mapsto \top, W_b \mapsto \top, W_c \mapsto \bot\;\}$, and the probability of this world is $P[W_a]\cdot P[W_b] \cdot P[\neg W_c] = p\cdot p\cdot (1-p)=p^2-p^3$.
|
||||
\end{Example}
|
||||
|
||||
Prior efforts to generalize incomplete databases to bags~\cite{feng:2019:sigmod:uncertainty,DBLP:conf/pods/GreenKT07,DBLP:journals/sigmod/GuagliardoL17} replace the boolean annotations with natural numbers.
|
||||
Prior efforts to generalize incomplete databases to bags~\cite{feng:2019:sigmod:uncertainty,DBLP:conf/pods/GreenKT07,DBLP:journals/sigmod/GuagliardoL17} replace the Boolean annotations with natural numbers.
|
||||
Analogously, we generalize the above model of Set-PDBs to bags by using natural-number-valued random variables (i.e., $Dom(W_i) \subseteq \mathbb N$) and positive natural number constants.
|
||||
Without loss of generality, we assume that input relations are sets (i.e. $Dom(W_i) = \{0, 1\}$), while query evaluation follows bag semantics.
|
||||
We contrast bag and set query evaluation with the following example:
|
||||
|
||||
\begin{Example}\label{ex:bag-vs-set}
|
||||
Continuing the prior example, we are given the following boolean (resp,. count) query
|
||||
Continuing the prior example, we are given the following Boolean (resp,. count) query
|
||||
$$\poly() :- R(A), E(A, B), R(B)$$
|
||||
The lineage of the result in a Set-PDB (resp., Bag-PDB) is a boolean (resp., polynomial) formula over random variables annotating the input relations (i.e., $W_a$, $W_b$, $W_c$).
|
||||
Because the boolean query has only a nullary relation, we write $Q(\cdot)$ to denote the function mapping variable assignments to a concrete value for the lineage in the corresponding possible world:
|
||||
The lineage of the result in a Set-PDB (resp., Bag-PDB) is a Boolean (resp., polynomial) formula over random variables annotating the input relations (i.e., $W_a$, $W_b$, $W_c$).
|
||||
Because the Boolean query has only a nullary relation, we write $Q(\cdot)$ to denote the function mapping variable assignments to a concrete value for the lineage in the corresponding possible world:
|
||||
\begin{align*}
|
||||
\poly_{set}(W_a, W_b, W_c) &= W_aW_b \vee W_bW_c \vee W_cW_a\\
|
||||
\poly_{bag}(W_a, W_b, W_c) &= W_aW_b + W_bW_c + W_cW_a
|
||||
|
|
@ -126,7 +128,7 @@ The polynomials evaluate as:
|
|||
&\poly_{bag}(1, 1, 0) = 1 \cdot 1 + 1\cdot 0 + 0 \cdot 1 = 1
|
||||
\end{align*}
|
||||
The Set-PDB query is satisfied in this possible world, while the Bag-PDB query produces a nullary tuple with a multiplicity of 1.
|
||||
The marginal probability (resp., expected count) of this query is computed over all possible worlds:
|
||||
The marginal probability (resp., expected count) of this query is computed over all possible worlds:\AR{What is $\mu$ below?}
|
||||
{\small
|
||||
\begin{align*}
|
||||
P[\poly_{set}] &= \sum_{w_i \in \{\top,\bot\}} \mu(\poly_{set}(w_a, w_b, w_c))P[W_a = w_a,W_b = w_b,W_c = w_c]\\
|
||||
|
|
@ -136,7 +138,7 @@ P[\poly_{set}] &= \sum_{w_i \in \{\top,\bot\}} \mu(\poly_{set}(w_a, w_b, w_c))P[
|
|||
\end{Example}
|
||||
|
||||
Note that the query of \Cref{ex:bag-vs-set} in set semantics is indeed \sharpphard, since it non-hierarchical~\cite{10.1145/1265530.1265571}.
|
||||
To see why computing this probability is hard, observe that the clauses of the disjunctive normal form boolean lineage are neither independent nor disjoint, forcing~\cite{DBLP:journals/vldb/FinkHO13} the use of Shannon decomposition, which is at worst exponential in the size of the input.
|
||||
To see why computing this probability is hard, observe that the clauses of the disjunctive normal form Boolean lineage are neither independent nor disjoint, leading to e.g.~\cite{DBLP:journals/vldb/FinkHO13} the use of Shannon decomposition, which is at worst exponential in the size of the input.
|
||||
% \begin{equation*}
|
||||
% \expct\pbox{\poly(W_a, W_b, W_c)} = W_aW_b + W_a\overline{W_b}W_c + \overline{W_a}W_bW_c = 3\prob^2 - 2\prob^3
|
||||
% \end{equation*}
|
||||
|
|
@ -206,7 +208,7 @@ As a further interesting feature of this example, note that $\expct\pbox{W_i} =
|
|||
|
||||
\subsection{Superlinearity of Bag PDBs}
|
||||
Moving forward, we focus exclusively on bags and drop the subscript from $\poly_{bag}$.
|
||||
Consider the cartesian product of $\poly$ with itself:
|
||||
Consider the Cartesian product of $\poly$ with itself:
|
||||
\begin{equation*}
|
||||
\poly^2() := \rel(A), E(A, B), \rel(B),\; \rel(C), E(C, D), \rel(D)
|
||||
\end{equation*}
|
||||
|
|
@ -218,8 +220,8 @@ For example:
|
|||
\poly^2(W_a, W_b, W_c) = \left(W_aW_b + W_bW_c + W_cW_a\right) \cdot \left(W_aW_b + W_bW_c + W_cW_a\right).
|
||||
\end{equation*}
|
||||
}
|
||||
This factorized expression can be easily modeled as an expression tree, as in \Cref{fig:intro-q2-etree}.
|
||||
In contrast, the equivalent SOP representation is
|
||||
This factorized expression can be easily modeled as an expression tree, as in \Cref{fig:intro-q2-etree}
|
||||
while the equivalent SOP representation is
|
||||
\begin{equation*}
|
||||
W_a^2W_b^2 + W_b^2W_c^2 + W_c^2W_a^2 + 2W_a^2W_bW_c + 2W_aW_b^2W_c + 2W_aW_bW_c^2.
|
||||
\end{equation*}
|
||||
|
|
@ -246,15 +248,16 @@ With $\poly^2$ as an example, we have:
|
|||
\end{align*}
|
||||
\SF{Should this be like $\tilde{\poly^2}$ to avoid ambiguous?}
|
||||
Observe that the reduced polynomial is a closed form formula for the expected count (i.e., $\expct\pbox{\poly^2} = \rpoly(P\pbox{W_a=1}, P\pbox{W_b=1}, P\pbox{W_c=1})$).
|
||||
Also note that our initial example polynomial $\poly$ is already in reduced form.
|
||||
Also note that the $\poly$ in~\Cref{ex:bag-vs-set} is already in reduced form.
|
||||
|
||||
The reduced form of a polynomial can be obtained in a linear scan over the clauses of a SOP encoding of the polynomial.
|
||||
In prior work on PDBs, where this encoding is implicitly assumed, computing the expected count is linear in the size of the encoding.
|
||||
In general however, compressed encodings of the polynomial can be exponentially smaller in $k$ for $k$-products --- the query $\poly^k$ obtained by taking the cartesian product of $k$ copies of $\poly$ has a factorized encoding of size $6\cdot k$, while the SOP encoding is of size $2\cdot 3^k$.
|
||||
This leads us to the central question of this paper:
|
||||
\begin{Question}
|
||||
Is it always the case that the expectation of a nullary count query in a Bag-PDB can be computed in time linear in the size of the \emph{compressed} lineage polynomial?
|
||||
\end{Question}
|
||||
In general however, compressed encodings of the polynomial can be exponentially smaller in $k$ for $k$-products --- the query $\poly^k$ obtained by taking the Cartesian product of $k$ copies of $\poly$ has a factorized encoding of size $6\cdot k$, while the SOP encoding is of size $2\cdot 3^k$.
|
||||
This leads us to the \textbf{central question of this paper}:
|
||||
\begin{quote}
|
||||
{\em
|
||||
Is it always the case that the expectation of an UCQ in a Bag-PDB can be computed in time linear in the size of the \emph{compressed} lineage polynomial?}
|
||||
\end{quote}
|
||||
If the answer is yes, then it is possible for Bag-PDBs to achieve performance competitive with deterministic databases.
|
||||
The answer, unfortunately, is no, and an approximation algorithm is required.
|
||||
|
||||
|
|
@ -264,12 +267,21 @@ The answer, unfortunately, is no, and an approximation algorithm is required.
|
|||
% \end{equation*}
|
||||
% The factorized output polynomial consists of a product of three identical three-way summations, while the SOP encoding is exponential --- $3^3$ clauses to be precise.
|
||||
|
||||
\subsection{Overview of our results and techniques}
|
||||
Concretely, in this paper:
|
||||
(i) We show that conjunctive queries over a bag-$\ti$ are hard (i.e., superlinear in the size of a compressed lineage encoding) by reduction to counting the number of $3$-matchings over an arbitrary graph;
|
||||
(i) We show that conjunctive queries over a bag-$\ti$ are hard (i.e., superlinear in the size of a compressed lineage encoding) by reduction from counting the number of $k$-matchings over an arbitrary graph;
|
||||
(ii) We present an $(1-\epsilon)$-approximation algorithm for bag-$\ti$s and show that its complexity is linear in the size of the compressed lineage encoding;
|
||||
(iii) We generalize the approximation algorithm to bag-$\bi$s, a more general model of probabilistic data;
|
||||
(iv) We further generalize our results to higher moments, polynomial circuits, and prove RA+ queries, the processing time in approximation is within a constant factor of the same query processed deterministically.
|
||||
|
||||
Our hardness results follow by considering a suitable generalization of the lineage polynomial in Example~\ref{ex:bag-vs-set}. First it is easy to generalize the polynomial in Example~\ref{ex:bag-vs-set} to $\poly_G^k(X_1,\dots,X_n)$ that represents the edge set of a graph $G$ in $n$ vertices. Then $\inparen{\poly_G^k(X_1,\dots,X_n)}^k$ encodes as its monomials all subgraphs of $G$ with at most $k$ edges in it. This implies that the corresponding reduced polynomial $\rpoly_G^k(p,\dots,p)$ can be written as $\sum_{i=0}^2k c_i\cdot p^i$ and we observe that $c_{2k}$ is proportional to the number of $k$-matchings (computing which is \sharpwonehard\ ) in $G$. Thus, if we have access to $\rpoly_G^k(p_i,\dots,p_)$ for distinct values of $p_i$ for $0\le i\le 2k$, then we can setup a system of linear equations and compute $c_{2k}$ (and hence the number of $k$-matchings in $G$). This result, however, does not rule out the possibility that computing $\rpoly_G^k(p,\dots,p)$ for a {\em single specific} value of $p$ might be easy: indeed it is easy for $p=0$ or $p=1$. However, we are able to show that for any other value of $p$, computing $\rpoly_G^k(p,\dots,p)$ exactly will most probably require super-linear time. This reduction needs more work (and we cannot yet extend our results to $k>3$). Further, we have to rely on more recent conjectures in {\em fine-grained} complexity on e.g. the complexity of counting the number of triangles in $G$ and not more standard parameterized hardness like \sharpwonehard.
|
||||
|
||||
|
||||
The starting point of our approximation algorithm was the simple observation that for any lineage polynomial $\poly(X_1,\dots,X_n)$, we have $\rpoly(1,\dots,1)=Q(1,\dots,1)$ and if all the coefficients of $\poly$ are constants then $\poly(X_1,\dots,X_n)$ (which can be easily computed in linear time) is a $p^k$ approximation to the value $\rpoly(p,\dots,p)$ that we are after. If $p$ and $k=\deg(\poly)$ are constants, then this gives a constant factor approximation. We then use sampling to get a better approximation factor of $(1\pm \eps)$: we sample monomials from $\poly(X_1,\dots,X_n)$ and do an appropriate weighted sum of their coefficients. Standard tail bounds then allow us to get our desired approximation scheme. To get a linear runtime, it turns out that we need the following properties from our compressed representation of $\poly$: (i) be able to compute $\poly(X_1,\dots,X_n)$ in linear time and (ii) be able to sample monomials from $\poly(X_1,\dots,X_n)$ quickly as well. For the ease of exposition, we start off with expression trees (see~\Cref{fig:intro-q2-etree} for an example) and show that they satisfy both of these properties. Later we show that it easy to show that these properties also extend to polynomial circuits as well (we essentially show that in the requires time bound, we can simulate access to the `unrolled' expression tree by considering the polynomial circuit).
|
||||
|
||||
We also formalize our claim since our approximation algorithm runs in time linear in the size of the polynomial circuit, we show that we can approximate the expected output tuple multiplicities with only a $O(\log{Z})$ overhead (where $Z$ is the number of output tuples) over the runtime of a broad class of query processing algorithms. We also observe that our results trivially extend to the problem of computing higher moments of the tuple multiplicity (instead of just the expectation).
|
||||
|
||||
\paragraph{Paper Organization.} We present some relevant background and setup our notation in~\Cref{sec:background}. We present our hardness results in~\Cref{sec:hard} and our approximation algorithm in~\Cref{sec:algo}. We present some (easy) generalizations of our results in~\Cref{sec:gen}. We do a quick overview of related work in~\Cref{sec:related-work} and conclude with some open questions in~\Cref{sec:concl-future-work}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
|
|
|
|||
|
|
@ -253,6 +253,11 @@
|
|||
\newcommand{\SR}[1]{\todo[inline, backgroundcolor=white]{\textbf{Note to self:$\,$} #1}}
|
||||
\newcommand{\AR}[1]{\todo[inline, color=green]{\textbf{Atri says:$\,$} #1}}
|
||||
|
||||
%\newcommand{\AR}[1]{}
|
||||
%\newcommand{\AH}[1]{}
|
||||
%\newcommand{\SF}[1]{}
|
||||
%\newcommand{\BG}[1]{}
|
||||
|
||||
|
||||
|
||||
%
|
||||
|
|
|
|||
3
main.tex
3
main.tex
|
|
@ -87,7 +87,8 @@ sensitive=true
|
|||
%%%%%%%%%%%%%%%%%%%%
|
||||
% \textbullet Modelling Uncertainty as Attribute-level Taints and its Relationship to Provenance}
|
||||
|
||||
\title{Standard Operating Procedure in PDBs Considered Harmful for Bags}
|
||||
\title{Standard Operating Procedure in PDBs Considered Harmful}
|
||||
\subtitle{(for bags)}
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%%%%%%%%%%%%%%%%%%%%
|
||||
|
|
|
|||
|
|
@ -65,13 +65,24 @@ $\poly(\vct{X})$ = $\poly(X_{\block_1, 1},\ldots, X_{\block_1, \abs{\block_1}},$
|
|||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
\begin{definition}[Modding with a set]\label{def:mod-set}
|
||||
Let $S$ be a {\em set} of polynomials over $\vct{X}$. Then $\poly(\vct{X})\mod{S}$ is the polynomial obtained by taking the mod of $\poly(\vct{X})$ over {\em all} polynomials in $S$ (the order does not matter).
|
||||
\end{definition}
|
||||
For example when $S_0=\inset{X^2-X, Y^2-Y}$, taking the polynomial in~\cref{eq:poly-eg} mod $S_0$, we get $2X+3XY-2Y$.
|
||||
|
||||
\begin{Definition}\label{def:mod-set-polys}
|
||||
Given the set of BIDB variables $\inset{X_{b,i}}$, define
|
||||
\[\mathcal{B}=\inset{X_{b,i}\cdot X_{b,j}|\text{ for every block } b \text{and } i\ne j},\]
|
||||
\[\mathcal{T}=\inset{X_{b,i}^2-X_{b,i}|\text{ for every block } b \text{and } i}.\]
|
||||
\end{Definition}
|
||||
|
||||
\begin{Definition}[Reduced \bi Polynomials]\label{def:reduced-bi-poly}
|
||||
Let $\poly(\vct{X})$ be a \bi-lineage polynomial.
|
||||
The reduced form $\rpoly(\vct{X})$ of $\poly(\vct{X})$ is defined as
|
||||
\begin{equation*}
|
||||
\rpoly(\vct{X}) = \smbOf{\poly(\vct{X})} \mod X_i^2 - X_i \mod X_{\block_s, t}X_{\block_s, u}
|
||||
\rpoly(\vct{X}) = \smbOf{\poly(\vct{X})} \mod \mathcal{T} \mod \mathcal{B}%X_i^2 - X_i \mod X_{\block_s, t}X_{\block_s, u}
|
||||
\end{equation*}
|
||||
for all $i$ in $[\numvar]$ and for all $s$ in $\ell$, such that for all $t, u$ in $[\abs{\block_s}]$, $t \neq u$.
|
||||
%for all $i$ in $[\numvar]$ and for all $s$ in $\ell$, such that for all $t, u$ in $[\abs{\block_s}]$, $t \neq u$.
|
||||
\end{Definition}
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
|
|
@ -125,7 +136,7 @@ Note the following fact:
|
|||
\]
|
||||
\end{Proposition}
|
||||
|
||||
The proof is in~\Cref{sec:proofs-background}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
|
@ -147,7 +158,7 @@ Note that in the preceding lemma, we have assigned $\vct{p}$
|
|||
%(introduced in \Cref{subsec:def-data})
|
||||
to the variables $\vct{X}$. Intuitively, \Cref{lem:exp-poly-rpoly} states that when we replace each variable $X_i$ with its probability $\prob_i$ in the reduced form of a \bi-lineage polynomial and evaluate the resulting expression in $\mathbb{R}$, then the result is the expectation of the polynomial.
|
||||
|
||||
The proof for~\Cref{lem:exp-poly-rpoly} can be seen in~\Cref{sec:proofs-background}.
|
||||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
|
|
@ -161,7 +172,7 @@ If $\poly$ is a \bi-lineage polynomial, then the expectation of $\poly$, i.e., $
|
|||
\end{Corollary}
|
||||
\AH{What if $\poly$ is not in \abbrSMB form?}
|
||||
|
||||
We defer the proof for~\Cref{cor:expct-sop} in~\Cref{sec:proofs-background}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
|
|
|||
|
|
@ -51,6 +51,8 @@ We use $\evald{\cdot}{\db}$ to denote the result of evaluating query $\query$ ov
|
|||
Let $\semNX$ denote the set of polynomials over variables $\vct{X}$ with natural number co-efficients and exponents.
|
||||
Consider now the semiring $(\semNX, +, \cdot, 0, 1)$ whose domain is $\semNX$ and addition and multiplication are standard addition and multiplication of polynomials. We will utilize $\semNX$-databases $\db$ paired with probability distributions to represent $\semN$-PDBs.\BG{Need more motivation?} To justify the use of $\semNX$-databases, we need to show that we can encode any $\semN$-PDB in this way and that the query semantics over this representation coincides with query semantics over $\semN$-PDB. For that it will be opportune to define representation systems for $\semN$-PDBs.\BG{cite}
|
||||
|
||||
Before we proceed, unless otherwise mentioned, all subsequent proofs for~\Cref{sec:background} can be found in~\Cref{sec:proofs-background}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
\begin{Definition}[Representation System]\label{def:representation-syste}
|
||||
A representation system for $\semN$-PDBs is a tuple $(\reprs, \rmod)$ where $\reprs$ is a set of representations and $\rmod$ associates with each $\repr \in \reprs$ an $\semN$-PDB $\pdb$. We say that a representation system is \emph{closed} under a class of queries $\qClass$ if for any query $\query \in \qClass$ we have:
|
||||
|
|
@ -91,7 +93,7 @@ Importantly, as the following proposition shows, any finite $\semN$-PDB can be e
|
|||
$\semNX$-PDBs are a complete representation system for $\semN$-PDBs that is closed under $\raPlus$ queries.
|
||||
\end{Proposition}
|
||||
|
||||
The proof can be seen in~\Cref{sec:proofs-background}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
|
@ -110,7 +112,7 @@ Since $\semNX$-PDBs $\pxdb$ are a complete representation system for $\semN$-PDB
|
|||
\[ \expct_{\idb \sim \pd}[\query(\db)(t)] = \expct_{\vct{X} \sim \pd'}\pbox{\poly(\vct{X})} \]
|
||||
\end{Proposition}
|
||||
|
||||
The proof can be seen in~\Cref{sec:proofs-background}.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
|
|
|||
Loading…
Reference in a new issue