ODIn
/
paper-BagRelationalPDBsAreHard
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Changed \pdb to \dbbase in S 1 where appropriate.

master
Aaron Huber 2021-09-09 09:37:45 -04:00
parent bfbf92b0cb
commit d7b906dd41
4 changed files with 35 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
intro-rewrite-070921.tex
View File

@ -22,10 +22,10 @@ of tuple $\tup$.
We are interested in the data complexity of this problem (i.e. we think of $Q$ as being of constant size). Unless stated otherwise, we implicitly assume the probability distribution $\pd$, and for notational convenience use $\expct\pbox{\cdot}$ instead of $\expct_\pd\pbox{\cdot}$.
A common encoding of probabilistic databases (e.g., in \cite{IL84a,Imielinski1989IncompleteII,Antova_fastand,DBLP:conf/vldb/AgrawalBSHNSW06} and many others) relies on annotating tuples with lineages, propositional formulas that describe the set of possible worlds that the tuple appears in.
Each valuation of the random variables appearing in this formula corresponds to one possible world.
Given a joint probability distribution over such assignments, the marginal probability of a query result tuple $\tup$ is the probability that the lineage formula of $\tup$ evaluates to true.
Given a joint probability distribution over such assignments, the marginal probability of a query result tuple $\tup$ is the probability that the lineage formula of $\tup$ evaluates to true. Given a \abbrBPDB $\pdb$, we refer to the above encoding of $\pdb$ as \dbbaseName and denote it as $\dbbase$.
The bag semantics analog of a lineage formula is a provenance polynomial $\apolyqdt$~\cite{DBLP:conf/pods/GreenKT07}, a polynomial with integer coefficients and exponents over integer random variables $\vct{\randWorld}$ encoding the multiplicity of input tuples.
Analog to set-semantics, computing the expected multiplicity of a tuple reduces to computing the expectation of this polynomial. We drop $\query$, $\pdb$, and $\tup$ from $\apolyqdt$ when they are clear from the context or irrelevant to the discussion.
Analog to set-semantics, computing the expected multiplicity of a tuple reduces to computing the expectation of this polynomial. We drop $\query$, $\dbbase$, and $\tup$ from $\apolyqdt$ when they are clear from the context or irrelevant to the discussion.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@ -34,7 +34,7 @@ Given an $\raPlus$ query $\query$, \abbrBPDB $\pdb$, and output tuple $\tup$, co
multiplicity of $\apolyqdt$ ($\expct_\pd\pbox{\apolyqdt}$).
\end{Problem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\AH{I think that \Cref{prob:bag-pdb-poly-expected} needs to define the all worlds distribution $\pdassign$ over the set $\vct{W}\in\{0, 1\}^\numvar$, as well as the assumption or justification that $\pd \equiv \pdassign$.}
\AH{I think that \Cref{prob:bag-pdb-poly-expected} needs to define the all worlds distribution $\pdassign$ over the set $\vct{W}\in\{0, 1\}^\numvar$, as well as the assumption or justification that $\pd \equiv \pdassign$. The prose ``propositional fomulas that dscribe the set of possible worlds...'' perhaps `justifies' using $\pd$.}
Note that, if $\apolyqdt$ is given, then \Cref{prob:bag-pdb-query-eval} reduces to \Cref{prob:bag-pdb-poly-expected} (see \Cref{subsec:expectation-of-polynom-proof} for the proof). Evaluating queries over probabilistic databases in this fashion (first computing a tuple's lineage and then calculating the expectation of the lineage) has been referred to as \textit{intensional query evaluation}~\cite{DBLP:series/synthesis/2011Suciu}. In this work, we study the complexity of \Cref{prob:bag-pdb-poly-expected} for several models of probabilistic databases and various encodings of such polynomials, considering the size of the encoding as the input size. % specifically, the bag semantics version of tuple-independent probabilistic bag-databases (\abbrTIDB) and block-independent probabilistic databases (\abbrBIDB).
% Our main technical focus is on studying the complexity of this problem for various encoding of such polynomials.
@ -42,7 +42,7 @@ However, as we will show, these results have implications for the complexity of
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\mypar{\abbrTIDB\xplural}
%Solving~\cref{prob:bag-pdb-query-eval} for arbitrary $\pd$ is hopeless since we need exponential space to repreent an arbitrary $\pd$.
%Solving~\Cref{prob:bag-pdb-query-eval} for arbitrary $\pd$ is hopeless since we need exponential space to repreent an arbitrary $\pd$.
We initially focus on tuple-independent probabilistic bag-databases\footnote{See \cite{DBLP:series/synthesis/2011Suciu} for a survey of set-\abbrTIDBs; The bag encoding is analogous~\cite{DBLP:conf/pods/GreenKT07}.} (\abbrTIDB), a compressed encoding of probabilistic databases where the presence of each individual tuple (out of a total of $\numvar$ input tuples) in a possible world is modeled as an independent probabilistic event.\footnote{
This model is exactly the definition of \abbrTIDB{}s \cite{VS17} under classical set semantics.
Mirroring the implementation of bag relations in production database systems (e.g., Postgresql, DB2), we model tuples with possible multiplicities greater than one by replacing each input tuple with as many copies as its largest possible multiplicity.
@ -53,7 +53,9 @@ We initially focus on tuple-independent probabilistic bag-databases\footnote{See
}
% OK: I tidied things up a touch.
%\BG{The footnote is still a bit hard to follow I think, but I do not have a great suggestion on how to improve it.}
We will denote the $n$ tuples in the database by $t_1,\dots,t_\numvar$. Each of the $2^n$ possible worlds in $\Omega$ can be encoded as a string in $\{0,1\}^\numvar$. In particular, any vector $\vct{W}=\inparen{W_1,\dots,W_n}\in \{0,1\}^\numvar$ represents a world that has $\tup_i$ in it iff $W_i=1$. Furthermore, $\pd$ is compactly described by a tuple $\vct{p}=\inparen{p_1,\dots,p_n}$, which induces the Bernoulli distribution over vectors $\vct{W}\in\{0,1\}^\numvar$ where each $i\in [n]$, $\probOf(W_i=1)=p_i$. Finally for each $\vct{W}\in\{0,1\}^\numvar$, we define $\pdb_{\vct{W}}$ as the world represented by $\vct{W}$.
We will denote the $n$ tuples in the database by $t_1,\dots,t_\numvar$. Each of the $2^n$ possible worlds in $\Omega$ can be encoded as a string in $\{0,1\}^\numvar$. In particular, any vector $\vct{W}=\inparen{W_1,\dots,W_n}\in \{0,1\}^\numvar$ represents a world that has $\tup_i$ in it iff $W_i=1$. Furthermore, $\pd$ is compactly described by a tuple $\vct{p}=\inparen{p_1,\dots,p_n}$, which induces the Bernoulli distribution over vectors $\vct{W}\in\{0,1\}^\numvar$ where each $i\in [n]$, $\probOf(W_i=1)=p_i$. Finally for each $\vct{W}\in\{0,1\}^\numvar$, we define $\pdb_{\vct{W}}$
\AH{Where do we use this notation? If we use this somewhere, should we maybe use $\db_{\vct{\randWorld}}$ instead?}
as the world represented by $\vct{W}$.
%Atri: Stuff below was confusing, so am re-writing it.
%A \abbrTIDB encodes a compatible $\pdb$ as a deterministic database $\encodedDB$ with $\numvar$ tuples, each annotated with a probability $\prob_\tup$, and with $\pd$
@ -64,7 +66,7 @@ We will denote the $n$ tuples in the database by $t_1,\dots,t_\numvar$. Each of
\BG{REMOVED:
When $\pdb$ is a \abbrTIDB, for every output tuple $\tup$, $\query\inparen{\pdb}\inparen{\tup}$ can be encoded by a polynomial, with variables in $\vct{X}$.
Green, Karvounarakis, and Tannen established (\cite{DBLP:conf/pods/GreenKT07}; see \cref{fig:nxDBSemantics}) that for any $\raPlus$ query $\query$ and \abbrTIDB $\pdb$, there exists a polynomial $\poly_\tup\inparen{\vct{X}}$ following the standard addition and multiplication operators over Natural numbers (i.e., $\semN$-semiring semantics), such that $\query\inparen{\pdb_{\vct{W}}}\inparen{\tup} = \poly_\tup\inparen{\vct{W}}$.
Green, Karvounarakis, and Tannen established (\cite{DBLP:conf/pods/GreenKT07}; see \Cref{fig:nxDBSemantics}) that for any $\raPlus$ query $\query$ and \abbrTIDB $\pdb$, there exists a polynomial $\poly_\tup\inparen{\vct{X}}$ following the standard addition and multiplication operators over Natural numbers (i.e., $\semN$-semiring semantics), such that $\query\inparen{\pdb_{\vct{W}}}\inparen{\tup} = \poly_\tup\inparen{\vct{W}}$.
This in turn implies that $\expct\pbox{\query\inparen{\pdb}\inparen{\tup}} = \expct_{\vct{W}\sim\pd}\pbox{\poly_\tup\inparen{\vct{\randWorld}}}$.}
Thanks to linearity of expectation, simple polynomial-time algorithms exist for computing the expectation of a lineage polynomial $\apolyqdt$ when $\pdb$ is a \abbrTIDB and $\query$ is an $\raPlus$ query.
@ -73,7 +75,7 @@ Thanks to linearity of expectation, simple polynomial-time algorithms exist for
% for computing exact results for bag-probabilistic count queries $Q$ over \abbrTIDB{}s.
However, it is also known that since we are considering data complexity, that {\em deterministic} query processing for the same query $Q$ can also be done in polynomial time.
If our notion of efficiency were simply achieving a polynomial time algorithm, then we would be done.
However, in practice (and in theory), we care about the {\em fine-grained} complexity of deterministic query processing (i.e. we care about the exact exponent in our polynomial runtime). Given that there is huge literature on fine grained complexity of deterministic query processing, a natural (informal) specialization of~\cref{prob:bag-pdb-query-eval} is:
However, in practice (and in theory), we care about the {\em fine-grained} complexity of deterministic query processing (i.e. we care about the exact exponent in our polynomial runtime). Given that there is huge literature on fine grained complexity of deterministic query processing, a natural (informal) specialization of~\Cref{prob:bag-pdb-query-eval} is:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{Problem}[Informal problem statement]\label{prob:informal}
@ -87,7 +89,7 @@ For any query $\query$, is it the case that the {\em fine-grained complexity} of
%The problem of deterministic query evaluation is known to be \sharpwonehard\footnote{A problem is in \sharpwone if the runtime of the most efficient known algorithm to solve it is lower bounded by some function $f$ of a parameter $k$, where the growth in runtime is polynomially dependent on $f(k)$, i.e. $\Omega\inparen{\numvar^{f(k)}}$.} in data complexity for general $\query$. For example, the counting $k$-cliques query problem (where the parameter $k$ is the size of the clique) is \sharpwonehard since (under standard complexity assumptions) it cannot run in time faster than $n^{f(k)}$ for some strictly increasing $f(k)$.
%In this paper, we begin to explore whether the problem of bag-probabilistic query evaluation (which we relate to deterministic query processing more precisely below) falls into this same complexity class.
We note that an answer in the affirmative for~\cref{prob:informal} indicates that bag-probabilistic databases can be competitive with classical deterministic databases, opening the door for deployment in practice.
We note that an answer in the affirmative for~\Cref{prob:informal} indicates that bag-probabilistic databases can be competitive with classical deterministic databases, opening the door for deployment in practice.
% Atri: Converting sub-section to para since it saves space
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@ -102,7 +104,7 @@ and $\expct\pbox{\apolyqdt\inparen{\vct{\randWorld}}}$ is the marginal probabil
%Atri: Again if we have a reviewer who does not know what \sharpp is then we are in trouble
%\footnote{\sharpp is the counting version for problems residing in the NP complexity class.}
in general, and proved that a dichotomy exists for this problem for the class of union of conjunctive queries (with the same expressive power as $\raPlus$), where the runtime of $\query(\pdb)$ is either polynomial or \sharpphard in data complexity. %for any polynomial-time deterministic query.
Thus, for the hard queries, the answer to~\cref{prob:informal} is {\em no} for set-PDBs (under the standard complexity assumption that $\sharpp\ne \polytime$).
Thus, for the hard queries, the answer to~\Cref{prob:informal} is {\em no} for set-PDBs (under the standard complexity assumption that $\sharpp\ne \polytime$).
Concretely, easy queries in this dichotomy can be answered through so-called \emph{extensional} query evaluation, where probability computation is inlined into normal deterministic query processing.
This is possible, because queries on the easy side of the dichotomy can always be rewritten into a form that guarantees that, for every relational operator in the query, the presence of every tuple in the operator's output is governed by either a conjunction or disjunction of \emph{independent} events.
@ -127,14 +129,14 @@ However, there exist some queries for which \abbrBPDB\xplural are a more natural
%END Needs to be noted.
% Atri: Removing stuff below as per conversation with Oliver on matrix on Aug 26
%A natural question is whether or not we can quantify the complexity of computing $\expct\pbox{\poly_\tup\inparen{\vct{\randWorld}}}$ separately from the complexity of deterministic query evaluation, effectively dividing \abbrPDB query evaluation into two steps: deterministic query evaluation\footnote{Given input $\pdb$, this step includes outputting every tuple $\tup$ that satisfies $\query$, annotated with its lineage polynomial ($\poly_\tup$) which is computed inline across the query operators of $\query$.\cite{Imielinski1989IncompleteII}\cite{DBLP:conf/pods/GreenKT07}} and computing expectation. Viewing \abbrPDB query evaluation as these two seperate steps is also known as intensional evaluation \cite{DBLP:series/synthesis/2011Suciu}, illustrated in \cref{fig:two-step}.
%The first step, which we will refer to as \termStepOne (\abbrStepOne), consists of computing both $\query\inparen{\db}$ and $\poly_\tup(\vct{X})$.\footnote{Assuming standard $\raPlus$ query processing algorithms, computing the lineage polynomial of $\tup$ is upperbounded by the runtime of deterministic query evaluation of $\tup$, as we show in \cref{sec:circuit-runtime}.} The second step is \termStepTwo (\abbrStepTwo), which consists of computing $\expct\pbox{\poly_\tup(\vct{\randWorld})}$. Such a model of computation is nicely followed in set-\abbrPDB semantics \cite{DBLP:series/synthesis/2011Suciu}, where $\poly_\tup\inparen{\vct{X}}$ must be computed separate from deterministic query evaluation to obtain exact output when $\query(\pdb)$ is hard since evaluating the probability inline with query operators (extensional evaluation) will only approximate the actual probability in such a case. The paradigm of \cref{fig:two-step} is also analogous to semiring provenance, where $\semNX$-DB\footnote{An $\semNX$-DB is a database whose tuples are annotated with elements from the set of polynomials with variables in $\vct{X}$ and natural number coeficients and exponents.} query processing \cite{DBLP:conf/pods/GreenKT07} first computes the query and polynomial, and the $\semNX$-polynomial can then subsequently evaluated over a semantically appropriate semiring, e.g. $\semN$ to model bag semantics. Further, in this work, the intensional model lends itself nicely in separating the concerns of deterministic computation and the probability computation.
%A natural question is whether or not we can quantify the complexity of computing $\expct\pbox{\poly_\tup\inparen{\vct{\randWorld}}}$ separately from the complexity of deterministic query evaluation, effectively dividing \abbrPDB query evaluation into two steps: deterministic query evaluation\footnote{Given input $\pdb$, this step includes outputting every tuple $\tup$ that satisfies $\query$, annotated with its lineage polynomial ($\poly_\tup$) which is computed inline across the query operators of $\query$.\cite{Imielinski1989IncompleteII}\cite{DBLP:conf/pods/GreenKT07}} and computing expectation. Viewing \abbrPDB query evaluation as these two seperate steps is also known as intensional evaluation \cite{DBLP:series/synthesis/2011Suciu}, illustrated in \Cref{fig:two-step}.
%The first step, which we will refer to as \termStepOne (\abbrStepOne), consists of computing both $\query\inparen{\db}$ and $\poly_\tup(\vct{X})$.\footnote{Assuming standard $\raPlus$ query processing algorithms, computing the lineage polynomial of $\tup$ is upperbounded by the runtime of deterministic query evaluation of $\tup$, as we show in \Cref{sec:circuit-runtime}.} The second step is \termStepTwo (\abbrStepTwo), which consists of computing $\expct\pbox{\poly_\tup(\vct{\randWorld})}$. Such a model of computation is nicely followed in set-\abbrPDB semantics \cite{DBLP:series/synthesis/2011Suciu}, where $\poly_\tup\inparen{\vct{X}}$ must be computed separate from deterministic query evaluation to obtain exact output when $\query(\pdb)$ is hard since evaluating the probability inline with query operators (extensional evaluation) will only approximate the actual probability in such a case. The paradigm of \Cref{fig:two-step} is also analogous to semiring provenance, where $\semNX$-DB\footnote{An $\semNX$-DB is a database whose tuples are annotated with elements from the set of polynomials with variables in $\vct{X}$ and natural number coeficients and exponents.} query processing \cite{DBLP:conf/pods/GreenKT07} first computes the query and polynomial, and the $\semNX$-polynomial can then subsequently evaluated over a semantically appropriate semiring, e.g. $\semN$ to model bag semantics. Further, in this work, the intensional model lends itself nicely in separating the concerns of deterministic computation and the probability computation.
Analogous to set-probabilistic databases, we focus on the intensional model of query evaluation, as illustrated in \cref{fig:two-step}.
Given input $\pdb$ and $\query$, the first step, which we will refer to as \termStepOne (\abbrStepOne), outputs every tuple $\tup$ that possibly satisfies $\query$, annotated with its lineage polynomial ($\poly$), which is computed inline
Analogous to set-probabilistic databases, we focus on the intensional model of query evaluation, as illustrated in \Cref{fig:two-step}.
Given input $\dbbase$ and $\query$, the first step, which we will refer to as \termStepOne (\abbrStepOne), outputs every tuple $\tup$ that possibly satisfies $\query$, annotated with its lineage polynomial ($\poly$), which is computed inline
\AH{While correct, I wonder if the average reviewer could confuse the language (based on the previous discussion of extensional evaluation) and think that the \emph{probability computation} is computed \emph{inline}.}
across the query operators of $\query$~\cite{Imielinski1989IncompleteII,DBLP:conf/pods/GreenKT07}.
We show in \cref{sec:circuit-runtime} that, assuming a standard $\raPlus$ query evaluation algorithm, the cost of constructing the lineage polynomial for tuples in a query result is upper-bounded by the runtime of generating those tuples through deterministic query evaluation.
We show in \Cref{sec:circuit-runtime} that, assuming a standard $\raPlus$ query evaluation algorithm, the cost of constructing the lineage polynomial for tuples in a query result is upper-bounded by the runtime of generating those tuples through deterministic query evaluation.
In other words, the first step is in \sharpwonehard,
\AH{\sharpwonehard is not defined.}
allowing us to focus on the complexity of the second step, \termStepTwo (\abbrStepTwo), which consists of computing $\expct\pbox{\poly(\vct{\randWorld})}$.
@ -143,10 +145,10 @@ There is significant precedent for the intensional model in \abbrBPDB{}s, as sev
Notably, intensional query evaluation also mirrors the approach of semiring provenance~\cite{DBLP:conf/pods/GreenKT07}, where $\semNX$-DB\footnote{An $\semNX$-DB is a database whose tuples are annotated with standard polynomials, i.e. elements from $\semNX$ connected by multiplication and addition operators.} query processing first constructs a $\semNX$-polynomial for each result tuple, which can then be evaluated over a semantically appropriate semiring (e.g. $\semN$ for bag semantics multiplicities).
Finally, the intensional model lends itself nicely to separating the concerns of deterministic query evaluation and probabilistic reasoning.
For bag-\abbrPDB $\pdb$ and query $Q$, let $\timeOf{\abbrStepOne}(Q,\pdb)$ denote the runtime of \abbrStepOne (Lineage Computation) and similarly for $\timeOf{\abbrStepTwo}(Q,\pdb)$ (Expectation Computation).
For bag-\abbrPDB $\pdb$ and query $Q$, let $\timeOf{\abbrStepOne}(\query,\dbbase)$ denote the runtime of \abbrStepOne (Lineage Computation) and similarly for $\timeOf{\abbrStepTwo}(Q,\dbbase)$ (Expectation Computation).
%Atri: Don't see what the sentence below is adding, so removing
%Given bag-\abbrPDB query $\query$ and \abbrTIDB $\pdb$ with $\numvar$ tuples, let us go a step further and assume that computing $\poly_\tup$ is lower bounded by the runtime of determistic query computation of $\query$ (e.g. when $\abs{\textnormal{input}} \leq \abs{\textnormal{output}}$).
When $\poly(\vct{X})$ is in standard monomial basis (\abbrSMB)\footnote{A polynomial is in \abbrSMB when it is a sum of products of variables (a variable can occur more than once), where each product of variables is unique.}, by linearity of expectation and independence of \abbrTIDB, it follows that $\timeOf{\abbrStepTwo}(Q,\pdb)$ is $O(|\poly_\tup(\vct{X})|)$ and thus also $\bigO{\timeOf{\abbrStepOne}(Q,\pdb)}$.
When $\poly(\vct{X})$ is in standard monomial basis (\abbrSMB)\footnote{A polynomial is in \abbrSMB when it is a sum of products of variables (a variable can occur more than once), where each product of variables is unique.}, by linearity of expectation and independence of \abbrTIDB, it follows that $\timeOf{\abbrStepTwo}(Q,\dbbase)$ is $O(|\poly_\tup(\vct{X})|)$ and thus also $\bigO{\timeOf{\abbrStepOne}(Q,\dbbase)}$.
\AH{Is this obvious enough for the typical reviewer to realize?}
Recall that $\prob_i$ denotes the probability of tuple $\tup_i$ (i.e. $\probOf\pbox{W_i = 1}$) for $i \in [\numvar]$. Consider another special case when for all $i$ in $[\numvar]$, $\prob_i = 1$.
% Replaced the stuff below with something more auccint
@ -155,13 +157,13 @@ When $\poly(\vct{X})$ is in standard monomial basis (\abbrSMB)\footnote{A polyno
%the size of the arithemetic circuit
%, since we can essentially push expectation through multiplication of variables dependent on one another.\footnote{For example in this special case, computing $\expct\pbox{(X_iX_j + X_\ell X_k)^2}$ does not require product expansion, since we have that $p_i^h x_i^h = p_i \cdot 1^{h-1}x_i^h$.}
In this case, we have for any output tuple $\tup$, $\expct\pbox{\poly(\vct{W})}=\Phi(1,\dots,1)$.
Thus, we have another case where $\timeOf{\abbrStepTwo}(Q,\pdb)$ is $\bigO{\timeOf{\abbrStepOne}(Q,\pdb)}$ and we again achieve deterministic query runtime for $\query\inparen{\pdb}$ (up to a constant factor). These observations introduce our first formalization of~\Cref{prob:informal}:
Thus, we have another case where $\timeOf{\abbrStepTwo}(\query,\dbbase)$ is $\bigO{\timeOf{\abbrStepOne}(\query,\dbbase)}$ and we again achieve deterministic query runtime for $\query\inparen{\pdb}$ (up to a constant factor). These observations introduce our first formalization of~\Cref{prob:informal}:
\begin{Problem}\label{prob:big-o-step-one}
Given bag-\abbrPDB $\pdb$, $\raPlus$ query $\query$ and output tuple $\tup$, is it \emph{always} the case that $\timeOf{\abbrStepTwo}(Q,\pdb)$ is $\bigO{\timeOf{\abbrStepOne}(Q,\pdb)}$?
Given bag-\abbrPDB $\pdb$, $\raPlus$ query $\query$ and output tuple $\tup$, is it \emph{always} the case that $\timeOf{\abbrStepTwo}(\query,\dbbase)$ is $\bigO{\timeOf{\abbrStepOne}(\query,\dbbase)}$?
\end{Problem}
If the answer to \cref{prob:big-o-step-one} is yes, then the query evaluation problem over bag \abbrPDB\xplural is of the same complexity as deterministic query evaluation, and probabilistic databases can offer performance competitive with deterministic databases.
If the answer to \Cref{prob:big-o-step-one} is yes, then the query evaluation problem over bag \abbrPDB\xplural is of the same complexity as deterministic query evaluation, and probabilistic databases can offer performance competitive with deterministic databases.
The main insight of the paper is that to answer~\Cref{prob:big-o-step-one}, the representation of $\poly(\vct{X})$ matters. One can have compact representations of $\poly(\vct{X})$ (e.g., resulting from optimizations like projection push-down~\cite{DBLP:books/daglib/0020812}, which produce factorized representations
%Atri: footnote below was not informative: used an example instead
@ -169,7 +171,7 @@ The main insight of the paper is that to answer~\Cref{prob:big-o-step-one}, the
of $\poly(\vct{X})$.
For example, in~\Cref{fig:two-step}, $B(Y+Z)$ is a factorized representation of the SMB-form $BY+BZ$. To capture such factorizations, this work uses (arithmetic) circuits\footnote{An arithmetic circuit has variable and/or numeric inputs, with internal nodes representing either an addition or multiplication operator.}
as the representation system of $\poly(\vct{X})$.
These are a natural fit to $\raPlus$ queries, as each operator maps to either a $\circplus$ or $\circmult$ operation \cite{DBLP:conf/pods/GreenKT07}. The standard query evaluation semantics depicted in \cref{fig:nxDBSemantics} illustrate this.
These are a natural fit to $\raPlus$ queries, as each operator maps to either a $\circplus$ or $\circmult$ operation \cite{DBLP:conf/pods/GreenKT07}. The standard query evaluation semantics depicted in \Cref{fig:nxDBSemantics} illustrate this.
\begin{figure}
@ -212,7 +214,7 @@ These are a natural fit to $\raPlus$ queries, as each operator maps to either a
\end{figure}
In other words, we can capture the size of a factorized lineage polynomial by the size of its correspoding arithmetic circuit $\circuit$ (which we denote by $|\circuit|$).
More importantly, our results in \cref{sec:circuit-runtime} show that, assuming a standard $\raPlus$ query evaluation algorithm for \abbrStepOne (\termStepOne), given the arithmetic circuit $\circuit$ corresponding to lineage polynomial output at the end of \abbrStepOne, we always have $|\circuit|\le \bigO{\timeOf{\abbrStepOne}(Q,\pdb)}$. Given this, we study the following stronger version of~\Cref{prob:big-o-step-one}:
More importantly, our results in \Cref{sec:circuit-runtime} show that, assuming a standard $\raPlus$ query evaluation algorithm for \abbrStepOne (\termStepOne), given the arithmetic circuit $\circuit$ corresponding to lineage polynomial output at the end of \abbrStepOne, we always have $|\circuit|\le \bigO{\timeOf{\abbrStepOne}(Q,\pdb)}$. Given this, we study the following stronger version of~\Cref{prob:big-o-step-one}:
\OK{This is still wrong. It should be phrased in terms of $\qruntime(Q, \dbbase)$... but I think it's going to require changes to \Cref{prob:big-o-step-one}}
%Atri: Replaced the text below by the above. I know I had talked about $|\circuit|^k$ but I think the stuff below breaks the flow a bit
@ -233,7 +235,7 @@ Note that an answer in the affirmative to the above question, implies an affirma
\mypar{Our Results} In this paper we tackle~\Cref{prob:big-o-step-one} to~\Cref{prob:intro-stmt}.
Concretely, we make the following contributions:
(i) %Under fine grained hardness assumption,
We show that the answer to~\Cref{prob:big-o-step-one} is \textit{no} in general for exact computation. %\cref{prob:intro-stmt} for bag-\abbrTIDB\xplural is not true in general
We show that the answer to~\Cref{prob:big-o-step-one} is \textit{no} in general for exact computation. %\Cref{prob:intro-stmt} for bag-\abbrTIDB\xplural is not true in general
% \sharpwonehard in the size of the lineage circuit
In fact, via a
reduction from counting the number of $k$-matchings over an arbitrary graph, we show that the problem of \abbrStepTwo (\termStepTwo) is \sharpwonehard. I.e., not only is the answer to~\Cref{prob:intro-stmt} no, but \abbrStepTwo cannot be solved in fully polynomial time, i.e. there is no algorithm for \abbrStepTwo with runtime that grows as $f(k)\cdot |\circuit|^d$, where $k$ is the degree of the corresponding lineage polynomial and $d$ is any fixed constant.\footnote{We would like to note that it is a well-known result in deterministic query computation that \abbrStepOne is also \sharpwonehard. What our result says is that \abbrStepTwo is \sharpwonehard\emph{even if} we exclude the complexity of \abbrStepOne .}
@ -247,13 +249,13 @@ We further note that in our hardness proofs, we have $|\circuit|=\Theta\inparen{
In contrast, known approximation techniques in set-\abbrPDB\xplural are at most quadratic in the size of the compressed lineage encoding.\AR{cite?}
%Atri: The footnote below does not add much
%\footnote{Note that this doesn't rule out queries for which approximation is linear});
(iii) We generalize the \abbrPDB data model considered by the approximation algorithm to a class of bag-Block Independent Disjoint Databases (see \cref{subsec:tidbs-and-bidbs}) (\abbrBIDB\xplural); (iv) We further prove that for \raPlus queries
(iii) We generalize the \abbrPDB data model considered by the approximation algorithm to a class of bag-Block Independent Disjoint Databases (see \Cref{subsec:tidbs-and-bidbs}) (\abbrBIDB\xplural); (iv) We further prove that for \raPlus queries
%\AH{This point \emph{\Large seems} weird to me. I thought we just said that the approximation complexity is linear in step one, but now it's as if we're saying that it's $\log{\text{step one}} + $ the runtime of step one. Where am I missing it?}
%\OK{Atri's (and most theoretician's) statements about complexity always need to be suffixed with ``to within a log factor''}
we can approximate the expected output tuple multiplicities (for all output tuples {\em simultanesouly} with only $O(\log{Z})$ overhead (where $Z$ is the number of output tuples) over the runtime of a broad class of query processing algorithms (see \Cref{app:sec-cicuits}). We also observe that our results trivially extend to higher moments of the tuple multiplicity (instead of just the expectation).
\mypar{Overview of our Techniques} All of our results rely on working with a {\em reduced} form of the lineage polynomial $\poly$. In fact, it turns out that for the TIDB (and BIDB) case, computing the expected multiplicity is {\em exactly} the same as evaluating this reduced polynomial over the probabilities that define the TIDB/BIDB. Next, we motivate this reduced polynomial.
Consider the query $\query(\pdb)$ defined as follows over the bag relations of \cref{fig:two-step}:
Consider the query $\query(\pdb)$ defined as follows over the bag relations of \Cref{fig:two-step}:
\begin{lstlisting}
SELECT 1 FROM OnTime a, Route r, OnTime b
WHERE a.city = r.city1 AND b.city = r.city2
@ -266,7 +268,9 @@ The lineage polynomial for $Q^2$ is given by $\poly^2\inparen{A, B, C, D, X, Y,
\inparen{AXB + BYD + BZC}^2\\
=A^2X^2B^2 + B^2Y^2D^2 + B^2Z^2C^2 + 2AXB^2YD + 2AXB^2ZC + 2B^2YDZC.
\end{multline*}
By exploiting linearity of expectation of summand terms, and further pushing expectation through independent \abbrTIDB variables, the expectation $\expct\limits_{\vct{\randWorld}\sim\pdassign}\pbox{\Phi^2\inparen{\vct{\randWorld}}}$ then is:\footnote{The random variable corresponding to a formal variable $A$ is denoted $\randWorld_A$, with probability drawn from $\pdassign$.}
By exploiting linearity of expectation of summand terms, and further pushing expectation through independent \abbrTIDB variables, the expectation
\AH{If we choose to use $\pd$ in \Cref{prob:bag-pdb-poly-expected}, then we either need to follow the same convention here OR introduce the notation $\pdassign$ before using it.}
$\expct\limits_{\vct{\randWorld}\sim\pdassign}\pbox{\Phi^2\inparen{\vct{\randWorld}}}$ then is:\footnote{The random variable corresponding to a formal variable $A$ is denoted $\randWorld_A$, with probability drawn from $\pdassign$.}
\begin{footnotesize}
\begin{multline*}
@ -294,7 +298,7 @@ With $\Phi^2\inparen{A, B, C, D, X, Y, Z}$ as an example, we have:
\end{align*}
Note that we have argued that for our specific example the expectation that we want to compute is $\widetilde{\Phi^2}(\probOf\pbox{A=1},$ $\probOf\pbox{B=1}, \probOf\pbox{C=1}), \probOf\pbox{D=1}, \probOf\pbox{X=1}, \probOf\pbox{Y=1}, \probOf\pbox{Z=1})$.
%It can be verified that the reduced polynomial parameterized with each variable's respective marginal probability is a closed form of the expected count (i.e., $\expct\limits_{\vct{\randWorld}\sim\pd}\pbox{\Phi^2\inparen{\vct{X}}} = \widetilde{\Phi^2}(\probOf\pbox{A=1},$ $\probOf\pbox{B=1}, \probOf\pbox{C=1}), \probOf\pbox{D=1}, \probOf\pbox{X=1}, \probOf\pbox{Y=1}, \probOf\pbox{Z=1})$).
In fact, the following lemma shows that this equivalence holds for {\em all} $\raPlus$ queries over TIDB (proof in \cref{subsec:proof-exp-poly-rpoly}).
In fact, the following lemma shows that this equivalence holds for {\em all} $\raPlus$ queries over TIDB (proof in \Cref{subsec:proof-exp-poly-rpoly}).
\begin{Lemma}\label{lem:tidb-reduce-poly}
Let $\pdb$ be a \abbrTIDB over $n$ input tuples
%\OK{Should this be $\vct{W}$?} $\vct{X} = \{X_1,\ldots,X_\numvar\}$
@ -307,7 +311,7 @@ For any \abbrTIDB-lineage polynomial $\poly\inparen{\vct{X}}$ based on $\query\i
$
\end{Lemma}
To prove our hardness result we show that for the same $Q$ considered in the example above, for an arbitrary product width $k$, the query $Q^k$ is able to encode various hard graph-counting problems\footnote{While $\query$ is the same, our results assume $\bigO{\numvar}$ tuples rather than the constant number of tuples appearing in \cref{fig:two-step}}. We do so by analyzing how the coefficients in the (univariate) polynomial $\widetilde{\Phi}\left(p,\dots,p\right)$ relate to counts of subgraphs in an arbitrary graph $G$ (which is used to define the $Route$ relation in $\query$) isomorphic to various graphs with $k$ edges.
To prove our hardness result we show that for the same $Q$ considered in the example above, for an arbitrary product width $k$, the query $Q^k$ is able to encode various hard graph-counting problems\footnote{While $\query$ is the same, our results assume $\bigO{\numvar}$ tuples rather than the constant number of tuples appearing in \Cref{fig:two-step}}. We do so by analyzing how the coefficients in the (univariate) polynomial $\widetilde{\Phi}\left(p,\dots,p\right)$ relate to counts of subgraphs in an arbitrary graph $G$ (which is used to define the $Route$ relation in $\query$) isomorphic to various graphs with $k$ edges.
For an upper bound on approximating the expected count, it is easy to check that if all the probabilties are constant then ${\Phi}\left(\prob_1,\dots, \prob_n\right)$ (i.e. evaluating the original lineage polynomial over the probability values) is a constant factor approximation. For example, using $\query^2$ from above, using $\prob_A$ to denote $\probOf\pbox{A = 1}$ (and similarly for the other six variables), we can see that
\begin{align*}

2
macros.tex
View File

@ -204,7 +204,7 @@
\newcommand{\poly}{\Phi}
\newcommand{\polyOf}[1]{\poly[#1]}
\newcommand{\polyqdt}[3]{\polyOf{#1,#2,#3}}
\newcommand{\apolyqdt}{\polyqdt{\query}{\pdb}{\tup}}
\newcommand{\apolyqdt}{\polyqdt{\query}{\dbbase}{\tup}}
\newcommand{\tupvar}[2]{X_{#1,#2}}
\newcommand{\atupvar}{\tupvar{\rel}{\tup}}
\newcommand{\polyX}{\poly\inparen{\vct{\pVar}}}%<---let's see if this proves handy

4
prob-def.tex
View File

@ -23,12 +23,10 @@ When the underlying DAG is a tree (with edges pointing towards the root), the st
%As stated in \Cref{def:circuit}, every internal node has at most two incoming edges, is labeled as an addition or a multiplication node, and has no limit on its outdegree.
%Note that if we limit the outdegree to one, then we get expression trees.
\begin{Example}
The circuits in \Cref{fig:two-step} encode their respective polynomials in column $\poly$.
%\circuit in \Cref{fig:circuit-express-tree} encodes the polynomial $XY + WZ$.
Note that each circuit \circuit encodes a tree, with edges pointing towards the root.
\end{Example}
%\begin{figure}[t]
% \begin{subfigure}[b]{0.45\linewidth}

4
two-step-model.tex
View File

@ -37,7 +37,7 @@
Chicago & Bremen & $Z$ & 1.0 \\
\end{tabular}};
%label below cylinder
\node[below=0.2 cm of cylinder]{{\LARGE$ \pdb$}};
\node[below=0.2 cm of cylinder]{{\LARGE$ \dbbase$}};
%First arrow
\node[single arrow, right=0.25 of cylinder, draw=black, fill=black!65, text=white, minimum height=0.75cm, minimum width=0.25cm](arrow1) {\textbf{\abbrStepOne}};
\node[above=of arrow1](arrow1Label) {$\query$};
@ -106,7 +106,7 @@
\end{tabular}
};
%label below rectangle
\node[below=0.2cm of rect]{{\LARGE $\query(\pdb)\inparen{\tup}\equiv \poly\inparen{\vct{X}}$}};
\node[below=0.2cm of rect]{{\LARGE $\query(\dbbase)\inparen{\tup}\equiv \poly\inparen{\vct{X}}$}};
%Second arrow
\node[single arrow, right=0.25 of rect, draw=black, fill=black!65, text=white, minimum height=0.75cm, minimum width=0.25cm](arrow2) {\textbf{\abbrStepTwo}};
%Expectation computation; (output of step 2)