+
+ Count-Distinct
+
+
+ $3$
+ $5$
+ $4$
+ $4$
+ $2$
+ $4$
+ $3$
+ $\ldots$
+
+
+
+
+ $3$
+ $5$
+ $4$
+ $2$
+ $\ldots$
+
+
+
+ Challenge: To avoid double counting, we need to track which values of $A$ we've seen. $O(|A|)$ memory required.
+
+
+
+
+
+ The Coin Flip Game
+
+ Start with 0 points and flip a coin
+
+
+
- Tails (🐕)
+ - Get a point and flip again.
+
+
+
- Heads (👽)
+ - Game over.
+
+
+
+
+
+
+
+
+
+ | Flips | Score |
|---|
+ |
+ (👽)
+ | 0 |
+ |
+ (🐕)
+ (👽)
+ | 1 |
+ |
+ (🐕)
+ (🐕)
+ (🐕)
+ (🐕)
+ (🐕)
+ (👽)
+ | 5 |
+
+
+
+
+
+ | Flips | Score | Probability |
+ E[# Games] |
+
|---|
+ | (👽) | 0 | 0.5 |
+ 2 |
+
+ | (🐕)(👽) | 1 | 0.25 |
+ 4 |
+
+ | (🐕)(🐕)(👽) | 2 | 0.125 |
+ 8 |
+
+ | (🐕)$\times N$ (👽) | $N$ | $\frac{1}{2^{N+1}}$ |
+ $2^{N+1}$ |
+
+
+ If I told you that in a series of games, my best score was $N$, you might expect that I played $2^{N+1}$ games.
+ To do that, I only need to track my top score!
+
+
+
+ Idea: Simulate coin flips with a hash function
+
+ ... take the index of the lowest-order nonzero bit
+
+
+
+
+ | Object | Hash Bits | Score |
|---|
+ | $O_1$ | 01011011 | 0 |
+ | $O_2$ | 00110111 | 0 |
+ | $O_3$ | 00111000 | 3 |
+ | $O_4$ | 10010010 | 1 |
+ | $O_3$ | 00111000 | 3 |
+ | | 3 |
+
+
+ Estimate: $2^{3+1} = 16$
+ Duplicates can't raise the top score!
+
+
+
+ Problem: Noisy estimate!
+ Idea 1: Instead of your top score, track the lowest score you have not gotten yet ($R$).
+
+
+
+
+ | Object | Hash Bits | Score |
|---|
+ | $O_1$ | 01011011 | 0 |
+ | $O_2$ | 00110111 | 0 |
+ | $O_3$ | 00111000 | 3 |
+ | $O_4$ | 10010010 | 1 |
+ | $O_3$ | 00111000 | 3 |
+ | | {0, 1, 3} $R = 2$ |
+
+
+ Estimate: $\frac{2^R}{\phi} = \frac{2^{2}}{0.77351} \approx 5.2$
+
+
+
+ Idea 2: Compute several estimates in parallel and average estimates.
+
+
+
+ Flajolet-Martin Sketches
+ ($\approx$ HyperLogLog)
+
+
+ - For each record...
+
+ - Hash each record
+ - Find the index of the lowest-order non-zero bit
+ - Add the index of the bit to a set
+
+ - Find $R$, the lowest index not in the set
+ - Estimate Count-Distinct as $\frac{2^R}{\phi}$ ($\phi \approx 0.77351$)
+ - Repeat (in parallel) as needed
+
+
+
+
+
+
+ Group-By Count
+ Problem: Need a counter for each individual A
+
+
+
+ Idea: Keep only one counter!
+
+
+
+ 
+ No... seriously
+
+
+
+ $$\delta(O_i) = \begin{cases} \textbf{if } h(O_i) = 0 \mod 2 & \textbf{then } -1 \\ \textbf{if } h(O_i) = 1 \mod 2 & \textbf{then } +1\end{cases}$$
+
+
+
+ $$\sum_i \delta(O_i)$$
+
+
+
+
+ | Object | $\delta(O_i)$ | Running Count |
|---|
+ | $O_3$ | -1 | -1 |
+ | $O_1$ | +1 | 0 |
+ | $O_4$ | -1 | -1 |
+ | $O_2$ | +1 | 0 |
+ | $O_4$ | -1 | -1 |
+ | $O_1$ | +1 | 0 |
+ | $O_3$ | -1 | -1 |
+ | $O_3$ | -1 | -2 |
+ | $O_1$ | +1 | -1 |
+
+
+
+
+
+
+ | $Total =$ |
+ | $\texttt{COUNT_OF}(O_i) \cdot \delta(O_i)$ |
+ | $+ \sum_{j \neq i}\texttt{COUNT_OF}(O_j) \cdot \delta(O_j)$ |
+
+
+
+ | $E[\sum_{j}\texttt{COUNT_OF}(O_j) \cdot \delta(O_j)]$= |
+ | $\frac{1}{2}\sum \texttt{COUNT_OF}(O_j)$ |
+ | $ - \frac{1}{2}\sum \texttt{COUNT_OF}(O_j)$ |
+
+
+
+ $$Total \approx \texttt{COUNT_OF}(O_i) \cdot \delta(O_i) + 0$$
+
+
+
+
+ Running total was $-1$
+
+ | Object | $\delta(O_i)$ | Estimate |
|---|
+ | $O_1$ | +1 | -1 |
+ | $O_2$ | +1 | -1 |
+ | $O_3$ | -1 | +1 |
+ | $O_4$ | -1 | +1 |
+
+
+ Not... so... great
+
+
+
+ Problem 1: All of the objects use the same counter (no way to differentiate an estimate for $O_1$ from $O_2$).
+ Problem 2: The estimate is really noisy
+
+
+
+ Idea 1: Multiple Buckets ($h(x)$ picks a bucket)
+ Idea 2: Multiple Trials ($h \rightarrow h_1, h_2, \ldots$; $\delta \rightarrow \delta_1, \delta_2, \ldots$)
+
+
+<%
+ prng = Random.new(2019)
+ num_trials = 2
+ num_buckets = 2
+ num_objects = 4
+ all_fns = (0...num_objects).map {
+ (0...num_trials).map {
+ [ prng.rand(2)*2-1,
+ prng.rand(num_buckets)
+ ] } }
+%>
+
+
+ | Object |
+ <% (1..num_trials).each do |i| %>
+ $h_<%=i%>(O_i)$ |
+ $\delta_<%=i%>(O_i)$ |
+ <% end %>
|---|
+ <% all_fns.each.with_index do |o_fns, i| %>
+ | $O_<%=i+1%>$ |
+ <% o_fns.each do |d, h| %>
+ Bucket <%=h+1%> |
+ <%=d%> |
+ <% end %>
+ <% end %>
+
+
+
+ <% m = (0...num_trials).map { [0]*num_buckets } %>
+ <% log = [] %>
+ <% [ 2, 1, 4, 1, 2, 1 ].each do |i| %>
+ <% o_fns = all_fns[i-1]; %>
+
+ Objects Seen: $<%= log.map { |l| "O_#{l}" }.join(",") %>$
+
+
+ | Bucket 1 | Bucket 2 |
|---|
+ <% m.each.with_index do |buckets, trial| %>
+ | Trial <%=trial%> |
+ <% buckets.each do |cnt| %><%= cnt %> | <% end %>
+
+ <% end %>
+
+
+
+ | Object |
+ <% (0...num_trials).each do |i| %>Trial <%=i+1%> | <% end %>
+ Estimate | Real |
|---|
+ <% (0...num_objects).each do |o| %>
+ | $O_<%=o+1%>$ |
+ <% est = 0; (0...num_trials).each do |i| %>
+ <% delta, bucket = all_fns[o][i]; est += m[i][bucket] * delta %>
+ <%= m[i][bucket] * delta %> |
+ <% end %>
+ <%= est.to_f/num_trials %> |
+ <%= log.select { |x| x == o+1 }.count %> |
+
+ <% end %>
+
+ <%
+ log.push(i)
+ o_fns.each.with_index do |d_h, trial|
+ d, h = d_h
+ m[trial][h] += d
+ end
+ %>
+
+ <% end %>
+
+
+ In practice, use Median and not Mode to combine trials
+
+
+
+
+