WDSS Papers and Code

class: center, middle, inverse, title-slide

# WDSS Papers and Code
## Reading of Variational Learning on Aggregate Outputs with Gaussian Process (Law, Sejdinovic et al, 2018)
### Fernando A. Zepeda Herrera
### 15 February 2022

---

<div>
<style type="text/css">.xaringan-extra-logo {
width: 180px;
height: 120px;
z-index: 0;
background-image: url(img/logo_banner.jpeg);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('div')
          logo.classList = 'xaringan-extra-logo'
          logo.href = null
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

.center[

### Plan for Today

]

.pull-left[

<br>

1. Aggregated Outputs
  
1. Gaussian Processes

1. Variational Inference

1. Examples and Discussion

<br>

<div class="figure">
<img src="img/Paper_Fig_3.png" alt="Figure 3 from the paper" width="2309" />
<p class="caption">Figure 3 from the paper</p>
</div>
]

.pull-right[

]

---
class: center, middle

# Aggregated Outputs

Main problem addressed by Law, Sejdinovic, Cameron, Lucas, Flaxman, Battle, and Fukumizu (2018)

---

## Aggregated Outputs: Ex. Desease Prevalence

.pull-left[
<img src="img/aggr_ex_01.png" width="4197" />
]

.pull-right[
<img src="img/aggr_ex_02.png" width="4197" />
]

---

## Aggregated Outputs: Ex. Desease Prevalence

.pull-left[
<div class="figure">
<img src="img/Paper_Fig_3_b.png" alt="Figure 3(b)" width="80%" />
<p class="caption">Figure 3(b)</p>
</div>

Source: All three images from Law Sejdinovic et al. (2018).
]

.pull-right[
<img src="img/Paper_Fig_10.png" width="60%" />

<img src="img/Paper_Fig_11.png" width="60%" />
]

---

## Aggregated Outputs

.center[
<img src="img/aggr_problem.png" width="5432" />
]

---

## Aggregated Outputs

.center[
<img src="img/aggr_problem_01.png" width="60%" />
]

---

## Aggregated Outputs

.center[
<img src="img/aggr_problem_02.png" width="60%" />
]

---

## Aggregated Outputs

.center[
<img src="img/aggr_problem_03.png" width="60%" />
]

---

## Aggregated Outputs: Bag Observation Model

.panelset.sideways[
.panel[.panel-name[Data]

We have the following data

1. A **bag** of inputs `$x^a = \lbrace x_1^a, x_2^a, \dots, x_{N_a}^a\rbrace$` coming from `$N_a$` individuals

1. A total of `$n$` such bags

1. For each bag we have its **aggregated output** `$y^a$`

Hence, our dataset is:

`$$(\lbrace x_i^1 \rbrace_{i=1}^{N_1}, y^1)\;(\lbrace x_i^2 \rbrace_{i=1}^{N_2}, y^2)\;\cdots\;(\lbrace x_i^1 \rbrace_{i=1}^{N_n}, y^n)\;$$`

]

.panel[.panel-name[Model]
The aggregated output is related to the bag via de following **Bag Observation Model**

`$$y^a|x^a \sim p(y|\eta^a)$$` 
`$$\eta^a := \sum_{i=1}^{N_a} w_i^a\;\eta(x_i^a)$$`

where the `$w_i$` represent (possible) fixed weights. The inference goal is on the individual *mean parameters* `$\eta(x_i^a)$`.

]

.panel[.panel-name[Poisson Example]

- Suppose we have aggregated counts,

- But we know the population sizes at the individual level `$p_i^a$` (as well as other covariates `$x$`). Then, the bag population can be `$p^a=\sum p_i^a$`.

-We are interested in the individual rates `$\lambda_i^a$` of the following Poisson model:

`$$y^a|x^a \sim \text{Poisson}(p^a\lambda^a)$$` 
`$$\lambda^a := \sum_{i=1}^{N_a} \dfrac{p_i^a}{p^a}\;\lambda(x_i^a)$$`

]

.panel[.panel-name[But...]

#### Is it that simple?

- Non linear form of `$\eta(x_i^a)$`

- Usually *a lot* of individuals per bag

#### What to do?

]

---
class: inverse, center, middle

# VBAgg

## Variational Bag Aggregated Model with Gaussian Processes

Law Sejdinovic et al. (2018)

---

## VBAgg

.panelset.sideways[

.panel[.panel-name[Poisson Example]

- Suppose we have aggregated counts,

- But we know the population sizes at the individual level `$p_i^a$` (as well as other covariates `$x$`). Then, the bag population can be `$p^a=\sum p_i^a$`.

-We are interested in the individual rates `$\lambda_i^a$` of the following Poisson model:

`$$y^a|x^a \sim \text{Poisson}(p^a\lambda^a)$$` 
`$$\lambda^a := \sum_{i=1}^{N_a} \dfrac{p_i^a}{p^a}\;\lambda(x_i^a)$$`

]

.panel[.panel-name[GP]

Model the complicated and unobserved relationship as a **Gaussian Process**

`$$\lambda(x_i^a) = \Psi(f(x_i^a)) \qquad f\sim GP(\mu,k)$$`

where `$\Psi: \mathbb{R} \to \mathbb{R}^+$` is a suitable transformation like `$\Psi(f) = \exp(f)$` or   `$\Psi(f) = f^2$`.

]

.panel[.panel-name[Inducing Points]

Since GP are complicated and we have many individuals, they propose also to consider some **inducing points** `$\lbrace u_l\rbrace_{l=1}^m$`, which are the evaluations of the process at *landmark points* `$\lbrace z_l\rbrace_{l=1}^m$`, so that

`$$u_l := f(z_l),$$`

so that

`$$u \sim N_m(\mu_Z, K_{Z,Z})$$`

]

.panel[.panel-name[Likelihood]

We are then interested in the following (log) likelihood:

`$$\log p(Y) = \log \int\int p(Y, f, u) dfdu = \log \int\int \prod_{a=1}^n p(y^a|\eta^a)p(f | u) p(u)  df du$$`

which is quite complicated, so they propose using **Variational Inference** to approximate it.

]

---
class: center, middle

# Gaussian Processes Detour

[Betancourt (2020)](https://betanalpha.github.io/assets/case_studies/gaussian_processes.html)

---
class: center, middle

# Variational Inference Detour

[Vetrov (2019)](https://github.com/bayesgroup/deepbayes-2019/tree/master/lectures/day1)

---
class: center, middle

# Back to the Paper

[Law Sejdinovic et al. (2018)](https://proceedings.neurips.cc/paper/2018/hash/24b43fb034a10d78bec71274033b4096-Abstract.html)

---
class: center, middle

# As for the code part...

[Law (2018)](https://github.com/hcllaw/VBAgg)

---

# References

#### Main Paper and Repository
Law, H. C., D. Sejdinovic, E. Cameron, et al. (2018). "Variational
Learning on Aggregate Outputs with Gaussian Processes". In: _Advances
in Neural Information Processing Systems_. Ed. by S. Bengio, H.
Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett.
Vol. 31. Curran Associates, Inc. URL:
[https://proceedings.neurips.cc/paper/2018/hash/24b43fb034a10d78bec71274033b4096-Abstract.html](https://proceedings.neurips.cc/paper/2018/hash/24b43fb034a10d78bec71274033b4096-Abstract.html).

Law, H. C. (2018). _VBAgg. GitHub repository_. URL:
[https://github.com/hcllaw/VBAgg](https://github.com/hcllaw/VBAgg).

#### Detours
Betancourt, M. (2020). _Robust Gaussian Process Modeling_. URL:
[https://betanalpha.github.io/assets/case_studies/gaussian_processes.html](https://betanalpha.github.io/assets/case_studies/gaussian_processes.html).

Vetrov, D. (2019). _Variational Inference_. URL:
[https://github.com/bayesgroup/deepbayes-2019/tree/master/lectures/day1](https://github.com/bayesgroup/deepbayes-2019/tree/master/lectures/day1).