CMU 11-785 L23 Variational Autoencoders

EM for PCA

With complete information

• If we knew $z$ for each $x$, estimating $A$ and $D$ would be simple

$$x=Az+E$$

$$P(x\mid z)=N(Az,D)$$

• Given complete information $\left(x_1,z_1\right),\left(x_2,z_2\right)$

$$\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x,z)=\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x\mid z)$$

$$=\underset{A,D}{\mathrm{argmax}}\sum _{(x,Z)}\log \frac{1}{\sqrt{(2\pi {)}^d|D|}}\exp \left(-0.5(x-Az{)}^T{D}^{-1}(x-Az)\right)$$

• We can get a close form solution: $A=X{Z}^{+}$
• But we don’t have $Z$ => missing

With incomplete information

• Initialize the plane
• Complete the data by computing the appropriate $z$ for the plane
  • $P(z|X;A)$ is a delta, because $E$ is orthogonal to $A$
• Reestimate the plane using the $z$
• Iterate

Linear Gaussian Model

• PCA assumes the noise is always orthogonal to the data
  • Not always true
• The noise added to the output of the encoder can lie in any direction (uncorrelated)
• We want a generative model: to generate any point
  • Take a Gaussian step on the hyperplane
  • Add full-rank Gaussian uncorrelated noise that is independent of the position on the hyperplane
    • Uncorrelated: diagonal covariance matrix
    • Direction of noise is unconstrained

With complete information

$$x=Az+e$$

$$P(x\mid z)=N(Az,D)$$

• Given complete information $X=\left[x_1,x_2,\dots \right],Z=\left[z_1,z_2,\dots \right]$

$$\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x,z)=\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x\mid z)$$

$$=\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log \frac{1}{\sqrt{(2\pi {)}^d|D|}}\exp \left(-0.5(x-Az{)}^T{D}^{-1}(x-Az)\right)$$

$$=\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}-\frac{1}{2}\log |D|-0.5(x-Az{)}^T{D}^{-1}(x-Az)$$

• We can also get closed form solution

With incomplete information

Option 1

• In every possible way proportional to $P(z|x)$ (Gaussian)
• Compute the solution from the completed data

$$\underset{A,D}{\mathrm{argmax}}\sum _x{\int }_{-\infty }^{\infty }p(z\mid x)\left(-\frac{1}{2}\log |D|-0.5(x-Az{)}^T{D}^{-1}(x-Az)\right)dz$$

• The same as before

Option 2

• By drawing samples from $P(z|x)$
• Compute the solution from the completed data

The intuition behind Linear Gaussian Model

• $z\sim N(0,I)$ => $Az$
  • The linear transform stretches and rotates the K-dimensional input space onto a Kdimensional hyperplane in the data space
• $X=Az+E$
  • Add Gaussian noise to produce points that aren’t necessarily on the plane

• The posterior probability $P(z|x)$ gives you the location of all the points on the plane that could have generated $x$ and their probabilities

• What about data that are not Gaussian distributed close to a plane?

  • Linear Gaussian Models fail

• How to do that

Non-linear Gaussian Model

• $f(z)$ is a non-linear function that produces a curved manifold
  • Like the decoder of a non-linear AE
• Generating process
  • Draw a sample $z$ from a Uniform Gaussian
  • Transform $z$ by $f(z)$
    • This places $z$ on the curved manifold
  • Add uncorrelated Gaussian noise to get the final observation

• Key requirement
  • Identifying the dimensionality $K$ of the curved manifold
  • Having a function that can transform the (linear) $K$-dimensional input space (space of $z$ ) to the desired $K$-dimensional manifold in the data space

With complete data

$$x=f(z;\theta )+e$$

$$P(x\mid z)=N(f(z;\theta ),D)$$

• Given complete information $X=\left[x_1,x_2,\dots \right],Z=\left[z_1,z_2,\dots \right]$

$${\theta }^{\star },D^{\star }=\underset{\theta ,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x,z)=\underset{\theta ,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x\mid z)$$

$$=\underset{\theta ,D}{\mathrm{argmax}}\sum _{(x,Z)}\log \frac{1}{\sqrt{(2\pi {)}^d|D|}}\exp \left(-0.5(x-f(z;\theta ){)}^T{D}^{-1}(x-f(z;\theta ))\right)$$

$$=\underset{\theta ,D}{\mathrm{argmax}}\sum _{(x,Z)}-\frac{1}{2}\log |D|-0.5(x-f(z;\theta ){)}^T{D}^{-1}(x-f(z;\theta ))$$

• There isn’t a nice closed form solution, but we could learn the parameters using backpropagation

Incomplete data

• The posterior probability is given by

$$P(z\mid x)=\frac{P(x\mid z)P(z)}{P(x)}$$

• The denominator

$$P(x)={\int }_{-\infty }^{\infty }N(x;f(z;\theta ),D)N(z;0,D)dz$$

• Can not have a closed form solution
  • Try to approximate it

• We approximate $P(z|x)$ as

$$P(z\mid x)\approx Q(z,x)=\mathrm{Gaussian}N(z;\mu (x),\Sigma (x))$$

• Sample $z$ from $N(z;\mu (x;\varphi ),\sigma (x;\varphi ))$ for each training instance
  • Draw $K$-dimensional vector $\epsilon$ from $N(0,I)$
  • Compute $z=\mu (x;\phi )+\Sigma (x;\phi {)}^{0.5}\epsilon$
• Reestimate $\theta$ from the entire “complete” data
  • Using backpropagation

$$L(\theta ,D)=\sum _{(x,z)}\log |D|+(x-f(z;\theta ){)}^T{D}^{-1}(x-f(z;\theta ))$$

$${\theta }^{\star },D^{\star }=\underset{\theta ,D}{\mathrm{argmin}}L(\theta ,D)$$

• Estimate $\phi$ using the entire “complete” data
  • Recall $Q(z,x)=N(z;\mu (x;\phi ),\Sigma (x;\phi ))$ must approximate $P(z|x)$ as closely as possible
  • Define a divergence between $Q(z,x)$ and $P(z|x)$

Variational AutoEncoder

• Non-linear extensions of linear Gaussian models
• $f(z;\theta )$ is generally modelled by a neural network
• $\mu (x;\phi )$ and $\Sigma (x;\phi )$ are generally modelled by a common network with two outputs

• However, VAE can not be used to compute the likelihoood of data
  • $P(x;\theta )$ is intractable
• Latent space
  • The latent space $z$ often captures underlying structure in the data $x$ in a smooth manner
  • Varying $z$ continuously in different directions can result in plausible variations in the drawn output