[024] Latest generative models

• Relativistic (average) GAN/RGAN/RaGAN: The relativistic discriminator: a key element missing from standard GAN
• Glow: Better Reversible Generative Models

According to this paper, previous framework of GAN has two drawbacks: (1) the discriminator cannot utilize the fact that half of the data being examined is fake; (2) the generator can only make the fake ones look like the real ones, but it cannot make the real ones look faker than the fake one. The second drawback acts like a limitation to the optimal solution of the generator: “it can at best generate as realistic as the given one”. But, if possible, we want the generator to output more confusing sample that can surpass the real one (given a same discriminator).

Previously, the discriminator is limited to output a single probability for a single sample, like , where  is usually an activation function with range from 0 to 1 and  is a score for the sample . In this paper, the authors switch it with , which means the discriminator compares two samples to make a judgement. 

With this new design, (1) the discriminator can really “compare” two samples simultaneously rather than assigning probability independently; (2) the generator can make the score of the fake data  higher than the one of the real data  rather than making the probability  approaching .

To avoid the randomness when sampling pairs, the authors purpose an averaged version: the original RGAN loss   is replaced by  , where  is an average score of the real fake data. Please refer to the original paper for more details.

Based on their description, RGAN or RaGAN (when mixed with other techniques) can (1) achieve higher stability, (2) use less update iterations of the discriminator, and (3) generate better quality using limited samples.

TL;DR: Make the single-sample judgment  pairwise  ,which works better with an average version  .