Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. Kim, Y., Wiseman, S., Miller, A.C., Sontag, D., Rush, A.M.: Semi-amortized variational autoencoders. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-softmax. Hoffman, M.D., Blei, D.M., Bach, F.R.: Online learning for latent Dirichlet allocation. Griffiths, T.L., Steyvers, M.: Finding scientific topics. In: Advances in Neural Information Processing Systems, vol. Goodfellow, I.J., et al.: Generative adversarial nets. 31–40 (2009)ĭieng, A.B., Wang, C., Gao, J., Paisley, J.W.: TopicRNN: a recurrent neural network with long-range semantic dependency. In: Proceedings of the Biennial GSCL Conference From Form to Meaning: Processing Texts Automatically, vol. 3, 993–1022 (2003)īouma, G.: Normalized (Pointwise) Mutual information in collocation extraction.

Keywordsīlei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. The experimental results showed that our VAE improved the existing VAE proposals for a half of the data sets in terms of perplexity or of normalized pairwise mutual information (NPMI). Therefore, we feed to our encoder network a concatenation of token-wise information and document-wide information, where the former is implemented as word embedding and the latter as the document-wide mean of word embeddings. Third, while we can sample token-wise topic assignments with the Gumbel-softmax trick, we should consider document-wide contextual information for a better approximation. Therefore, we adopt the Gumbel-softmax trick. However, this posterior is categorical and thus cannot be approximated by continuous distributions like Gaussian. Second, after marginalizing out topic probabilities, we need to approximate the posterior probabilities of token-wise topic assignments. This marginalizing out has not been considered in the existing VAE proposals for topic modeling to the best of our knowledge. First, we marginalize out per-document topic probabilities by following the proposal by Mimno et al. We can implement VI as VAE by using a neural network called encoder to obtain parameters of approximate posterior. The variational inference (VI) for Bayesian models approximates the true posterior distribution by maximizing a lower bound of the log marginal likelihood of observations. This paper proposes a new variational autoencoder (VAE) for topic models.

Wise token