Image GPT

GPT Transformers Generative Models Footnotes A Measured through logistic regression on learned features (linear probe). B A transformer is trained to maximize the likelihood, and thus is mode covering, which automatically ensures the diversity of its samples. C The original analysis by synthesis idea is more an argument for generative models with latent variables, but because generative models without latent variables were so much better at modeling the data distribution, we thought the analysis-by-synthesis conjecture should hold for them as well. D We only show linear probe accuracy on ImageNet for iGPT-XL since other experiments did not finish before we needed to transition to different supercomputing facilities. E To extract features for a linear probe, we take the post layernorm attention block inputs at some layer and average pool over the sequence dimension. F To fine-tune, we take the post layernorm transformer output and average pool over the sequence dimension as input for the classification head. G A generative model which learns features in a purely unsupervised fashion. References 1 2 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., & Polosukhin, I. “ Attention is All you Need⁠(opens in a new window) .” In NeurIPS 2017. 3 4 Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). “ Language Models are Unsupervised Multitask Learners⁠(opens in a new window) .” Technical Report, OpenAI. 5 Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). “ RoBERTa: A Robustly Optimized BERT Pretraining Approach⁠(opens in a new window) .” arXiv preprint. 6 Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. (2019). “ Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer⁠(opens in a new window) .” arXiv preprint. 7 8 Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). “ Deep Contextualized Word Representations⁠(opens in a new window) .” In NAACL 2018. 9 10 11 Ke N., Goyal, A., Bilaniuk,O., Binas, J., Mozer, M., Pal, C., Bengio, Y (2018). “ Sparse attentive backtracking: Temporal credit assignment through reminding⁠(opens in a new window) .” In NeurIPS 2018. 12 13 14 Kolesnikov, A. & Beyer, L. & Zhai, X., Puigcerver, J., Yung, J., Gelly, S., Houlsby, N. (2019). “ Big Transfer (BiT): General Visual Representation Learning⁠(opens in a new window) .” arXiv preprint. 15 Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., & Chen, Z. (2019) “ GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism⁠(opens in a new window) .” In NeurIPS 2019. 16 17 18 Erhan, D., Bengio, Y., Courville, A., Manzagol, P., Vincent, P., Bengio, S. (2010). “ Why does unsupervised pre-training help deep learning?⁠(opens in a new window) .” In JMLR 2010. 19 20 21 22 23 24 25 Henaff, O., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S., Oord, A. (2019). “ Data-Efficient Image Recognition with Contrastive Predictive Coding⁠(opens in a new window) .” arXiv preprint. 26 27 Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., & Tran, D. (2018). “ Image transformer⁠(opens in a new window) .” In ICML 2018. 28 29 30 31 32 33 34 35 36 Le, Q. V., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J. & Ng, A. Y. (2012). “ Building high-level features using large scale unsupervised learning⁠(opens in a new window) .” In ICML 2012. 37 38 39 40 41 42 43 44 45 Hjelm, R., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018). “ Learning deep representations by mutual information estimation and maximization⁠(opens in a new window) .” In ICLR 2019. 46 47 Rives, A., Goyal, S., Meier, J., Guo, D., Ott, M., Zitnick, C., Ma, J., Fergus, R. (2019). “ Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences⁠(opens in a new window) .” bioRxiv preprint. 48 49 50 51 52 53 54 55 Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X. (2016). “ Improved techniques for training gans⁠(opens in a new window) .” In NeurIPS 2016. 56 57 Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C. (2019). “ MixMatch: A Holistic Approach to Semi-Supervised Learning⁠(opens in a new window) .” In NeurIPS 2019. 58 59 Sohn, K., Berthelot, D., Li, C., Zhang, Z., Carlini, N., Cubuk, E., Kurakin, A., Zhang, H., Raffel, C. (2020). “ Fixmatch: Simplifying semi-supervised learning with consistency and confidence⁠(opens in a new window) .” arXiv preprint. 60 Authors Mark Chen, Alec Radford, Ilya Sutskever Acknowledgments Foremost, we would like to acknowledge our paper co-authors Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, and David Luan. Thanks to the following for their feedback on this work and contributions to this release: Vedant Misra, Noah Golmant, Johannes Otterbach, Pranav Shyam, Aditya Ramesh, Yura Burda, Harri Edwards, Chris Hallacy, Jeff Clune, Jack Clark, Irene Solaiman, Ryan Lowe, Greg Brockman, Kelly Sims, David Farhi, Will Guss, Quoc V. Le, and Ashish Vaswani. Editor: Ashley Pilipiszyn Design: Justin Jay Wang Cover artwork: Ben Barry