Publication

You can also browse my Google Scholar profile. * denotes equal contribution; + denotes corresponding author.


  1. Masked Diffusion Transformer is a Strong Image Synthesizer
    Shanghua Gao, Pan Zhou+, Ming-Ming Cheng, Shuicheng Yan
    ICCV, 2023, [PDF] [Code]
    SoTA image generative model on ImageNet 256x256; 13x faster learning speed than DiT (core of SORA)

  2. EditAnything: Empowering Unparalleled Flexibility in Image Editing and Generation
    Shanghua Gao, Zhijie Lin, Xingyu Xie, Pan Zhou+, Ming-Ming Cheng, Shuicheng Yan
    ACMMM, 2023, [PDF] [Code]
    the first a few pioneers for highly-flexible image editing, e.g., cross-image dragging like try-on, region-interactive editing, controllable layout generation, and virtual character replacement.

  3. Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior
    Zike Wu, Pan Zhou+, Xuanyu YI, Xiaoding Yuan, Hanwang Zhang
    CVPR, 2024, [Axriv] [Code]
    the first ODE-sampling guided Score Distillation Sampling for 3D generation

  4. Prototypical Contrastive Learning of Unsupervised Representations
    Junnan Li, Pan Zhou, Caiming Xiong, Steven Hoi
    ICLR, 2021, [Axriv] [Bibtex] [Blog] [Code], 900+ citations,
    the first clustering contrastive learning method to learn high-level semantics, i.e., data cluster structure

  5. MetaFormer Baselines for Vision
    Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang
    TPAMI & CVPR, 2023, [Axriv] [Code], 600+ citations,
    replacing attention with simple pooling still achieves high performance, breaking "attention is all you need" and revealing network design principle

  6. Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
    Xingyu Xie*, Pan Zhou*, Huan Li, Zhouchen Lin, Shuicheng Yan
    [PDF] [Code]
    2X-faster and SoTA optimizer on 15+ networks like ResNet, ConvNext, ViT, Swin, MAE, BERT, GPT2, LLAMA, Dreamfusion, DiT, PPO in RL, etc. Included by popular deep-learning codebases like NVIDIA NeMo for LLM, HuggingFace Timm and OpenMMLab for CV tasks, Jittor of Tsinghua University for 3D.

  7. Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training
    Pan Zhou, Xingyu Xie, Zhouchen Lin, Kim-Chuan Toh, Shuicheng Yan
    JMLR & ICLR, 2024 [Code]
    accelerate AdamW/Adam/LAMB/SGD by 1.5x on vision and language modeling tasks.

  8. Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
    Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Hoi, and Weinan E
    NeurIPS, 2020, [PDF] [SUPP] [Axriv] [Bibtex] [Code] [Slides] [Poster], 200+ citations
    the first theory to explain "why SGD generalizes better than ADAM in deep learning"


Full Publications

    2024

  1. Instant3D: Instant Text-to-3D Generation
    Ming Li, Pan Zhou, Jia-Wei Liu, Jussi Keppo, Min Lin, Shuicheng Yan, Xiangyu Xu
    International Journal of Computer Vision (IJCV), 2024
    [Axriv] [Code]

  2. Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training
    Pan Zhou, Xingyu Xie, Zhouchen Lin, Kim-Chuan Toh, Shuicheng Yan
    Journal of Machine Learning Research (JMLR), 2024
    [Code]

  3. Enhancing Visual Grounding in Vision-Language Pre-Training with Position-Guided Text Prompts
    Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, Shuicheng Yan
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024
    [PDF] [Code] ,

  4. Towards Understanding Convergence and Generalization of AdamW
    Pan Zhou, Xingyu Xie, Zhoucheng Lin, Shuicheng Yan
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024
    [PDF] [Supp]

  5. Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Multimodal Humor Generation
    Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, Pan Zhou+
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

  6. InceptionNeXt: When Inception Meets ConvNeXt
    Weihao Yu, Pan Zhou, Shuicheng YAN, Xinchao Wang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

  7. Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior
    Zike Wu, Pan Zhou+, Xuanyu YI, Xiaoding Yuan, Hanwang Zhang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

  8. Friendly Sharpness-Aware Minimization
    Tao Li, Pan Zhou+, Zhengbao He, Xinwen Cheng, Xiaolin Huang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

  9. Few-shot Learner Parameterization by Diffusion Time-steps
    Zhongqi Yue, Pan Zhou+, Richang Hong, Hanwang Zhang, Qianru Sun
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

  10. Diffusion Time-step Curriculum for One Image to 3D Generation
    Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou+, Joo Hwee Lim, Hanwang Zhang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
    [Axriv] [Code]

    11. 2023

  11. MetaFormer Baselines for Vision
    Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023
    [Axriv] [Code] ,

  12. ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection
    Zhongzhan Huang, Pan Zhou+, Shuicheng Yan, Liang Lin
    Neural Information Processing Systems (NeurIPS), 2023
    [Axriv] [Code]

  13. Masked Diffusion Transformer is a Strong Image Synthesizer
    Shanghua Gao, Pan Zhou+, Ming-Ming Cheng, Shuicheng Yan
    International Conference on Computer Vision (ICCV), 2023
    [PDF] [Code]

  14. EditAnything: Empowering Unparalleled Flexibility in Image Editing and Generation
    Shanghua Gao, Zhijie Lin, Xingyu Xie, Pan Zhou+, Ming-Ming Cheng, Shuicheng Yan
    ACM International Conference on Multimedia (ACMMM), 2023
    [PDF] [Code]

  15. STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition
    Ming Li, Xiangyu Xu, Hehe Fan, Pan Zhou, Jun Liu, Jia-Wei Liu, Jiahe Li, Jussi Keppo, Mike Zheng Shou, Shuicheng Yan
    International Conference on Computer Vision (ICCV), 2023
    [PDF]

  16. Contrastive Video Question Answering via Video Graph Transformer
    Junbin Xiao, Pan Zhou, Angela Yao, Yicong Li, Richang Hong, Shuicheng Yan, Tat-Seng Chua
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023
    [PDF] [Code]

  17. Position-guided Text Prompt for Vision-Language Pre-training
    Alex Jinpeng Wang, Pan Zhou+, Mike Zheng Shou, Shuicheng Yan
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023
    [Axriv] [Code] ,

  18. Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive Gradient Algorithms
    Pan Zhou, Xingyu Xie, Shuicheng Yan
    International Conference on Learning Representations (ICLR), 2023 (oral)
    [Axriv] [Code]

  19. Towards Understanding Why Mask Reconstruction Pretraining Helps in Downstream Tasks
    Jiachun Pan*, Pan Zhou*, Shuicheng Yan
    International Conference on Learning Representations (ICLR), 2023
    [Axriv]

  20. LPT: Long-tailed Prompt Tuning for Image Classification
    Bowen Dong, Pan Zhou, Shuicheng Yan, Wangmeng Zuo
    International Conference on Learning Representations (ICLR), 2023
    [Axriv] [Code] ,

  21. Iterative Graph Self-Distillation
    Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang, Xiaodan Liang, Eric P. Xing
    IEEE Transactions on Knowledge and Data Engineering (TKDE), 2023
    [Axriv]

    22. 2022

  22. Inception Transformer
    Chenyang Si*, Weihao Yu*, Pan Zhou, Yichen Zhou, Xinchao Wang, Shuicheng Yan
    Neural Information Processing Systems (NeurIPS), 2022 (oral)
    [Axriv] [Code]

  23. Mugs: A Multi-Granular Self-Supervised Learning Framework
    Pan Zhou*, Yichen Zhou*, Chenyang Si*, Weihao Yu, Teck Khim Ng, Shuicheng Yan
    Workshop of Neural Information Processing Systems, 2022.
    [Axriv] [Code]
    Top linear probing and KNN performance on ImageNet without extra data

  24. DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition
    Yuxuan Liang, Pan Zhou, Roger Zimmermann, Shuicheng Yan
    European Conference on Computer Vision (ECCV), 2022
    [Axriv] [Code]

  25. Video Graph Transformer for Video Question Answering
    Junbin Xiao, Pan Zhou, Tat-Seng Chua, Shuicheng Yan
    European Conference on Computer Vision (ECCV), 2022
    [Axriv] [Code]

  26. Self-Promoted Supervision for Few-Shot Transformer
    Bowen Dong, Pan Zhou, Shuicheng Yan, Wangmeng Zuo
    European Conference on Computer Vision (ECCV), 2022
    [Axriv] [Code]

  27. MetaFormer is Actually What You Need for Vision
    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022 (oral)
    [Axriv] [Code]

  28. Prototypical Graph Contrastive Learning
    Lin Shuai, Liu Chen, Pan Zhou, Hu Zi-yuan, Wang Shuojia, Zhao Ruihui, Zheng Yefeng, Lin Liang, Xing Eric, Liang Xiaodan
    IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022
    [Axriv] [Code]

    29. 2021

  29. A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
    Pan Zhou, Caiming Xiong, Xiaotong Yuan, Steven Hoi
    Neural Information Processing Systems (NeurIPS), 2021 (spotlight)
    [PDF] [SUPP] [Axriv] [Bibtex] [Code] [Slides] [Poster]

  30. Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond
    Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, Shuicheng Yan
    Neural Information Processing Systems (NeurIPS), 2021
    [PDF] [SUPP] [Bibtex] [Code] [Slides] [Poster]

  31. A Hybrid Stochastic-Deterministic Minibatch Proximal Gradient Method for Efficient Optimization and Generalization
    Pan Zhou, XiaoTong Yuan, Zhouchen Lin, and Steven Hoi
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021
    [PDF] [SUPP] [Bibtex]

  32. Prototypical Graph Contrastive Learning
    Task Similarity Aware Meta Learning: Theory-inspired Improvement on MAML
    Pan Zhou, Yingtian Zou, XiaoTong Yuan, Jiashi Feng, Caiming Xiong, and Steven Hoi
    International Conference on Uncertainty in Artificial Intelligence (UAI), 2021 (NeurIPS'20 Meta Learning Workshop Paper)
    [PDF] [SUPP] [Code]

  33. Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition
    Guolin Zheng, Yubei Xiao, Ke Gong, Pan Zhou, Xiaodan Liang, and Liang Lin
    Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021 (Findings)
    [Axriv]

  34. How Important is the Train-Validation Split in Meta-Learning?
    Yu Bai, Minshuo Chen, Pan Zhou, Tuo Zhao, Jason D. Lee, Sham Kakade, Huan Wang, Caiming Xiong
    International Conference on Machine Learning (ICML), 2021
    [Axriv]

  35. Prototypical Contrastive Learning of Unsupervised Representations
    Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi
    International Conference on Learning Representations (ICLR), 2021
    [Axriv] [Bibtex] [Blog] [Code]

  36. Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation
    Shuai Lin, Pan Zhou, Xiaodan Liang, Jianheng Tang, Ruihui Zhao, Ziliang Chen and Liang Lin
    Association for the Advancement of Artificial Intelligence (AAAI), 2021
    [Axriv] [Bibtex] [Code]

  37. Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition
    Yubei Xiao, Ke Gong, Pan Zhou, Guolin Zheng, Xiaodan Liang and Liang Lin
    Association for the Advancement of Artificial Intelligence (AAAI), 2021
    [Axriv] [Bibtex] [Code]

  38. Efficient Gradient Support Pursuit with Less Hard Thresholding for Cardinality-Constrained Learning
    Fanhua Shang, Bingkun Wei, Hongying Liu, Yuanyuan Liu, Pan Zhou, and Maoguo Gong
    IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2021
    [PDF]

    39. 2020

  39. Theory-Inspired Path-Regularized Differential Network Architecture Search
    Pan Zhou, Caiming Xiong, Richard Socher, and Steven Hoi
    Neural Information Processing Systems (NeurIPS), 2020 (oral)
    [PDF] [SUPP] [Axriv] [Bibtex] [Blog] [Code] [Slides] [Poster]

  40. Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
    Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Hoi, and Weinan E
    Neural Information Processing Systems (NeurIPS), 2020
    [PDF] [SUPP] [Axriv] [Bibtex] [Code] [Slides] [Poster]

  41. Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
    Yue Wu, Pan Zhou, Andrew Gordon Wilson, Eric Xing, and Zhiting Hu
    Neural Information Processing Systems (NeurIPS), 2020
    [PDF] [Axriv] [Bibtex] [Codes]

  42. Hybrid Stochastic-Deterministic Minibatch Proximal Gradient: Less-Than-Single-Pass Optimization with Nearly Optimal Generalization
    Pan Zhou and Xiaotong Yuan
    International Conference on Machine Learning (ICML), 2020
    [PDF] [Axriv] [Bibtex]

    43. 2019

  43. Efficient Meta Learning via Minibatch Proximal Update
    Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, Jiashi Feng
    Neural Information Processing Systems (NeurIPS), 2019 (spotlight)
    [PDF] [SUPP] [Bibtex] [Codes] [Slides] [Poster]

  44. Tensor Low-rank Representation for Data Recovery and Clustering
    Pan Zhou, Canyi Lu, Jiashi Feng, Zhouchen Lin, Shuicheng Yan
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019
    [PDF] [SUPP] [Bibtex] [Codes]

  45. Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds
    Pan Zhou, Xiaotong Yuan, Shuicheng Yan, Jiashi Feng
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019
    [PDF] [Bibtex]

  46. Generalized Majorization-Minimization for Non-Convex Optimization
    Hu Zhang, Pan Zhou, Yi Yang, Jiashi Feng
    International Joint Conference on Artificial Intelligence (IJCAI), 2019
    [PDF] [Bibtex]

  47. Faster First-Order Methods for Stochastic Non-Convex Optimization on Riemannian Manifolds
    Pan Zhou, Xiaotong Yuan, Jiashi Feng
    International Conference on Artificial Intelligence and Statistics (AISTATS), 2019
    [PDF] [Bibtex]

    48. 2018

  48. Efficient Stochastic Gradient Hard Thresholding
    Pan Zhou, Xiaotong Yuan, Jiashi Feng
    Neural Information Processing Systems (NeurIPS), 2018
    [PDF] [Bibtex] [Codes]

  49. New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity
    Pan Zhou, Xiaotong Yuan, Jiashi Feng
    Neural Information Processing Systems (NeurIPS), 2018
    [PDF] [Bibtex]

  50. Understanding Generalization and Optimization Performance of Deep CNNs
    Pan Zhou, Jiashi Feng
    International Conference on Machine Learning (ICML), 2018
    [PDF] [Axriv] [Bibtex]

  51. Deep Adversarial Subspace Clustering
    Pan Zhou, Yunqing Hou, Jiashi Feng
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
    [PDF] [Codes] [Bibtex]

  52. Empirical Risk Landscape Analysis for Understanding Deep Neural Networks
    Pan Zhou, Jiashi Feng
    International Conference on Learning Representations (ICLR), 2018
    [PDF] [Axriv] [Bibtex]

  53. Task Relation Networks
    Jianshu Li, Pan Zhou, Yunpeng Chen, Jian Zhao, Sujoy Roy, Yan Shuicheng, Jiashi Feng, and Terence Sim
    IEEE Winter Conference on Applications of Computer Vision (WACV), 2019

    54. 2017

  54. Outlier-Robust Tensor PCA
    Pan Zhou, Jiashi Feng
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
    [PDF] [SUPP] [Codes] [Bibtex]

  55. Tensor Factorization for Low-Rank Tensor Completion
    Pan Zhou, Canyi Lu, Zhouchen Lin, Chao Zhang
    IEEE Transactions on Image Processing (TIP), 2017
    [PDF] [SUPP] [Codes] [Bibtex]

  56. Dictionary Learning with Structured Noise
    Pan Zhou, Cong Fang, Zhouchen Lin, Chao Zhang, Edward Y. Chang
    Neurocomputing, 2017
    [PDF] [Bibtex]

  57. Feature Learning via Partial Differential Equation with Applications to Face Recognition
    Cong Fang, Zhenyu Zhao, Pan Zhou, Zhouchen Lin
    Pattern Recognition (PR), 2017
    [PDF] [Codes] [Bibtex]

    58. 2016

  58. Bilevel Model Based Discriminative Dictionary Learning for Recognition
    Pan Zhou, Chao Zhang, Zhouchen Lin
    IEEE Transactions on Image Processing (TIP), 2016
    [PDF] [SUPP] [Bibtex]

  59. Integrated Low-Rank-Based Discriminative Feature Learning for Recognition
    Pan Zhou, Zhouchen Lin, Chao Zhang
    IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2016
    [PDF] [SUPP] [Codes] [Bibtex]


Books and Patents

  1. Tensors for Data Processing
    Chapter 6 is contributed by Pan Zhou, Canyi Lu, Zhouchen Lin
    Elsevier, 2022. [PDF]

  2. Neural network based scene text recognition
    Pan Zhou, Peng Tang, Ran Xu, Chu Hong Hoi
    US Patent, 2022. [PDF]

  3. Systems and methods for contrastive learning with self-labeling refinement
    Pan Zhou, Caiming Xiong, Chu Hong Hoi
    US Patent, 2022. [PDF]

  4. System and method for differential architecture search for neural networks
    Pan Zhou, Chu Hong Hoi
    US Patent, 2021. [PDF]