Long Zhao @ Google

Brief Bio. I am a Senior Research Scientist at Google DeepMind. Before joining Google, I obtained my Ph.D. in Computer Science from Rutgers University in 2022, advised by Distinguished Professor Dimitris N. Metaxas.

My current research interests lie primarily in (1) large vision foundation models (e.g., video-language models, multimodal models, world models), (2) generative models (e.g., diffusion models, visual autoregressive models), (3) self-supervised representation learning (e.g., contrastive learning, mask modeling), and (4) contextualized machine perception (e.g., recognition, detection, segmentation, localization).

340 Main Street
Los Angeles, CA 90291
Google LAX

Email: longzh [at] google [dot] com
[Curriculum Vitae]

Background

Education

Sept. 2016 - Jan. 2022

Rutgers, The State University of New Jersey - New Brunswick. Piscataway, NJ

Ph.D. in Computer Science

Supervised by Prof. Dimitris N. Metaxas
Ph.D. Thesis: "Towards Efficient and Reliable Skeleton-Based Human Pose Modeling"
GPA: 4.0/4.0

Sept. 2012 - Jun. 2015

Tongji University. Shanghai, China

M.S. in Software Engineering

Supervised by Prof. Jinyuan Jia and Prof. Shuang Liang
Master Thesis: "Sketch-Based 3D Model Retrieval"

Sept. 2008 - Jul. 2012

Tongji University. Shanghai, China

B.Eng. in Software Engineering

GPA: 4.56/5.0, which is equivalent to 90.6 on 100 basis

Experience

May 2024 - Present

Applied Research, Google DeepMind. Los Angeles, CA, USA

Research Scientist.

Video Understanding (e.g., Multimodal Models, World Models)
Video Generation (e.g., Diffusion Models, Visual Autoregressive Models)
Human-Centric Perception (e.g., Recognition, Detection)

Nov. 2021 - May 2024

Perception Team, Google Research. Los Angeles, CA, USA

Research Scientist.

Video Foundation Models and Benchmarks [arXiv'23] [ICML'24]
Multimodal Learning with Vision-Language Models [ICLR'24] [CVPR'24]
Contextualized Machine Perception [ICCV'23] [CVPR'24] [CVPR'24]

Dec. 2020 - May 2021

Brain Team, Google Research. Mountain View, CA, USA

Student Researcher. Host: Dr. Han Zhang

Boosting Transformers for High-Resolution Image Generation [NeurIPS'21]
Improving Efficiency and Interpretability for Vision Transformers [AAAI'22 (Oral)]

May 2020 - Dec. 2020

Mobile Vision Team, Google Research. Los Angeles, CA, USA

Research Intern & Student Researcher. Host: Dr. Ting Liu

View-Disentangled Human Pose Representation Learning [CVPR'21 (Oral)]
View-Invariant, Occlusion-Robust Probabilistic Pose Embedding [IJCV'21]

Sept. 2016 - Present

Computer Science Department, Rutgers University. Piscataway, NJ, USA

Research Assistant. Supervised by Prof. Dimitris N. Metaxas

3D Human/Hand Pose Estimation from RGB Images [CVPR'19] [CVPR'20]
Face/Pose/Video Generation with Generative Adversarial Networks [IJCAI'18] [ECCV'18] [IJCV'20]
Domain Generalization via Adversarial Training & Meta-Learning [CVPR'20] [NeurIPS'20]
Representation Learning on Graphs with Graph Convolutional Networks [NeurIPS'19]

Dec. 2013 - Nov. 2014

Visual Computing Group, Microsoft Research Asia (MSRA). Beijing, China

Research Intern. Mentor: Dr. Yichen Wei

Generic Object Proposal Generation [CVPR'15]
Salient Object Detection [ACCV'14]
Won the award of excellence in "MSRA Stars of Tomorrow Internship Program"

Selected Publications

(^* indicates equal contributions. Please check Google Scholar for the full list of my publications.)

VideoPrism: A Foundational Visual Encoder for Video Understanding

Long Zhao^*, Nitesh B. Gundavarapu^*, Liangzhe Yuan^*, Hao Zhou^*, Shen Yan^#, Jennifer J. Sun^#, Luke Friedman^#, Rui Qian^#, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko⁺, Ting Liu⁺, Boqing Gong⁺. (^*equal primary contributions; ^#equal core technical contributions; ⁺equal senior contributions.)

International Conference on Machine Learning (ICML), 2024.

Distilling Vision-Language Models on Millions of Videos

Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, Philipp Krähenbühl, Liangzhe Yuan.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[arXiv]

Generating Enhanced Negatives for Training Language-Based Object Detectors

Shiyu Zhao, Long Zhao, Vijay Kumar B.G, Yumin Suh, Dimitris N. Metaxas, Manmohan Chandraker, Samuel Schulter.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[arXiv]

Taming Self-Training for Open-Vocabulary Object Detection

Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, Vijay Kumar B.G, Yumin Suh, Manmohan Chandraker, Dimitris N. Metaxas.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding

Yuanhao Xiong, Long Zhao, Boqing Gong, Ming-Hsuan Yang, Florian Schroff, Ting Liu, Cho-Jui Hsieh, Liangzhe Yuan.

International Conference on Learning Representations (ICLR), 2024.

[arXiv]

Out-of-Domain Generalization from a Single Source: An Uncertainty Quantification Approach

Xi Peng, Fengchun Qiao, Long Zhao.

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Volume 46, Issue 3, 2024.

[arXiv]

VideoGLUE: Video General Understanding Evaluation of Foundation Models

Liangzhe Yuan^*, Nitesh B. Gundavarapu^*, Long Zhao^*, Hao Zhou^*, Yin Cui, Lu Jiang, Xuan Yang, Menglin Jia, Tobias Weyand, Luke Friedman, Mikhail Sirotenko, Huisheng Wang, Florian Schroff, Hartwig Adam, Ming-Hsuan Yang, Ting Liu, Boqing Gong.

Technical Report, arXiv:2307.03166, 2023.

Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition

Qitong Wang, Long Zhao, Liangzhe Yuan, Ting Liu, Xi Peng.

IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Unified Visual Relationship Detection with Vision and Language Models

Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming-Hsuan Yang, Hartwig Adam, Ting Liu.

IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, Vijay Kumar B.G, Anastasis Stathopoulos, Manmohan Chandraker, Dimitris Metaxas.

European Conference on Computer Vision (ECCV), 2022.

Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning

Yuxiao Chen, Long Zhao, Jianbo Yuan, Yu Tian, Zhaoyang Xia, Shijie Geng, Ligong Han, Dimitris N. Metaxas.

European Conference on Computer Vision (ECCV), 2022.

COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality

Honglu Zhou, Asim Kadav, Aviv Shamsian, Shijie Geng, Farley Lai, Long Zhao, Ting Liu, Mubbasir Kapadia, Hans Peter Graf.

European Conference on Computer Vision (ECCV), 2022.

Global Matching with Overlapping Attention for Optical Flow Estimation

Shiyu Zhao, Long Zhao, Zhixing Zhang, Enyu Zhou, Dimitris Metaxas.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Are Multimodal Transformers Robust to Missing Modality?

Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, Xi Peng.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

[arXiv]

Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding

Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan O. Arik, Tomas Pfister.

AAAI Conference on Artificial Intelligence (AAAI), 2022. [Oral Presentation]

View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose

Ting Liu^*, Jennifer J. Sun^*, Long Zhao, Jiaping Zhao, Liangzhe Yuan, Yuxiao Wang, Liang-Chieh Chen, Florian Schroff, Hartwig Adam.

International Journal of Computer Vision (IJCV), Volume 130, Issue 1, 2022.

Improved Transformer for High-Resolution GANs

Long Zhao, Zizhao Zhang, Ting Chen, Dimitris N. Metaxas, Han Zhang.

Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.

Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization

Long Zhao, Yuxiao Wang, Jiaping Zhao, Liangzhe Yuan, Jennifer J. Sun, Florian Schroff, Hartwig Adam, Xi Peng, Dimitris Metaxas, Ting Liu.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. [Oral Presentation]

SMIL: Multimodal Learning with Severely Missing Modality

Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, Xi Peng.

AAAI Conference on Artificial Intelligence (AAAI), 2021.

Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness

Long Zhao, Ting Liu, Xi Peng, Dimitris Metaxas.

Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.

Towards Image-to-Video Translation: A Structure-Aware Approach via Multi-Stage Generative Adversarial Networks

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N. Metaxas.

International Journal of Computer Vision (IJCV), Volume 128, Issue 10, 2020.

Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge

Long Zhao, Xi Peng, Yuxiao Chen, Mubbasir Kapadia, Dimitris N. Metaxas.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[arXiv]

Learning to Learn Single Domain Generalization

Fengchun Qiao, Long Zhao, Xi Peng.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Rethinking Kernel Methods for Node Representation Learning on Graphs

Yu Tian^*, Long Zhao^*, Xi Peng, Dimitris N. Metaxas.

Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.

Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention

Yuxiao Chen, Long Zhao, Xi Peng, Jianbo Yuan, Dimitris N. Metaxas.

British Machine Vision Conference (BMVC), 2019.

Semantic Graph Convolutional Networks for 3D Human Pose Regression

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N. Metaxas.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Learning to Forecast and Refine Residual Motion for Image-to-Video Generation

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N. Metaxas.

European Conference on Computer Vision (ECCV), 2018.

CR-GAN: Learning Complete Representations for Multi-view Generation

Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, Dimitris N. Metaxas.

International Joint Conference on Artificial Intelligence (IJCAI), 2018.

Object Proposal by Multi-branch Hierarchical Segmentation

Chaoyang Wang, Long Zhao, Shuang Liang, Liqing Zhang, Jinyuan Jia, Yichen Wei.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.