Luowei Zhou

Robotics Institute

University of Michigan

I am a Ph.D. Candidate at University of Michigan. My advisor is Prof. Jason Corso in Department of EECS. I received my bachelor's degree in Engineering from Nanjing University in 2015.

My research focuses on vision-language embedding, such as captioning and question answering, and video action understanding. My work intensively relies on deep learning and general machine learning/optimization algorithms. My most recent efforts are on automatic video understanding, featured projects include large-scale cooking video dataset YouCook2, gounded video description, and dense video captioning. Previously, I worked on multi-agent reinforcement learning at Nanjing University. I have spent summer research interns at Facebook AI Research, MSR, and Salesforce Research.


  • [NEW] I am on the job market for a full-time researcher position on AI, CV+NLP, and Deep Learning. Feel free to drop me an email!
  • [11/2019] Our VLP work is accepted by AAAI 2020! Media coverages include MSR blog, VentureBeat, and Medium.
  • [09/2019] We introduce our work on Unified Vision-Language Pre-training (VLP), which achieves SotA on image captioning and VQA (datasets: COCO/VQA2.0/Flickr30k) with a single model architecture. The code is released on Github and feel free to try it out!
  • [09/2019] I am working with Prof. Justin Johnson on a new class on Deep Learning for Computer Vision at UMich.
  • [04/2019] We released the source code for our CVPR 2019 oral paper Grounded Video Description! The evaluation server for our dataset ANet-Entities is live on Codalab!
  • [04/2019] Our grounded video description paper is accepted by CVPR 2019 (oral). We made the ActivityNet-Entities dataset (158k bboxes on 52k captions) available at Github including evaluation scripts. Source code is on the way!
  • [07/2019] Our paper on Dynamic Graph Modules for Activity Recognition is accepted to BMVC 2019.
  • [04/2019] We released the source code for our BMVC 2018 work on weakly supervised video object grounding! The new YouCook2-BB dataset is available for download. The evaluation server is on Codalab!
  • [03/2019] Serve as a program committee member/reviewer for CVPR, ICCV, TPAMI, IJCV, AAAI, ACM MM etc.
  • [02/2019] We released the source code on dense video captioning (CVPR 2018).
  • [12/2018] Two invited talks at Samsung AI Toronto and NVIDIA AI Toronto.
  • [09/2018] Our weakly-supervised video grounding paper is accepted by BMVC 2018.
  • [03/2018] Our dense video captioning paper is accepted by CVPR 2018 (spotlight).
  • [02/2018] I will join Facebook AI Research (FAIR) for Research Intern in summer 2018.
  • [02/2018] I will co-organize CVPR 2018 Workshop on Fine-grained Instructional Video Understanding (FIVER).
  • [11/2017] Our paper on YouCook2 and procedure segmentation is accepted by AAAI 2018 (oral).
  • [01/2017] I will join Salesforce Metamind for Research Intern in summer 2017, supervised by Caiming Xiong.