Publications

OAWM: Generative World Model with Object-Level Representations and Actions

Published in In Submission(First Author), 2025

Learning the world model from videos has been deemed as a way to train foun- dation models for vision-based control tasks. Since most videos online doesn’t have action label, some methods propose to infer the latent action from videos to scale up the learning. However, current methods are restricted to the videos where there is a clear distinction between one agent (in motion) and environment (still), limiting the potential to use massive videos online. To address this limitation, we propose to infer the object-centric representations and actions from the video and learn a world model using them. This method solves the latent action inference ambiguity problem in scenes where there are multiple moving objects. It also nat- urally enables better counterfactual imagination and planning. Importantly, it’s way more computationally efficient during planning. Meanwhile we manage to generate high quality natural videos using these latent representations. Thereby, we are able to think abstractly meanwhile see it vividly.

Global Prefix-Tuning: Extremely Efficient Fine-Tuning for Shallow Alignment Using One Token

Published in ACL(submitted), 2024

We introduce Global-Token Tuner, an extremely parameter-efficient fine-tuning (PEFT) method for adapting Large Language Models (LLMs) that uses only a few or just one learnable token, regardless of model size. Global-Token Tuner employs a unique design that constructs a globally shared set of tunable tokens that modify the attention of every layer. Therefore no matter how base model change the tunable parameter remains relatively constant. We showed that our method can attain comparable performance with LoRA across plenty of common tasks while reducing parameter requirements from multiple millions or more to as few as 5 thousand. We also believe the discovery that even one token can effectively finetune LLMs illuminates the inner workings of LLMs.

Download Paper

Glancee: An Adaptable System for Instructors to Grasp Student Learning Status in Synchronous Online Classes

Published in CHI 2022, 2022

Synchronous online learning has become a trend in recent years. However, instructors often face the challenge of inferring audiences’ reactions and learning status without seeing their faces in video feeds, which prevents instructors from establishing connections with students. To solve this problem, based on a need-finding survey with 67 college instructors, we propose Glancee, a real-time interactive system with adaptable configurations, sidebar-based visual displays, and comprehensive learning status detection algorithms. Then, we conduct a within-subject user study in which 18 college instructors deliver lectures online with Glancee and two baselines, EngageClass and ZoomOnly. Results show that Glancee can effectively support online teaching and is perceived to be significantly more helpful than the baselines. We further investigate how instructors’ emotions, behaviors, attention, cognitive load, and trust are affected during the class. Finally, we offer design recommendations for future online teaching assistant systems.