Jitendra Malik

DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

DreamDojo：从大规模人类视频中学习通用机器人世界模型

Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, Linxi "Jim" Fan

机构 * NVIDIA ； HKUST（香港科技大学） ； UC Berkeley（加州大学伯克利分校） ； Stanford（斯坦福大学） ； KAIST（韩国科学技术院） ； UofT（多伦多大学） ； UCSD（加州大学圣地亚哥分校） ； UT Austin（德克萨斯大学奥斯汀分校） 纠错

AI总结 DreamDojo通过大规模人类视频学习通用机器人世界模型，解决动作标签稀缺问题，实现高精度物理理解和实时交互。

Comments Project page: https://dreamdojo-world.github.io/

SPIDER: Scalable Physics-Informed Dexterous Retargeting

SPIDER：可扩展的物理引导的灵活重新定位

Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, Francois Hogan

机构 * FAIR at Meta（Meta 的 FAIR） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

AI总结 SPIDER 是一种基于物理的重新定位框架，通过大规模物理采样和课程式虚拟接触指导，将人类运动数据转化为动态可行的机器人轨迹，提升策略学习效率。

Comments Project website: https://jc-bao.github.io/spider-project/

Coordinated Humanoid Manipulation with Choice Policies

协调的人形机器人操作与选择策略

Haozhi Qi, Yen-Jen Wang, Toru Lin, Brent Yi, Yi Ma, Koushil Sreenath, Jitendra Malik

机构 * UC Berkeley（加州大学伯克利分校） 纠错

AI总结 本文提出选择策略，通过模块化遥控和模仿学习，实现人形机器人在无结构环境中的协调操作与高效学习。

Comments Code and Website: https://choice-policy.github.io/

Tracking by Predicting 3-D Gaussians Over Time

通过时间预测三维高斯分布进行跟踪

Tanish Baranwal, Himanshu Gaurav Singh, Jathushan Rajasegaran, Jitendra Malik

机构 * University of California, Berkeley（加州大学伯克利分校） 纠错

AI总结 Video-GMAE通过时间预测三维高斯分布实现自监督视频跟踪，显著提升了Kinetics和Kubric数据集的跟踪性能。

OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

OTTER: 一种具有文本感知视觉特征提取的视觉-语言-动作模型

Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel

机构 * University of California, Berkeley（加州大学伯克利分校） ； Meta AI 纠错

AI总结 OTTER通过文本感知的视觉特征提取，提升视觉-语言-动作模型的零样本泛化能力。

Reconstructing Hand-Held Objects in 3D from Images and Videos

从图像和视频中重建手持物体的三维结构

Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik

机构 * UC Berkeley（伯克利大学） ； UT Austin（德克萨斯大学奥斯汀分校） ； Caltech（加州理工学院） 纠错

AI总结 本文提出了一种基于3D手和物体联合重建的方法，通过检索增强重建实现对视频中手持物体的高效三维重建。

Comments 3DV 2026, Project page: https://janehwu.github.io/mcc-ho

FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos

FoundationMotion: 自动标注与视频中空间运动的推理

Yulu Gan, Ligeng Zhu, Dandan Shan, Baifeng Shi, Hongxu Yin, Boris Ivanovic, Song Han, Trevor Darrell, Jitendra Malik, Marco Pavone, Boyi Li

机构 * MIT（麻省理工学院） ； NVIDIA（英伟达） ； UMich（密歇根大学） ； UC Berkeley（加州大学伯克利分校） ； Stanford University（斯坦福大学） 纠错

AI总结 FoundationMotion通过自动数据整理管道生成大规模细粒度运动数据集，提升开源模型在运动理解与空间推理任务中的性能。

Comments Code is available at https://github.com/Wolfv0/FoundationMotion/tree/main

From Generated Human Videos to Physically Plausible Robot Trajectories

从生成的人类视频到物理上合理的机器人轨迹

James Ni, Zekai Wang, Wei Lin, Amir Bar, Yann LeCun, Trevor Darrell, Jitendra Malik, Roei Herzig

机构 * University of California, Berkeley（加州大学伯克利分校） ； New York University（纽约大学） ； Johannes Kepler University（约翰·凯撒大学） 纠错

AI总结 本文提出GenMimic方法，通过物理感知强化学习策略，从生成的视频中实现零样本的人形机器人动作模仿，并建立了评估零样本泛化能力的基准。

Comments For project website, see https://genmimic.github.io

OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer

OSMO：开源触觉手套用于人到机器人技能转移

Jessica Yin, Haozhi Qi, Youngsun Wi, Sayantan Kundu, Mike Lambeta, William Yang, Changhao Wang, Tingfan Wu, Jitendra Malik, Tess Hellebrekers

机构 * Meta FAIR ； University of Michigan（密歇根大学） ； University of Pennsylvania（宾夕法尼亚大学） 纠错

AI总结 OSMO通过触觉手套实现人到机器人技能转移，利用触觉数据提升机器人操作性能。

Comments Project website: https://jessicayin.github.io/osmo_tactile_glove/

Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations

智能眼镜中的灵巧性：基于真实世界人类示范的多指机器人操控

Irmak Guzey, Haozhi Qi, Julen Urain, Changhao Wang, Jessica Yin, Krishna Bodduluri, Mike Lambeta, Lerrel Pinto, Akshara Rai, Jitendra Malik, Tingfan Wu, Akash Sharma, Homanga Bharadhwaj

机构 * New York University（纽约大学） ； Meta 纠错

AI总结 本文提出AINA框架，通过智能眼镜采集真实世界人类示范数据，实现多指机器人操控策略的学习，无需依赖传统机器人数据。

DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy

Yuran Wang, Ruihai Wu, Yue Chen, Jiarui Wang, Jiaqi Liang, Ziyu Zhu, Haoran Geng, Jitendra Malik, Pieter Abbeel, Hao Dong

机构 * Peking University（北京大学） ； University of California, Berkeley（加州大学伯克利分校） 纠错

Comments NeurIPS2025 Spotlight

The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio

Renhao Wang, Haoran Geng, Tingle Li, Feishi Wang, Gopala Anumanchipalli, Trevor Darrell, Boyi Li, Pieter Abbeel, Jitendra Malik, Alexei A. Efros

机构 * University of California, Berkeley（加州大学伯克利分校） 纠错

Comments Conference on Robot Learning 2025

End-to-end RL Improves Dexterous Grasping Policies

Ritvik Singh, Karl Van Wyk, Pieter Abbeel, Jitendra Malik, Nathan Ratliff, Ankur Handa

机构 * NVIDIA ； University of California, Berkeley（加州大学伯克利分校） 纠错

Comments See our blog post: https://e2e4robotics.com/

Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids

Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, Yuke Zhu

机构 * UC Berkeley（伯克利大学） ； NVIDIA（英伟达） ； UT Austin（奥斯汀大学） 纠错

Comments Published at CoRL 2025. Project page can be found at https://toruowo.github.io/recipe/

Visual Imitation Enables Contextual Humanoid Control

Arthur Allshire, Hongsuk Choi, Junyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, Angjoo Kanazawa

机构 * UC Berkeley（伯克利大学） 纠错

Comments Project website: https://www.videomimic.net/

Deep Sensorimotor Control by Imitating Predictive Models of Human Motion

Himanshu Gaurav Singh, Pieter Abbeel, Jitendra Malik, Antonio Loquercio

机构 * UC Berkeley（伯克利大学） ； University of Pennsylvania（宾夕法尼亚大学） 纠错

Comments Blog Post: https://hgaurav2k.github.io/trackr/

Embodied AI Agents: Modeling the World

Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hongyu Gong, Hervé Jégou, Alessandro Lazaric, Arjun Majumdar, Andrea Madotto, Franziska Meier, Florian Metze, Louis-Philippe Morency, Théo Moutakanni, Juan Pino, Basile Terver, Joseph Tighe, Paden Tomasello, Jitendra Malik

机构 * Meta AI Research（Meta AI 研究） 纠错

Whole-Body Conditioned Egocentric Video Prediction

Yutong Bai, Danny Tran, Amir Bar, Yann LeCun, Trevor Darrell, Jitendra Malik

机构 * UC Berkeley (BAIR)（伯克利大学（BAIR）） ； FAIR, Meta（Meta 公司 FAIR 实验室） ； New York University（纽约大学） 纠错

Comments Project Page: https://dannytran123.github.io/PEVA

TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy

Héctor Carrión, Yutong Bai, Víctor A. Hernández Castro, Kishan Panaganti, Ayush Zenith, Matthew Trang, Tony Zhang, Pietro Perona, Jitendra Malik

机构 * Tera AI ； UC Santa Cruz（加州大学圣克ruz分校） ； UC Berkeley（加州大学伯克利分校） ； California Institute of Technology（加州理工学院） 纠错

Comments Computer Vision, Pattern Recognition, Early-Fusion, Dataset, Data Augmentation

SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending

Yuxuan Kuang, Haoran Geng, Amine Elhafsi, Tan-Dzung Do, Pieter Abbeel, Jitendra Malik, Marco Pavone, Yue Wang

机构 * University of Southern California（南加州大学） ； Stanford University（斯坦福大学） ； Peking University（北京大学） ； University of California, Berkeley（加州大学伯克利分校） 纠错

A Learning-based Quadcopter Controller with Extreme Adaptation

Dingqi Zhang, Antonio Loquercio, Jerry Tang, Ting-Hao Wang, Jitendra Malik, Mark W. Mueller

机构 * High Performance Robotics Lab, Dept. of Mechanical Engineering, UC Berkeley（伯克利大学高性能机器人实验室，机械工程系） ； University of Pennsylvania（宾夕法尼亚大学） ； Dept. of Electrical Engineering and Computer Science, University of California at Berkeley（伯克利大学电子工程与计算机科学系） 纠错

Comments Accepted for the Transaction on Robotics (T-RO), April 2025

AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time

Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning, Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, Huan Zhang

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； UC Berkeley（伯克利加州大学） 纠错

Reconstructing People, Places, and Cameras

Lea Müller, Hongsuk Choi, Anthony Zhang, Brent Yi, Jitendra Malik, Angjoo Kanazawa

机构 * UC Berkeley（伯克利大学） 纠错

Comments Project website: muelea.github.io/hsfm

Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing

Jessica Yin, Haozhi Qi, Jitendra Malik, James Pikul, Mark Yim, Tess Hellebrekers

机构 * Meta AI Research（Meta AI研究） 纠错

Comments Website: https://jessicayin.github.io/tactile-skin-rl/. Accepted to ICRA 2025

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, Zipeng Lin

Comments Project website: https://robotics-transformer-x.github.io

RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning

Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Yue Wang, Jitendra Malik, Pieter Abbeel

机构 * UC Berkeley（加州大学伯克利分校） ； PKU（北京大学） ； USC（南加州大学） ； UMich（密歇根大学） ； UIUC（伊利诺伊大学厄巴纳-香槟分校） ； Stanford（斯坦福大学） ； CMU（卡内基梅隆大学） ； UCLA（加州大学洛杉矶分校） ； BIGAI（大疆创新） 纠错

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, Chelsea Finn

Comments Project website: https://droid-dataset.github.io/

Wolf: Dense Video Captioning with a World Summarization Framework

Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Linxi Fan, Yuke Zhu, Jan Kautz, Andrew Tao, Ming-Yu Liu, Sanja Fidler, Boris Ivanovic, Trevor Darrell, Jitendra Malik, Song Han, Marco Pavone

Poly-Autoregressive Prediction for Modeling Interactions

Neerja Thakkar, Tara Sadjadpour, Jathushan Rajasegaran, Shiry Ginosar, Jitendra Malik

Comments preprint

Interactive Task Planning with Language Models

Boyi Li, Philipp Wu, Pieter Abbeel, Jitendra Malik

Comments Transactions on Machine Learning Research (TMLR), 2025