|
|
Jose M. Alvarez I lead the Physical AI and Autonomous Systems Research Group at NVIDIA, CA, USA. We build autonomous systems that perceive, predict, simulate, and reason before they act, and that keep improving through closed-loop data and evaluation. |
News
- 1 Paper accepted to NeurIPS 2026!
- We are organizing the World Models in Physical AI workshop at NeurIPS 2026.
- I gave an invited lecture at the ELLIS Summer School on Autonomous Driving in Barcelona: End-to-End Models for Autonomous Driving: From Direct Prediction to Counterfactual Deliberation on the Edge.
- I gave an invited talk at the DriveX Workshop @ ECCV 2026: Beyond Prediction: Counterfactual Deliberation for L4 Autonomy.
- I gave a talk at Advanced Factories 2026 in Barcelona: Physical AI: The Next Step beyond Perception, Generative and Agentic AI.
- I gave an invited talk at the AUTOPILOT Workshop @ CVPR 2026: Teaching Physical AI to Think Ahead: The Proactive Evaluation Flywheel for Safe Autonomy.
- I gave an invited talk at the VOCVALC Workshop @ CVPR 2026: Pixels to Propositions: Language as the Backbone of Video Understanding.
- I gave a talk at TechEx North America 2026 in San Jose: Accelerating Autonomous Mobility with NVIDIA DRIVE.
- 2 Papers accepted to ECCV 2026!
- 4 Papers accepted to CVPR 2026!
- I gave a talk at the Voxel51 meetup in Silicon Valley on Self-Improving AI Models.
- DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with VLM has been accepted to ICRA 2026!
- ChronoEdit has been accepted to ICLR 2026!
- We weill be organizing 3 Workshops at CVPR 2026:
- Workshop on Autonomous Driving
- Exploring the Next Generation of Data
- The 5th Workshop on Transformers for Vision and Multimodal AI
- 1 Paper accepted to AAAI 2026: DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning
- We are organizing the 3rd Workshop on Scalable 3D Scene Generation.
- I gave the opening Keynote at DLBCN on self-improving AI models.
- 2 Invited Talks at ICCV 2025 Workshops. Stay Tuned
- I will organize, together with Antonio M. Lopez and Dariu Gavrila (Program Chair), the Intelligent Vehicle Conference in 2028 in Barcelona!
- Keynote at the Deep Learning Symposium in Barcelona, in December 2025. See you all in Barcelona.
- Keynote talk at FISITA 2025 (Intelligent Safety Conference 2025), Xi'an China. Stay Tuned.
- 1 Paper accepted to NeurIPS 2025!
- June 2025: Our Our Generalized Trajectory Scoring Method won the End-to-End Driving challenge at CVPR 2025!
- 1 Paper accepted to ICCV 2025!
- Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training
- 4 Papers accepted to CVPR 2025!
- MDP: Multidimensional Vision Model Pruning with Latency Constraint
- PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
- Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video
- OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counter Factual Reasoning
- We are organizing 2 tutorials at CVPR 2025.
- Full-Stack, GPU-based Acceleration of Deep Learning tutorial.
- Advancing Data Strategies for AI success (Continuous Data Cycle via Foundation Models).
- Our paper Advancing Weight and Channel Sparsification with Enhanced Saliency has been accepted to WACV 2025!
- We are organizing three workshops at CVPR 2025:
- Workshop on Exploring the Next Generation of Data.
- The 8th Edition of Workshop on Autonomous Driving.
- The 4th Edition of Transformers for Vision (T4V workshop).
- September 2024: I will talk at the Green Foundation Models Workshop and the Camera based Driving workshop at ECCV 2024.
- September 2024: Our paper Memorize What Matters: Emergent scene decomposition from multitraverse has been accepted as oral presentation to NeruIPS 2024.
- July 2024: Our paper SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation has been accepted to ECCV 2024.
- June 2024: Our HydraMDP approach won the End-to-End Driving at Scale challenge at CVPR 2024.
- June 2024: Our LLM Agent for driving was the second best model in Driving With Language Challenge at CVPR 2024.
- March 2024: 1 paper accepted at IV 2024, and Finalist for Best Student Paper award.
- Feb 2024: 4 papers accepted at CVPR 2024.
- Feb 2024: We will host the GPU-based DL Acceleration tutorial at CVPR 2024.
- Feb 2024: We will host the 7th edition of the Workshop on Autonomous Driving at CVPR 2024.
- Jan 2024: 2 papers accepted at ICLR 2024 on efficient and robust large models.
- July 2023: 6 new papers accepted at ICCV 2023
- Fully Attentional Networks with Self-emerging Token Labeling. Code available!
- FB-BEV: BEV Representation from Forward-Backward View Transformations. Code available!
- FocalFormer3D: Focusing on Hard Instance for 3D Object Detection. Code available!
- Parametric Depth Based Feature Representation Learning for Object Detection and Segmentation in Bird's-Eye View. Code available soon!
- Towards Viewpoint Robustness in Bird's Eye View Segmentation. Code and datasets available!
- End-to-end 3D Tracking with Decoupled Queries. Code available!
- April: We will be organizing a tutorial at ICCV, Paris, France 2023. Stay tunned.
- Learning with Noisy and Unlabeled Data for Large Models beyond Categorization.
- March: 2 new papers accepted at IV 2023
- Shen et al. Hardware-Aware Latency Pruning for Real-Time 3D Object Detection.
- Clemons et al. Leaf: Legacy Networks for Flexible Inference.
- March: 3 new papers accepted at CVPR 2023
- VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion
- Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions
- Vision Transformers are Good Mask Auto-Labelers. Code available!
- I will be talking about data for atonomous driving at Autonomous Vehicles USA 2023, April 17th, Anaheim, California.
- Full-Stack, GPU-based Acceleration of Deep Learning tutorial at CVPR 2023. Stay tunned.
- We are organizing the Workshop on Autonomous Driving at CVPR 2023.
- We are organizing the T4V Workshop: Transformers for Vision at CVPR 2023.
- Sept. 2022: 1 paper accepted at BMVC 2022.
- Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks
- September 2022: 2 papers accepted at NeurIPS 2022
- July 2022: 1 paper accepted at ECCV 2022.
- April 2022: Our paper on Robustness in Transformers has been accepted at ICML 2022
- March 2022: 7 new papers accepted at CVPR 2022
- FreeSOLO: Learning to Segment Objects without Annotations
- How Much More Data Do I Need? Estimating Requirements for Downstream Tasks
- A-ViT: Adaptive Tokens for Efficient Vision Transformer
- Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo
- Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection
- When to Prune? A Policy towards Early Structural Pruning
- Panoptic SegFormer: Delving deeper into panoptic segmentation with transformers
- Dec 2021: Presenting 2 papers at NeurIPS 2021:
- October 2021: Presenting our active learning paper at ICCV 2021
- 1 new paper accepted at ICML 2021
- 2 new papers accepted at ICLR 2021
- 1 new paper accepted at ICCV 2021
- 3 new papers accepted at CVPR 2021
- 1 new paper accepted at WACV 2021
- 1 new paper accepted at IEEE-IV 2021
- 2 new papers accepted at IEEE CVPR 2020
- 1 new paper accepted at IEEE-IV 2020
- 1 new paper accepted at NeurIPS 2020
- 1 new paper accepted at WACV 2020
- 1 new paper accepted at ICLR 2020
Featured Research
Multimodal Planners
End-to-end planners that generate and score diverse trajectory candidates, reason about alternatives, and learn from closed-loop feedback.
- LDiP: Large Discrete Policy: Advancing Explicit Behavior Modeling with Stochastic Iterative ScoringNeurIPS 2026
- ZTRS: Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory ScorerECCV 2026
- HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End DrivingECCV 2026
- DriveSuprim: Towards Precise Trajectory Selection for End-to-End PlanningAAAI 2026
- Generalized Trajectory Scoring for End-to-end Multimodal PlanningWinner, CVPR 2025 End-to-End Driving Challenge · Code
- Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop TrainingICCV 2025
- Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-DistillationWinner, CVPR 2024 End-to-End Driving Challenge · Code
Safety and Evaluation for Autonomous Systems
Measuring driving quality the way humans judge it, and testing and training systems proactively on the safety-critical cases that matter most.
- Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI DevelopmentICML 2026, Position Paper · Project
- DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language ModelsarXiv 2026
- DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language ModelsICRA 2026 · Project · Code
- Enhancing Autonomous Driving Safety with Collision Scenario IntegrationIROS 2025, Oral · Project
Data-Centric AV
Deciding which data to collect, label, and train on, so that models improve efficiently as data grows rather than by volume alone.
- Test-Time Coverage (TTCov): Test-Conditioned Data Curation for Deployment-Aware LearningarXiv 2026 · Project
- Scaling-Aware Data Selection for End-to-End Autonomous Driving SystemsCVPR 2026
- SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data AssimilationKDD 2025
- Vision Transformers are Good Mask Auto-LabelersCVPR 2023 · Code
- Optimizing Data Collection for Machine LearningNeurIPS 2022
- How Much More Data Do I Need? Estimating Requirements for Downstream TasksCVPR 2022
- Not All Labels Are Equal: Rationalizing the Labeling Costs for Training Object DetectionCVPR 2022 · Code
- Active Learning for Deep Object Detection via Probabilistic ModelingICCV 2021 · Code
Synthetic Data Generation
Using simulated and generated data to cover viewpoints, conditions, and scenarios that are rare or costly to capture on real roads.
- 4D-Surf: High-Fidelity Dynamic Scene Surface ReconstructionCVPR 2026
- ChronoEdit: Towards Temporal Reasoning for Image Editing and World SimulationICLR 2026 · Project · Code
- Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop TrainingICCV 2025
- Memorize What Matters: Emergent Scene Decomposition from MultitraverseNeurIPS 2024, Oral · Project · Code
- Towards Viewpoint Robustness in Bird’s Eye View SegmentationICCV 2023 · Code
- Contrastive Syn-to-Real GeneralizationICLR 2021 · Code
- Effective Use of Synthetic Data for Urban Scene Semantic SegmentationECCV 2018
Efficiency and Robustness
Making large perception and planning models fast enough for real-time deployment on the edge, and reliable under distribution shift.
- MDP: Multidimensional Vision Model Pruning with Latency ConstraintCVPR 2025
- Adaptive Sharpness-Aware Pruning for Robust Sparse NetworksICLR 2024
- Structural Pruning via Latency-Saliency KnapsackNeurIPS 2022 · Project
- Understanding The Robustness in Vision TransformersICML 2022 · Code
- A-ViT: Adaptive Tokens for Efficient Vision TransformerCVPR 2022 · Project · Code
- When to Prune? A Policy towards Early Structural PruningCVPR 2022
- SegFormer: Simple and Efficient Design for Semantic Segmentation with TransformersNeurIPS 2021 · Code
Invited Talks
Featured
Workshop
Teaching Physical AI to Think Ahead: The Proactive Evaluation Flywheel for Safe Autonomy
Workshop
Pixels to Propositions: Language as the Backbone of Video Understanding
Keynote
Self-improving Physical AI with agents in the loop
Keynote
Towards Safer and More Generalizable Autonomous Vehicles
All talks
- Lecture End-to-End Models for Autonomous Driving: From Direct Prediction to Counterfactual Deliberation on the Edge
- Workshop Beyond Prediction: Counterfactual Deliberation for L4 Autonomy
- Workshop Teaching Physical AI to Think Ahead: The Proactive Evaluation Flywheel for Safe Autonomy
- Workshop Pixels to Propositions: Language as the Backbone of Video Understanding
- Industry Accelerating Autonomous Mobility with NVIDIA DRIVE
- Industry Physical AI: The Next Step beyond Perception, Generative and Agentic AI
- Talk Self-improving (Physical) AI with reasoning in the loop
- Keynote Self-improving Physical AI with agents in the loop
- Workshop End-to-End Driving in the Era of Foundation Models
- Workshop Scalable Autonomous Driving with Reasoning in the Loop
- Workshop System-Level Optimization for Real-Time Inference
- Talk Less is More: Accelerating AI with Advanced Data Strategies
- Talk Less is More: Accelerating AI with Advanced Data Strategies
- Keynote Towards Safer and More Generalizable Autonomous Vehicles
- Workshop Multimodal Planners for Dynamic Environments
- Workshop Towards Safer and More Generalizable Autonomous Vehicles
- Industry From Research to Product: Transforming AV Technology with AI
- Seminar Towards Robust and Reliable Autonomous Vehicles with Foundation Models
- Seminar Towards Safe and Reliable Autonomous Vehicles
- Workshop Towards an Effective use of Foundation Models in Autonomous Driving
- Workshop Towards Robust and Reliable AV with Foundational Models
- Industry Achieving Better Accuracy in 3D Occupancy Prediction for Autonomous Driving
- Workshop Camera-based perception for AV: From Data Collection to network Robustness
- Workshop Optimizing large deep models for real-time inference
- Seminar Efficient Deep Learning at Scale
- Industry Towards Robust Perception Systems in Open World
- Seminar Deep Learning at Scale
- Industry Scaling-up Deep Learning for Autonomous Driving
- Tutorial Scaling-up Deep Learning for Autonomous Driving
- Workshop Scaling up Deep Learning for Autonomous Driving
- Workshop Efficient ConvNets for Perception in Autonomous Driving
- Industry Efficient Deep Model Selection
- Industry Efficient Networks for Real Time Scene Understanding
- Industry Large-scale scene understanding in constrained platforms
- Workshop Compacting ConvNets for end to end learning
Interns
- Michael Hu – PhD, New York University.
- Simon Gerstenecker – MSc, ETH Zurich.
- Zhenxin Li – MSc, Fudan Univ.
- Jingyu Song – PhD, University of Michigan.
- Tolga Dimlioglu, PhD New York University.
- Zhenxin Li – MSc, Fudan Univ.
- Shihao Wang – MSc, Beijing Institute of Technology
- Zi Wang – MSc, Carnegie Mellon Univ.
- Sihan Liu – MSc, Fudan Univ.
- Shan Wang – PhD, Australian National Univ.
- Nikita Durasov – PhD, EPFL
- Jenny Schmalfuss – PhD, University of Stuttgart
- Chonghao Sima – PhD, Hong Kong Univ.
- Jihao Liu – PhD, The Chinese University of Hong Kong.
- Feiyang Kang – PhD, Virginia Tech Univ.
- Vibashan VS – PhD, John Hopkins University
- David Wang – PhD, Tsinghua Univ.
- Shixing Yu – PhD, Univ. Texas Austin
- Yanwei Li – PhD, CUHK
- Zhiqi Li – PhD, Nanjing Univ.
- Zetong Yang – PhD, CUHK
- Tzofi Klinghoffer – PhD, MIT
- Bingyin Zhao – PhD, Clemson Univ.
- Annamarie Bair – PhD, CMU
- Yilun Chen – PhD, CUHK
- Alex Sun – MS, Stanford
- Justin Hsu – BS, Stanford
- Arnav Joshi – BS, Stanford
- Shubhranshu Singh – MS, CMU
- Jiayu Yang – PhD, Australian National University
- Marc Finzi – PhD, NYU
- Enze Xie – PhD, HKU
- Ryan Humble – PhD, Stanford
- Vlad Sobal – PhD, NYU
- Jianna Liu – BS, MIT
- Xinlong Wang – PhD, Univ. Adelaide
- Daquan Zhou – PhD, NUS
- Faith Johnson – PhD, Rutgers
- Javier Sagastuy Brena – PhD, Stanford
- Joshua Chen – MS, CMU (now Nvidia)
- Nithya Attaluri – BS, MIT
- Jessica Lee – MS, CMU
- Jiwoong Choi PhD, Seoul Nat. University (now NVIDIA)
- Ismail Elezi PhD TUM (now PostDoc TUM)
- Shuxuan Guo - PhD. EPFL
- Marvin Kim – MS CMU (now Waymo)
- Lilian Luong – BS MIT
- Cynthia Liu – MS, MIT (now Cerebras)
- Nadine Chang – PhD CMU
- Yerlan Idelbayev – PhD UC Merced (now Amazon)
- Xinnan Du – MS CMU (now Goldman Sachs)
- Hoang Vu Nguyen – PhD Stony Brooks Univ.
- Maying Shen – MS CMU (now Nvidia)
- Michael Zhang – BS Hardvard (now PhD Stanford)
- Akshay Chawla – MS CMU (now Vicarious)
- Kashyap Chitta – MS CMU (now PhD student MPI)
- Tony Wang – BS MIT
- Wenbo Guo – PhD Penn. State Univ.
- Nikhil Murthy – BS MIT
- Jiaming Zeng – PhD Stanford (now IBM Research)
- Yousef Hindy – MS Stanford (now NewCo)
- Kashyap Chitta – MS CMU (now PhD student MPI)
Awards and Recognitions
3×1st place at CVPR challenges
2Innovation Awards
3Best Paper / Poster Awards
ERCand Ramón y Cajal fellowships
- 1st Place, End-to-End Driving Challenge CVPR2025
- Runner-up Award and Innovation Award, RealADSim Workshop Challenge ICCV2025
- 1st Place, End-to-End Autonomous Driving at Scale Challenge CVPR2024challenge site · with Hydra-MDP
- Innovation Award, End-to-End Autonomous Driving at Scale Challenge CVPR2024
- 2nd Place, Driving with Language Models Challenge CVPR2024
- 1st Place, 3D Occupancy Prediction Challenge CVPR2023challenge site · with NVOCC
- SegFormer ranked #3 among the Top-10 Most Influential Papers NeurIPS2021
- Outstanding Research Award, “ErfNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation” IEEE T-ITS2020
- Top Ten Research Papers 2010–2019, “Road Detection based on Illuminance-Invariance” IEEE T-ITS2010–2019
- Best Paper Award, “Efficient ConvNet for Real-time Semantic Segmentation” IEEE IV2017
- Best Paper Award, “Exploiting Sparsity for Real Time Video Labelling” CVPR Workshop on Computer Vision for Vehicle Technology2013
- Best Poster Award, “Synchronization of Video Sequences from Free-Moving Cameras” IbPRIA2007
- Ramón y Cajal Fellowship Spain2016
- ERC Fellow Grant European Research Council2011
- Extraordinary Doctorate Award Barcelona, Spain2010–2011
- Best PhD Award Barcelona, Spain2010
- Outstanding Reviewer
Academic Service
General Chair
IEEE Intelligent Vehicles Symposium (IV 2028)
- ICLR2027
- CVPR2027
- NeurIPS202620252024
- ICML2026
- IJCAI202620252022
- IEEE IV202620252024202320222021202020192018
- IEEE ITSC20262018
- PAKDD2026
- ECCV2024
- KDD2024
- IEEE WACV201820172016
- ACM MM2017
- IEEE ICRA2015
- Associate Editor, IEEE T-PAMI2024 – present
- Associate Editor, IEEE T-ITS2012 – present
- Guest Editor, IEEE TCSVT Special Issue on Spatial Intelligence2026
- Associate Editor, IEEE T-ITS Special Issue on Robust and Efficient Vision Techniques for Intelligent Vehicles2018
- Chair, IEEE ACT Computer Science Chapter2013
Relevant Tutorial Organization
Active
- Full-Stack, GPU-based Acceleration of Deep Learning CVPR202520242023
- Advancing Data Strategies for AI Success: Continuous Data Cycle via Foundation Models CVPR
Past
- Learning with Noisy and Unlabeled Data for Large Models beyond Categorization ICCV2023
- Scaling up Deep Learning for Autonomous Driving IEEE IV
Relevant Workshop Organization
Active
- World Models in Physical AI New NeurIPS
- Workshop on Autonomous Driving CVPR
- Exploring the Next Generation of Data CVPR
- Transformers for Vision and Multimodal AI CVPR
- Scalable 3D Scene Generation and Geometric Scene Understanding ECCV / ICCV
Past
- Physics Based Vision meets Deep Learning (PBDL) CVPR
- Beyond Supervised Learning IEEE IV
- Embodied AI Workshop CVPR
- Deep-Vision Workshop CVPR
- Computer Vision for Road Scene Understanding ECCV / ICCV20182017201620152014
- Computer Vision in Vehicle Technology CVPR2018201720162015
- Color in Computer Vision ECCV / ICCV20172016201520142013
- Deep-Driving IEEE IV2017201620152014