Deheng Zhang

Deheng Zhang

Doctoral researcher at INSAIT, Sofia
with Prof. Luc Van Gool and Dr. Danda PaudelINSAIT（索菲亚）博士研究员
导师：Luc Van Gool 教授、Danda Paudel 博士

I want machines to learn a persistent, explicit 3D world from video, and to use it to reason and act in space.我希望让机器从视频中学到一个持久、显式的三维世界，并用它进行空间推理与行动。

  1. City University of Hong Kong香港城市大学
    BSc
  2. ETH Zürich苏黎世联邦理工学院（ETH Zürich）
    MSc
    Disney Research | Studios, Dec 2022 – May 2023: controllable 3D style transfer, CoARF (3DV 2024).
    VLG with Prof. Siyu Tang, Sep 2023 – Apr 2024: glossy-object inverse rendering, RISE-SDF (3DV 2025).    Disney Research | Studios，2022.12 – 2023.05：可控 3D 风格迁移 CoARF（3DV 2024）。
    VLG（Siyu Tang 教授），2023.09 – 2024.04：高光物体逆渲染 RISE-SDF（3DV 2025）。
  3. Max Planck Institute for Intelligent Systems, Tübingen马普智能系统研究所（图宾根）
    IMPRS-IS · Sep 2024 – Mar 2025
    3D scene understanding and Gaussian splatting.三维场景理解与 Gaussian Splatting。
  4. INSAIT, SofiaINSAIT（索菲亚）
    Doctoral researcher · Apr 2025 – now博士研究员 · 2025.04 至今
    Egocentric and spatial understanding, video world models, Gaussian ray tracing: EgoNight (ICLR 2026), StateSpaceDiffuser (NeurIPS 2025), ConeGaussian, PhiRIE.第一人称与空间理解、视频世界模型、Gaussian 光线追踪：EgoNight（ICLR 2026）、StateSpaceDiffuser（NeurIPS 2025）、ConeGaussian、PhiRIE。

→ / Space: next ← back L: EN / 中 F: full screen O: contents→ / 空格：下一页　← 上一页　L：中 / EN　F：全屏　O：目录

Two first-author projects两个一作项目

EgoNight: the same egocentric scene by day and by night

EgoNight

ICLR 2026

Can multimodal LLMs see in the dark?多模态大模型能在黑暗中看清吗？

The first benchmark for nighttime egocentric vision. Day–night aligned videos let the same question be asked under both lighting conditions, which improves annotation quality and isolates the effect of darkness. State-of-the-art MLLMs drop significantly from day to night.第一个夜间第一人称视觉 benchmark。昼夜对齐的视频让同一个问题可以在两种光照下提问，既提高了标注质量，也把“黑暗”这一因素单独隔离出来。最强的 MLLM 从白天到夜晚性能显著下降。

3,658 QA pairs3,658 个问答90 videos90 段视频12 question types12 类问题day–night aligned昼夜对齐
Project page ↗项目主页 ↗ Slides ↗幻灯片 ↗ Paper ↗论文 ↗ Dataset ↗数据集 ↗
ConeGaussian: zoom-out without aliasing compared with 3DGEER

ConeGaussian

in submission投稿中

A pixel sees a bundle of rays, not a single ray.一个像素看到的是一束光线，而不是一条。

Anti-aliasing for Gaussian ray tracing on any calibrated central camera. The pixel footprint is built from the camera's own neighbouring rays and the ray–Gaussian response is filtered in closed form, so one filter serves pinhole and strongly distorted fisheye cameras.面向任意标定中心相机的 Gaussian 光线追踪抗锯齿。用相机自身的相邻光线构造像素 footprint，并以闭式形式滤波光线–Gaussian 响应，同一个滤波器同时适用于针孔和强畸变鱼眼相机。

anti-aliased zoom-out缩小抗锯齿artifact-free zoom-in放大无伪影pinhole + fisheye针孔 + 鱼眼3DGEER and 3DGUT backbones3DGEER 与 3DGUT 两种主干
Project page ↗项目主页 ↗ Slides ↗幻灯片 ↗ arXiv ↗

StateSpaceDiffuser: long-term memory for diffusion world modelsStateSpaceDiffuser：给扩散世界模型加上长期记忆

NeurIPS 2025  Nedko Savov, Naser Kazemi, Deheng Zhang, Danda Pani Paudel, Xi Wang, Luc Van Gool

Diffusion world models predict the next frame from only the last few frames. Once a place leaves that window, it is forgotten, and returning to it produces a different scene. StateSpaceDiffuser adds a Mamba state-space branch that summarises the whole interaction history and conditions the diffusion model on it.扩散世界模型只根据最近几帧预测下一帧。一个地方一旦离开这个窗口就被遗忘，回到原处时会生成一个不同的场景。StateSpaceDiffuser 加入一个 Mamba 状态空间分支，概括全部交互历史，并用它来条件化扩散模型。

Recall example: diffusion-only rollout loses the scene, StateSpaceDiffuser recovers it
39.68 vs 26.13

average PSNR on MiniGrid recall with a 50-frame context, against the diffusion-only DIAMOND baseline (+51.9%). Human raters also prefer our CS:GO rollouts for long-context consistency.MiniGrid 回忆任务（50 帧上下文）的平均 PSNR，对比纯扩散基线 DIAMOND（+51.9%）。在 CS:GO 上，用户研究也更偏好我们的长程一致性。

What I took from it: the state is compressed into 256 dimensions and loses detail over long rollouts. A world model needs a persistent state, and I think the most useful one is explicit 3D.我的收获：这个状态被压缩到 256 维，长时间 rollout 会丢细节。世界模型需要一个持久的状态，而我认为最有用的状态是显式的 3D。

World models: learn the world from video, make it explicit in 3D世界模型：从视频中学习世界，再用 3D 把它显式化

Video data视频数据 the largest record of how规模最大的世界记录： the world looks and moves外观、运动、交互 Video diffusion model视频扩散模型 learns geometry, occlusion隐式学到几何、遮挡 and dynamics, implicitly和动态规律 Dense, structured 3D稠密、结构化的 3D explicit, persistent, metric:显式、持久、有尺度： objects, parts, joints物体、部件、关节 Spatial agents空间智能体 VLMs that reason in space,做空间推理的 VLM， VLAs and robots that act执行动作的 VLA 与机器人 scale规模 1 Extract: video → mesh提取：video → mesh 2 Diagnose: where does it break?诊断：到底卡在哪一步？ 3 Use: 3D for spatial skills使用：用 3D 提升空间能力 3D consistency trains and tests video models3D 一致性反过来训练、评测视频模型

Why start from video为什么从视频出发

  • Video is by far the most abundant data about the physical world, and it needs no 3D labels.视频是关于物理世界最丰富的数据，而且不需要 3D 标注。
  • To generate consistent video, a model has to capture parallax, occlusion, object permanence and plausible motion, at least partly.要生成一致的视频，模型必须（至少部分地）掌握视差、遮挡、物体恒存和合理的运动。

Why make it explicit in 3D为什么要显式成 3D

  • Pixels carry no metric scale, no object identity and no guarantee that the world persists (the StateSpaceDiffuser lesson).像素里没有尺度、没有物体身份，也不能保证世界是持久的（StateSpaceDiffuser 的教训）。
  • An agent needs something it can query and act on: distances, free space, contacts, what opens and how.智能体需要可查询、可操作的东西：距离、可通行空间、接触，以及什么能打开、怎么打开。

Points and Gaussians are samples of a scene, not its surfaces点云和高斯只是场景的采样，而不是它的表面

Pick a representation to redraw the scene.点击表头切换表征，左边的场景随之重画。

● native原生支持   ◐ with extra fitting or labels需额外拟合或标注   ○ not available不支持

From video to mesh: three routes, one open question从视频到 mesh：三条路线，一个开放问题

A

Reconstruct, then extract a surface先重建，再提取表面

video→pointmaps / depth→fusion→mesh

e.g. DUSt3R, VGGT, then TSDF, Poisson or 2DGS例：DUSt3R、VGGT，再接 TSDF、Poisson 或 2DGS

  • Anchored to the pixels, metric up to scale锚定在像素上，尺度可恢复
  • Only what was seen; noisy thin structures; one fused piece with no objects只覆盖看到的部分；细结构噪声大；融成一整块，没有物体
B

Generate natively in 3D直接在 3D 中生成

image(s)→3D latent→mesh

e.g. TRELLIS, Hunyuan3D例：TRELLIS、Hunyuan3D

  • Complete, clean, watertight shapes, hidden sides included形状完整、干净、封闭，包括看不到的一面
  • Mostly single objects from a few images; scenes, metric alignment and video input remain open多为少量图像下的单个物体；场景级、尺度对齐和视频输入仍未解决
C

Use the video model as a geometry prior把视频模型当作几何先验

video diffusion→depth / point videos→surface

e.g. DepthCrafter, GeometryCrafter, Aether例：DepthCrafter、GeometryCrafter、Aether

  • Temporally consistent geometry, with a prior learned from very large video data时间一致的几何，先验来自海量视频
  • Still 2.5D per frame; a surface and its structure must be recovered afterwards仍然是逐帧 2.5D；表面和结构需要事后恢复
My bet我的设想

Let observed pixels anchor what was seen, let the video prior generate what was not, and output an object-structured surface directly instead of fusing per-frame depth.让观测到的像素锚定看到的部分，让视频先验生成没看到的部分，并直接输出按物体组织的表面，而不是融合逐帧深度。

Test: does a video diffusion backbone trained to emit surface latents beat reconstruct-then-mesh on unseen regions and thin structures?检验：一个训练来输出表面 latent 的视频扩散主干，能否在未观测区域和细结构上胜过“先重建再提 mesh”？

Where is the boundary today?现在模型的边界到底在哪里？

Two hypotheses that call for different research. Click a stage to see what breaks there and how to test it.两个假设对应完全不同的研究方向。点击每个环节，看它会出什么问题、怎么检验。

H1Lifting is the bottleneck: video → 3D is not yet consistent瓶颈在“抬升”：video → 3D 这一步还不一致

H2Granularity: geometry is fine, structure is missing粒度不够：几何没问题，但缺结构

What breaks here这里会坏在哪

How to test it怎么检验

Oracle ladder. Replace one stage at a time with ground truth and measure the downstream gain. The stage whose oracle helps most is the current boundary.Oracle 阶梯：每次只把一个环节替换成真值，测下游任务的提升。哪个环节的 oracle 带来的提升最大，哪里就是当前的边界。

Pipeline as is现有流程baseline
→
+ true poses and depth+ 真值位姿与深度Δ task = ?
→
+ true mesh+ 真值 meshΔ task = ?
→
+ true instances+ 真值实例Δ task = ?
→
+ true parts and joints+ 真值部件与关节Δ task = ?

If the 3D is good enough: using it to teach spatial understanding如果 3D 足够好：用它来教会模型空间理解

VLMs

3D as a data engine3D 作为数据引擎

Render any viewpoint or trajectory of a reconstructed scene and compute answers from geometry: distance, size, direction, counting, routes. Labels are exact, difficulty is controllable, and an answer can be checked to stay the same across trajectories of the same scene.从重建的场景中渲染任意视角和轨迹，用几何直接计算答案：距离、尺寸、方向、计数、路径。标签精确，难度可控，还能检验同一场景不同轨迹下答案是否一致。

VLAs · robots

3D as a simulator3D 作为仿真器

Object-level scenes with collision and articulation turn captured places into environments where policies can be trained and evaluated, the real-to-sim direction we explore in PhiRIE. Drawers and doors only matter if they can open.带碰撞和铰接的物体级场景，可以把真实采集的地方变成训练和评测策略的环境，这正是我们在 PhiRIE 中探索的 real-to-sim 方向。抽屉和门只有能打开才有意义。

world models · agents世界模型 · 智能体

3D as persistent memory3D 作为持久记忆

Give a model an explicit scene state that it can query instead of re-perceiving every frame. This is the persistent state that StateSpaceDiffuser approximated with a compressed vector, made metric and inspectable.给模型一个可以查询的显式场景状态，而不是每一帧重新感知。这正是 StateSpaceDiffuser 用一个压缩向量近似的持久状态，只是变成了有尺度、可检查的形式。

The question I want to measure, not assume: which 3D fidelity does each consumer need? Metric geometry for measurement questions, object structure for counting and relations, articulation for manipulation.我想测量而不是假设的问题：不同的下游需要什么程度的 3D 保真？测量类问题需要尺度几何，计数与关系需要物体结构，操作需要铰接信息。

How my work connects to this agenda我的工作如何连到这条路线上

Video & world models视频与世界模型

how the world evolves over time世界如何随时间演化

StateSpaceDiffuserNeurIPS 2025
FrameWorkersin submission投稿中

3D representation & rendering3D 表征与渲染

where surfaces are and how they look表面在哪里、长什么样

ConeGaussianin submission投稿中
RISE-SDF3DV 2025
CoARF3DV 2024
SeasonScapesCVPR-W 2026
LangStreetin submission投稿中

Structured, interactive 3D结构化、可交互的 3D

what can be separated and moved什么能被分开、被移动

PhiRIEin submission投稿中

Spatial understanding空间理解与具身

what agents ask about and do智能体问什么、做什么

EgoNightICLR 2026
Kairosin submission投稿中
EgoSpotICRA-W 2026
Thank you谢谢
dehezhang2.github.io ↗ Google Scholar ↗ GitHub ↗ deheng.zhang@insait.ai
1 / 9