arxiv:2503.15265

DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning

Published on Mar 19

· Submitted by

zzzrw on Mar 20

#3 Paper of the day

Upvote

Authors:

Ruowen Zhao ,

Junliang Ye ,

Zhengyi Wang ,

Guangce Liu ,

Yiwen Chen ,

Yikai Wang ,

Abstract

Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/

View arXiv page View PDF Project page GitHub repository Add to collection

Community

zzzrw

Paper author Paper submitter 1 day ago

Abstract:
Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality.

Project page: https://zhaorw02.github.io/DeepMesh/
Code: https://github.com/zhaorw02/DeepMesh
Huggingface: https://huggingface.co/zzzrw/DeepMesh

HAKKYU

1 day ago

•

edited 1 day ago

Instead of having a human select a preference at the DPO stage, it would be more efficient to have more than two mesh structures, generate 3d rendered images for random cameras and lighting conditions (use shader which can amplify shading artifacts), and apply GRPO to select which one is more close to the rendered image of the original mesh in the dataset.