Editing Reality: In-Situ Co-Creation with Generative AI in Mixed Reality

Editing Reality: steps for sketching and generating objects in 3D space

Editing Reality is a mixed reality system that enables people to co-create with generative AI directly inside physical environments. Using speech, sketching, and direct manipulation, users can generate, reconstruct, erase, retexture, and iteratively modify real-anchored virtual and reconstructed scene elements in place.

ACME LabÌý

Diagram of the Editing Reality pipeline, showing user actions, multimodal prompts, AI processing, and generated mixed reality results.

Diagram of the Editing Reality pipeline, showing user actions, multimodal prompts, AI processing, and generated mixed reality results.

Editing Reality explores how generative AI can become part of an embodied, situated creative process in mixed reality. Instead of treating generation as a one-shot command on a screen, the system allows users to work inside the room they are modifying: they can speak or type prompts, sketch 3D forms, select objects, preview changes, and revise generated results directly in context.

The system supports in-situ generation, reconstruction, erasing, retexturing, sketch-based modification, iteration, and multi-user collaboration. It is designed for early-stage spatial ideation and situated prototyping, where designers can rapidly explore furniture, material changes, object transformations, and speculative modifications at 1:1 scale within the target environment.

Diagram showing the client-server pipeline, AI models, Quest headset users, and shared mixed reality session.

Diagram showing the client-server pipeline, AI models, Quest headset users, and shared mixed reality session.

Through iterative system development, a formative workshop, and expert review, the project frames generative MR authoring as a negotiated, spatial, and temporal process. The work highlights how users negotiate intent with the system, accumulate design history in space, evaluate results through bodily movement, experience waiting at the site of change, and gradually learn the AI system as a spatial actor.

A ten-step mixed reality workflow showing how Editing Reality generates a coffee table, reconstructs and retextures a couch, modifies it through sketch and prompt, and generates a floor lamp in a physical room.

A ten-step mixed reality workflow showing how Editing Reality generates a coffee table, reconstructs and retextures a couch, modifies it through sketch and prompt, and generates a floor lamp in a physical room.

Associated Researchers

Additional Researcher

Pritalee Kadam, Things Lab, University of Colorado Boulder
Amy Banić, Interactive Realities Lab, University of Wyoming

Publications

Suibi Che-Chuan Weng, Shih-Yu Ma, Sawyer Reinig, Pritalee Kadam, Ada Yi Zhao, Amy Banić, Ryo Suzuki, and Ellen Yi-Luen Do. 2026. "". In: Designing Interactive Systems Conference (DIS ’26). (Singapore, June 13–17, 2026).

Suibi Che-Chuan Weng, Yan-Ming Chiou andÌýEllen Yi-Luen Do. 2024. ""ÌýIn:Ìý2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR).Ìý(January 17-19, 2024).Ìý