Making Digital Twins More Accessible with AI×WebGL Starting a Proof of Concept Experiment Using 3D Gaussian Splatting
Tokyo (Feb 28, 2024) – stu (headquarters: Shibuya, Tokyo, CEO: Takayasu Kuroda) has started proof-of-concept experiments on “3D Gaussian Splatting,” an advanced AI-based photogrammetry technology. This innovation facilitates the creation of intricate textures previously challenging with conventional techniques, now accessible via web browsers and similar platforms integrated with WebGL.
A composite image of a point cloud at an intermediate stage of production (left) and a space captured using 3D Gaussian Splatting (right).
With AI technology progressing, its application spans diverse domains such as language processing, image/video comprehension, and audio analysis, fueling efficiency in previously unattainable creative tasks and fostering novel approaches.
In line with this AI-driven era, stu has undertaken a demonstration experiment leveraging “3D Gaussian Splatting,” an image analysis AI, to explore digital twin capabilities further.
■ Background
stu’s initiative aims to propel digital twin technology forward, leveraging “3D Gaussian Splatting” to craft photorealistic virtual replicas of real-world spaces. Building upon a track record in photogrammetry-based digital twins, stu’s experiment further evolves this technology through AI integration.
■Unique Real-time Renderer
Alongside, stu developed a proprietary real-time renderer tailored for this technology, enhancing accessibility through a WebGL-based viewer. This allows easy construction of 3D scenes from everyday videos, enabling seamless navigation from any web browser.
▼Overlooking a 3D Scene from a Video Using 3D Gaussian Splatting
■ Advantages of 3D Gaussian Splatting
This cutting-edge technology, introduced in 2023, outperforms conventional methods such as photogrammetry and NeRF (Neural Radiance Fields) by excelling in reflective texture representation and minimizing rendering loads.
■Utilization
Leveraging “3D Gaussian Splatting,” users can effortlessly transform real-world spaces into immersive 3D scenes, catering to various applications:
- Integration of captured data into the metaverse for realistic virtual environments.
- Virtual location scouting, eliminating the need for physical visits.
- Commercialization of 3D scenes as digital collectibles for enhanced fan engagement.
撮影イメージ
■Future prospects
This technology shows potential for evolving into “4D Gaussian Splatting,” adding a temporal dimension into its 3D counterpart. Unlike current technologies like 360-degree videos, this innovation offers a dynamic video experience, allowing viewers to navigate freely through changing environments from any perspective.
For instance, leveraging this advancement, artists’ “virtual digital figures” can be transformed into “spatial videos,” introducing novel experiences like “virtual live figures” performing songs. This enables fans to enjoy live performances from their favorite angle. stu’s extensive expertise in photogrammetry facilitates seamless integration of various 3D technologies into “3D Gaussian Splatting.”
This includes crafting music videos with added visual effects, pre-visualization, live productions, and immersive events in virtual realms enhanced with expressive character animations. Beyond entertainment, this technology extends to practical applications such as virtual product catalogs, online stores, and digital museum replicas, enriching user experiences across diverse fields.
■AI & Machine Learning Ventures
stu is dedicated to advancing AI and machine learning in the entertainment sector, with a focus on digitizing video production processes (DX) and offering innovative AI-driven experiences. Continuously pushing the boundaries, stu actively seeks collaborations with entertainment companies and municipalities for comprehensive development and demonstration projects. Interested parties are encouraged to reach out for partnership opportunities.
■Previous AI Applications
stu’s portfolio includes:
・”LAIV”: A real-time participatory virtual live system.
・”Scene Tracking Mask”: Automatic mask generation & scene detection technology.
Utilized in Kensaku Kakimoto’s short film “Tonomura”
・”SyncFlow AI”: Enhancing video production efficiency through generative AI.
Utilized to produce video content for the “Centennial Art Festival – Uchi Boso Art Fest -.”
■About stu
stu is a pioneering company dedicated to “reinventing entertainment,” driven by a creative team fostering boundary-breaking ideas. By integrating cutting-edge technologies in 5G, XR, and the metaverse with innovative content development practices like writers’ rooms and previsualization, stu is shaping a new frontier in entertainment. This convergence aims to establish global standards in video production workflows, heralding a transformative era in the entertainment industry.
Contact: contact@stu.inc
Official Website: https://stu.inc/
Official X (formerly Twitter): https://twitter.com/stuinc_