stu and Annex Digital Japan Jointly Develop "MaskPilot AI" for Professional Use, Implemented in "Kamen Rider Series" Special Effects TV Drama

2024.05.13

Release

stu Inc. and Annex Digital Japan Co., Ltd. have jointly developed “MaskPilot AI,” an AI mask generation system. After validating its effectiveness in various use cases, the system will be implemented in the masking process for the special effects TV drama “Kamen Rider Series.”

In recent years, AI development has been driving efficiency and innovation across various industries. Within this trend, stu aims to digitally transform video production workflows using AI and provide new experiences.

As part of this effort, in November 2023, stu developed “Scene Tracking Mask,” an AI mask generation technology that meets the needs of video production sites after thorough examination of video production workflows.

This time, Annex Digital Japan Co., Ltd., which handles numerous masking operations in Japan’s video production industry, has joined this project to jointly develop “MaskPilot AI,” an AI mask generation system for professional use.

■ Recent Challenges in AI Mask Technology and “MaskPilot AI”

During the joint development with Annex Digital Japan, which has extensive experience in masking operations, the biggest challenge identified in AI mask technology was data handling. Many of the powerful AI tools that have emerged recently operate on servers with ambiguous or undisclosed data handling policies, raising many concerns when dealing with unreleased data or intellectual property with high confidentiality. As a result, AI technology could not be fully utilized for masking in practical settings.

Considering these circumstances, “MaskPilot AI” was developed to ensure data security while aiming to create a system comparable to recent advanced AI tools.

Ensuring Data Security with On-Premise Server Support
“MaskPilot AI” is designed to operate on local on-premise servers (local servers that do not use cloud services). This ensures data security while maximizing the use of AI.

Automatic Rough Mask Generation with Intuitive Operation “MaskPilot AI” allows users to select objects for masking through intuitive UI operations and automatically generates rough masks. By combining multiple object selection features and automatic tracking of selected objects, it aims to significantly reduce the effort required for rough masking.

To verify the effectiveness of this system, we compared the total work time for manual operations, using this system, and using other AI tools in various cases. The comparison results showed that compared to fully manual masking, using this system reduced the total work time by an average of 40%, while other AI tools reduced it by an average of 38%. This demonstrated its effectiveness, achieving results comparable to existing powerful AI tools despite its on-premise server configuration.

■ Future Prospects for “MaskPilot AI”

With its effectiveness confirmed through verification, “MaskPilot AI” will now enter the operational phase. It will be implemented in the masking work for the special effects TV drama “Kamen Rider Series,” further improving the system’s usability and rough mask generation performance. Currently, it only generates rough masks automatically, but in the future, stu and Annex Digital Japan will continue joint development with the aim of automatically generating high-definition masks that could only be achieved manually.

■ stu’s AI and Machine Learning R&D

stu is engaged in AI and machine learning R&D in the entertainment sector, aiming to digitally transform video production processes and provide new AI-powered experiences.

stu is constantly exploring new possibilities with AI and will continue to actively engage in providing technology to companies and various projects. We are seeking entertainment companies and local governments interested in full-scale development and proof-of-concept experiments. Please feel free to contact us.

Some examples of AI utilization so far:

Proof of concept for “3D Gaussian Splatting” for easy 3D space construction Started proof of concept for 3D Gaussian Splatting to facilitate digital twin creation, making it easily accessible through web browsers when combined with WebGL.

“SyncFlow AI” workflow optimization system for generative AI Developed “SyncFlow AI,” a system to enhance efficiency and expand the range of expression in video production using generative AI, provided video for the “Hyakunengo Art Festival – Uchiboso Art Festival -“

“Scene Tracking Mask” automatic mask generation and scene detection technology Utilized “Scene Tracking Mask,” a system with automatic mask generation and scene detection technologies for captured video, to provide technology for the short film “TONOMURA” by filmmaker Kensaku Kakimoto

Real-time participatory virtual live system “LAIV” Developed “LAIV,” a participatory virtual system where real-time direction changes based on participants’ comments using generative AI, providing a new VR live experience enabling interactive direction between artists and participants

About stu Inc.

stu is a cutting-edge creative technology company on a mission to reinvent entertainment. Headquartered in Shibuya, Tokyo, and led by CEO Takayasu Kuroda, stu stands at the intersection of advanced tech and immersive experiences, blending 5G, XR, and AI with innovative visual and experiential design. Our borderless creative team pushes the boundaries of what’s possible, developing next-generation entertainment content that captivates audiences.

From our edge tech initiatives in the metaverse to our content development featuring world-class production workflows like writers’ rooms and pre-visualization, stu is pioneering the future of entertainment. We collaborate with industry leaders and government bodies to bring visionary ideas to life, setting new standards in the digital age.

Founded in 2017, stu continues to grow rapidly, attracting top talent and partnering with global brands to create unforgettable experiences. Our work spans virtual productions, live events, interactive installations, and cutting-edge digital platforms, always with an eye toward innovation and audience engagement.

Join us as we shape the next era of entertainment.

Contact: contact@stu.inc
Website: stu.inc

X (formerly Twitter): twitter.com/stuinc_

About Annex Digital Japan Co., Ltd.

Annex Digital Japan, headquartered in Minato, Tokyo, and led by CEO Teppei Wakabayashi, engages in various video-related services, including live-action video production, CG production, digital signage, and XR content creation. The company uniquely conducts planning and sales at its Tokyo headquarters, while production is carried out at its subsidiary studios worldwide, primarily in China, Cambodia, and Poland.

Contact: contact@annex-digital.com
Website: annex-digital.com/