Lyric App Framework
A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

2020-2026
CHI '23

Abstract

We propose a "lyric app," a novel media format of interactive lyric-driven visual art. We developed a framework to support its development, analyzed 52 contest submissions created in the wild in 2020–2021, and gained insights into the future of creative culture and the intersection of music and programming.

The programming contests have continued since the paper was published. A 2025 Japanese journal paper reported an analysis of four years of submissions. In 2026, the number of submissions increased substantially compared with the previous year.

Programming Contests

We made the framework publicly available in 2020 and have been conducting an annual programming contest in "Hatsune Miku Magical Mirai" — a live show and exhibition, connecting with Hatsune Miku through creativity.

Presentation materials

CHI 2023 Video Preview (30s)

CHI 2023 Pre-recorded Presentation Video

Publications

Cite this!2023CHI '23

Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

Jun Kato, Masataka Goto
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
AIST Best Paper Award 2025TAF Award 2025 (Encouragement Prize)ACM CHI '23 Honorable Mention Award
Copy to the clipboard

Revision history

11/13/2026
Updated links to "Hatsune Miku Magical Mirai" programming contest websites.
04/20/2026
Added project hero image and updated project text and 2026 thumbnail
03/29/2026
Updated links to "Hatsune Miku Magical Mirai" programming contest websites.
09/02/2023
Added links to "Hatsune Miku Magical Mirai" programming contest websites.
04/19/2023
Added a pre-recorded presentation video
04/16/2023
Added a video preview
03/31/2023
Added a project page

Relevant projects

TextAlive

An integrated design environment for kinetic typography; desktop app revamped as a web service in 2015.

Creativity Support Environments

To realize a creative society in which people want to create content with the help of AI technologies, a sustainable ecosystem in which content inspires the next creation serves a critical role.

I shed light on the creative culture in such an ecosystem, build a "creativity support environment (CSE)" that supports the content creation and distribution process, and aim to establish Human-Computer Interaction (HCI) technologies that engineer, nurture, and maintain the creative culture.

2022-2026CHI EA '23