Lyric App Framework

We propose a "lyric app," a novel media format of interactive lyric-driven visual art. We developed a framework to support its development, collected 52 examples in the wild, and gained insights into the future of creative culture and the intersection of music and programming.

The programming contests continue to be held even after the paper was published, with a 2025 Japanese journal paper reporting analysis results from four years, and the 2026 contest is ongoing.

We made the framework publicly available in 2020 and has been conducting an annual programming contest in "Hatsune Miku Magical Mirai" — a live show and exhibition, connecting with Hatsune Miku through creativity.

Where?	Room Y07 + Y08, Congress Center Hamburg (CCH)
When?	Thu, Apr 27 \| 09:00 - 09:14
CHI '23 Program	Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications - CHI '23

CHI 2023 Video Preview (30s)

CHI 2023 Pre-recorded Presentation Video

Cite this!2023CHI '23

Jun Kato, Masataka Goto

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

ACM CHI '23 Honorable Mention Award

第40回電気通信普及財団賞（テレコム学際研究賞奨励賞）

産総研論文賞2025（優秀賞）

Slides PDF Resolve DOI

BibTeX:

@inproceedings{chi2023-kato-lyric-app-framework,
  title = {Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications},
  author = {Kato, Jun and Goto, Masataka},
  year = {2023},
  booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
  location = {Hamburg, Germany},
  publisher = {ACM},
  address = {New York, NY, USA},
  series = {CHI '23},
  doi = {10.1145/3544548.3580931},
  isbn = {9781450394215},
  url = {https://doi.org/10.1145/3544548.3580931},
  abstract = {Lyric videos have become a popular medium to convey lyrical content to listeners, but they present the same content whenever they are played and cannot adapt to listeners’ preferences. Lyric apps, as we name them, are a new form of lyric-driven visual art that can render different lyrical content depending on user interaction and address the limitations of static media. To open up this novel design space for programmers and musicians, we present Lyric App Framework, a web-based framework for building interactive graphical applications that play musical pieces and show lyrics synchronized with playback. We designed the framework to provide a streamlined development experience for building production-ready lyric apps with creative coding libraries of choice. We held programming contests twice and collected 52 examples of lyric apps, enabling us to reveal eight representative categories, confirm the framework’s effectiveness, and report lessons learned.},
  articleno = {124},
  numpages = {18},
  keywords = {toolkit, multimedia control, music synchronization},
  slides = {https://junkato.jp/publications/chi2023-kato-lyric-apps-slides.pdf},
  project = {lyric-app-framework},
  addition = {ACM CHI '23 Honorable Mention Award; 第40回電気通信普及財団賞（テレコム学際研究賞 奨励賞）; 産総研 論文賞2025（優秀賞） },
  funding = {JST CREST JPMJCR20D4 and JST ACCEL JPMJAC1602 and JST ACT-X JPMJAX22A3}
}

Copy to the clipboard

04/20/2026

Added project hero image and updated project text and 2026 thumbnail

03/29/2026

Updated links to "Hatsune Miku Magical Mirai" programming contest websites.

09/02/2023

Added links to "Hatsune Miku Magical Mirai" programming contest websites.

04/19/2023

Added a pre-recorded presentation video

04/16/2023

Added a video preview

03/31/2023

Added a project page

An integrated design environment for kinetic typography; desktop app revamped as a web service in 2015.

2014-2026CHI '15Creativity Music PwE Communication Live IDE Design

To realize a creative society in which people want to create content with the help of AI technologies, a sustainable ecosystem in which content inspires the next creation serves a critical role.

I shed light on the creative culture in such an ecosystem, build a "creativity support environment (CSE)" that supports the content creation and distribution process, and aim to establish Human-Computer Interaction (HCI) technologies that engineer, nurture, and maintain the creative culture.

2022-2026CHI EA '23Creativity

Lyric App Framework
A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

Abstract

Programming Contests

Presentation materials

Publications

Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

Revision history

Relevant projects

Lyric App FrameworkA Web-based Framework for Developing Interactive Lyric-driven Musical Applications