Integrated Support for Developing Interactive Camera-Based Programs

UIST '12
Keywords: PwE, Live, IDE, NUI
For those interested in this IDE: Unfortunately, there's no short term plan for the distribution. You can still use Kinect Studio that partially provides similar features (recording and replaying data stream) though without tight integration with the IDE.


The increasing popularity of interactive camera-based programs highlights the inadequacies of conventional IDEs in developing these programs given their distinctive attributes and workflows. We present DejaVu, an IDE enhancement that eases the development of these programs by enabling programmers to visually and continuously monitor program data in consistency with the frame-based pipeline of computer-vision programs; and to easily record, review, and reprocess temporal data to iteratively improve the processing of non-reproducible camera input. DejaVu was positively received by three experienced programmers of interactive camera-based programs in our preliminary user trial.

This research was done during my internship at Microsoft Research Asia HCI Group. My mentors were Xiang Cao and Sean McDirmid.

Presentation materials

UIST 2012 Demo Video

UIST 2012 Slides

DejaVu IDE

We added two interlinked components to a general purpose development environment.




  • Visualizing various variable contents by drag-and-drop
  • Continuously showing camera input, variable contents, window output
  • Sketching capability


  • Auto-recording, replaying (input streams) & refreshing (variables) to handle non-reproducible input
  • Contents synchronized with canvas


Cite this!2012UIST '12

DejaVu: Integrated Support for Developing Interactive Camera-Based Programs

Jun Kato, Sean McDirmid, Xiang Cao
Proceedings of the 25th Annual ACM symposium on User Interface Software and Technology, pp.189-196
Copy to the clipboard