A Toolkit for Prototyping Robotic Things

DIS '12
Keywords: Toolkit, Physical, Robot
Phybots is open-source at GitHub and can be downloaded at


There are many toolkits for physical UIs, but most physical UI applications are not locomotive. When the programmer wants to make things move around in the environment, he faces difficulty related to robotics. Toolkits for robot programming, unfortunately, are usually not as accessible as those for building physical UIs.

To address this interdisciplinary issue, we propose Phybots, a toolkit that allows researchers and interaction designers to rapidly prototype applications with locomotive robotic things. The contributions of this research are the combination of a hardware setup, software API, its underlying architecture and a graphical runtime debug tool that supports the whole prototyping activity.

The paper introduces the toolkit, applications and lessons learned from three user studies.

Overview of hardware setup for Phybots

Presentation materials

Example applications

DIS 2012 Slides

Phybots Toolkit

Toolkits for:Physical UIsPhybotsRobots
Target users:HCI researchers & interaction designersRobotics people
Software APIs:Low-level and staticHigh-level and extensible
Hardware size & cost:Small & cheap (Phidget Kit $200, Ikimo Robot $160)Medium to large & expensive ($700-)
Comparison between toolkits

APIs for 2D Locomotion

Phybots assumes a simple and inexpensive hardware setup of a camera looking down on the floor and visual markers attached to the top of physical objects. This hardware setup can easily be deployed to everyday spaces such as a working desk, a dining table or in a kitchen or living room.

Given the hardware setup, Phybots provides APIs for two-dimensional localization and locomotion of floor-based robotic things. These APIs are similar to those for graphical applications in which object positions are defined by screen coordinates, and support moving to a goal, pushing an object toward a goal, tracing a path, and other custom behavior by specifying a vector field.

Centralized and Extensible Architecture

Phybots APIs are built on top of a centralized and extensible software architecture that runs on one personal computer and manages all physical objects of interest.

The programmer has direct access to these objects including cameras, robotic things, and entities with visual markers as instances of the Camera, Robot and Entity interfaces, respectively. Time-consuming tasks including locomotion APIs are represented by Task interface.

The toolkit can be easily extended on top of these software stacks to support other types of sensors, roobts, and tasks. In addition, a Workflow class represents a directed graph consisting of a set of Robot and Task instances, and enables higher-level task management such as handling multiple tasks by one robot in serial or by two robots in parallel.

Runtime Debug Tool

Phybots provides a runtime debug tool that supports a test phase, beyond the programming phase, as an essential part of the prototyping process.

In the robot application test phase, when the robot fails to accomplish its task, the user usually has to order the same task to be performed again. Because it is too costly to restart a test run for each error, the programmer often wants to fix the error right away without restarting.

To support this workflow, a graphical runtime debug tool enables API parameter reconfiguration, restart, pause, or resume of the interaction at run-time. For example, when the robotic thing fails to move to a desired destination, the programmer can change the threshold of the marker detection algorithm by dragging a slider and then give the robot another try.

Overview of the Phybots runtime debug tool


Cite this!2012DIS '12

Phybots: A Toolkit for Making Robotic Things

Jun Kato, Daisuke Sakamoto, Takeo Igarashi
Proceedings of the 9th Conference on Designing Interactive Systems, pp.248-257
Copy to the clipboard

A Toolkit for Easy Development of Mobile Robot Applications with Visual Markers and a Ceiling Camera

2009UIST '09 Demo
Daisuke Sakamoto, Jun Kato, Masahiko Inami, Takeo Igarashi
The 22nd Annual ACM Symposium on User Interface Software and Technology