Computer scientists at the University of Toronto have developed an interactive camera robot that is intended to make it easier to shoot instructional videos. The robot called Stargazer automatically creates dynamic tutorials and removes the restrictions of working with a static camera.
“The teachers are there to teach. The role of the robot is to help with the filming, that is, to do the heavy lifting,” says Jiannan Li, a doctoral student in the computer science department of the Faculty of Arts & Science and principal investigator of the Stargazer project the problem. The scientists describe the project in the study “Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor Cues”published in Proceedings of the 2023 CHI Conference on Human Factor in Computing Systems.
The robot guides a single smartphone camera on a seven-degree-of-freedom robotic arm. This allows it to move with the video subject and independently track objects of interest. The system recognizes subtle hints from the teacher, such as body movements, gestures and language. They are captured by the camera and other sensors such as a depth camera. The speech is picked up by a wireless microphone and forwarded to the Microsoft Azure Speech-to-Text language software. The transcribed text is interpreted by the GPT-3 language model along with a user-defined prompt. It translates the teacher’s intention into a suitable camera image section.
Recommended Editorial Content
With your consent, an external YouTube video (Google Ireland Limited) will be loaded here.
Always load YouTube video
The video shows how Stargazer works.
The control commands for the camera are carried out naturally, as the researchers write. For example, pointing to individual objects is enough to cause the camera to pan. In addition to gestures, speech is also analyzed and converted into robot control commands. For example, if the teacher says they are inserting one object into another from above, Stargazer immediately translates this into a top-down camera view to choose the optimal frame for the viewer. This works almost in real time. The robot does not require specific movement instructions. The lecturer can fully concentrate on communicating with the viewer.
Satisfactory instructional videos
In the study, the system had to prove its functionality. Six teachers each completed two practice sessions before they could produce instructional videos with Stargazer. They showed how to maintain skateboards, made interactive sculptures and showed how to set up virtual reality headsets. According to the researchers, all participants were able to produce videos of satisfactory quality with the robot camera.
The team is now working on expanding the vocabulary of subtle intentions to improve collaboration with the robot and produce even more beautiful videos. In particular, the researchers want to examine the combination of signals such as gaze and body language more closely. The potential of camera drones and camera robots on wheels is also to be researched in order to automatically produce videos in larger environments.
The scientists admit that Stargazer is not yet made for the masses, but there is a market for professional robotic film equipment. The robotic arm used is also still too expensive for teaching applications, and it also relies on external sensors, which makes the application even more complicated.
(olb)