Researchers have found ways of combining the views of 480 video cameras mounted in a two-story geodesic dome to perform large-scale 3-D motion reconstruction, including volleyball games, the swirl of air currents, and even a cascade of confetti.
Though the research was performed in a specialized, heavily instrumented video laboratory, Yaser Sheikh, an assistant research professor of robotics at Carnegie Mellon University, says the techniques might eventually be applied to large-scale reconstructions of sporting events or performances captured by hundreds of cameras wielded by spectators.
In contrast to most previous work, which typically has involved just 10 to 20 video feeds, the researchers didn’t have to worry about filling in gaps in data—their camera system can track 100,000 points at a time.
Instead, they have to figure out how to choose which of the hundreds of video trajectories can see each of those points and select only those camera views for the reconstruction.
“At some point, extra camera views just become ‘noise,’” says Hanbyul Joo, a PhD student at the Robotics Institute. “To fully leverage hundreds of cameras, we need to figure out which cameras can see each target point at any given time.”
How to Choose Cameras
The research team developed a technique for estimating visibility that uses motion as a cue. In contrast to motion capture systems that use balls or other markers, the researchers used established techniques for automatically identifying and tracking points based on appearance features—in this case, distinctive patterns.
For each point, the system then seeks to determine which cameras see motion that is consistent with that point.
For instance, if a point on a person’s chest is being tracked and most cameras show that point is moving to the right, a camera that picks up motion in the opposite direction is probably seeing a person or object that is in between the target and the camera. Or it may indicate the person has turned and the chest is no longer visible to the camera.
In either case, the system knows that the camera cannot see the target point and that its video feed is not useful for 3-D reconstruction involving that point.
Other researchers have been able to use images from a large number of cameras, such as smartphones, to create 3-D reconstructions of still images, Joo notes. But without methods such as the visibility estimation technique, 3-D motion reconstruction at such a large scale has not been possible.
Capturing Confetti
In their laboratory, called the Panoptic Studio, the researchers have 480 video cameras, plus an additional 30 high-definition video cameras, arrayed all around and halfway up the walls of a geodesic dome that can easily accommodate 10 people.
Such a dense array of cameras enables the researchers to perform 3-D motion reconstructions not previously possible. These include 3-D reconstructions of a person tossing confetti into the air, with each piece of paper tracked until it reaches the floor.
In another case, confetti is fed into a fan, enabling a motion capture of the airflow. “You couldn’t put markers on the paper without changing the flow,” Joo explains.
Likewise, such techniques might be used for reconstruction of the motion of animals, which typically can’t be instrumented. The researchers have used the Panoptic Studio to capture the fine details of people interacting, including college students chatting or a child’s evaluation by a psychologist for signs of autism.
The researchers presented the findings at the Computer Vision and Pattern Recognition conference on June 24-27 in Columbus, Ohio.
The National Science Foundation and a Samsung Scholarship supported the work.
Source: Carnegie Mellon. Republished from Futurity under Creative Commons license 3.0. Read the original.
