Single-Photon Camera Enables Video Playback at Any Timescale

Computational imaging researchers at the University of Toronto captured an odd signal with their unique camera. While running experiments in the lab using the camera, a single-photon avalanche diode (SPAD), the Toronto Computational Imaging Group picked up an unexplained 80-kilohertz (kHz) flicker.

Using their powerful free-running imaging sensor, they detected each individual photon as it arrived from various light sources and logged its precise arrival time, down to a trillionth of a second. After acquiring all the data hitting each pixel, the team applied an algorithm that allowed them to create videos reconstructing the light at any given moment over an extreme range, from seconds to picoseconds.

The University of Toronto team used MATLAB^® to acquire the individual photon timestamp data and to control the moving components in their setups for their imaging technique, which the group dubbed a “microscope for time.”

The single-pixel SPAD has a 60-millimeter by 60-millimeter detection head with a tiny sensor in the middle. When the SPAD detected a weird 80 kHz signal, at first the researchers wondered whether they had merely encountered some artifact. On closer inspection, the team uncovered the source.

“It turns out that we have T8 LED replacements for fluorescent bulbs in the lab. Those actually flicker at 80 kilohertz,” Kutulakos said. “We didn’t even know this was happening.”

At the recent International Conference on Computer Vision (ICCV) in Paris, the team’s paper on passive ultra-wideband single-photon imaging received a prestigious award only given to two papers from thousands that computer vision experts around the globe submitted.

Potential applications for their technique include new types of 3D imaging and lidar systems as well as scientific imaging—for example, to capture biological events across multiple timescales or for astronomical observation of brief pulses of light that coincide with fast radio bursts.

Billions of Frames per Second

Single-photon avalanche diodes only became available more widely off-the-shelf within the past decade. The device’s high cost meant that very few computer science and computer vision laboratories had access to them, Kutulakos noted. Recent sensor developments prompted the University of Toronto team to ask fresh questions about problems that scientists in other fields have been touching upon for decades.

Astronomers specializing in gamma-ray astronomy, for example, deployed detectors to pick up individual particles and timestamp them. But these astronomers were more interested in periodic illumination from variable stars than the exact time-varying function describing star brightness since that’s what the physics required, Kutulakos explained.

In the Toronto Computational Imaging Group lab, postdoc Sotiris Nousias and Ph.D. student Mian Wei first used a SPAD to capture timing information for photons from light-emitting diodes.

“We had the LEDs flickering and looked at the output stream,” Nousias said. “We observed that the stream actually had the same pattern as the flicker and then worked to find the mathematical connection. That was our inspiration for the project.”

Nousias and Wei, joint first authors on the ICCV paper, wondered what information they could obtain if they had access to all the photons. Scientists tend to operate SPADs in special synchronized ways. Instead, the team operated the sensor in asynchronous mode. From there, the researchers sought to connect discrete photon arrivals with an underlying continuous function that describes the time-varying intensity of light.

Zooming Through Time

Wei estimated that, without access to the technical computing software, integrating each new component could take days to weeks. “Double-checking that the components worked as intended would add even more time,” he said.

“Because of our setups we can say, ‘Okay, I have this new Windows^® machine. I’ll take the code that has been running before and then port it over,’” Wei continued. “Now someone else can make use of their code and run the system as well.”

Dynamic Imaging Possibilities

The researchers hope their breakthrough inspires new advancements. One area ripe for exploration is flash lidar, a technique that produces a high-resolution depth image from a diffused pulsed laser rather than from a scanning pulsed laser. Typically, this involves sending light into the scene, measuring how long it takes for that light to return, and using this timing information to measure distance.

“One of the caveats is that you need to synchronize your light source with your system for this mechanism to work,” Wei explained. “Given our ability to measure signals asynchronously, we’re interested in understanding whether we could use an asynchronous light source to do flash lidar.”

In biology, events like protein folding or binding can happen across timescales from nanoseconds to milliseconds. Lindell said the team has been discussing their imaging techniques with biology experts at the university.

“The fact that these events are happening at these timescales is interesting to us, and the potential to observe them in a way that people haven’t been able to so far is enticing,” he said.

Computational refocusing might have applications in other sciences. Astrophysicists have wanted to understand what causes fast radio bursts ever since a radio telescope first detected the phenomenon 20 years ago. These mysterious energy blasts occur across the sky and only last milliseconds, but each one is incredibly powerful. Computational refocusing could enable the detection of FRBs in the optical domain.

“We don’t know when they will happen,” Nousias said. “You need to monitor the sky and then go back into the data at the nanosecond timescale to observe them. Now we have the tool to do this.”

Kutulakos anticipates that their SPAD method could help detect problems or defects in engineered mechanical components that standard inspections miss. When an engine has trouble, vibrations might indicate an issue, but the exact source may prove elusive without more information. By training a camera on the engine and collecting all the data, you could find that needle in a haystack, he said.

The group is also collaborating with other microelectronics researchers to expand what the cameras can do. A single-pixel camera enables the team to produce videos from repetitive motions, such as spinning fan blades, but truly dynamic events call for a 2D sensor.

Next, the team plans to work on improving their technique. Reconstructing the signals remains fairly time-consuming and computationally inefficient, Lindell noted. So, they’re working on capturing events across all the timescales in real time.

“I think there will be results that we didn’t anticipate as we point this camera around the lab or even outside,” Lindell said. “I’m excited to see what we find. It will be a new way of capturing the world.”