Big Data in the Night Sky

After years of planning, the Vera C. Rubin Observatory in Chile will begin capturing images of the night sky next year. Its 8.4-meter telescope — the Simonyi Survey Telescope — will photograph a portion of the sky every thirty seconds, completing a survey of the entire visible sky every few nights. Scientists expect the resulting data to answer some of the most fundamental questions about the Universe.

The telescope will collect so much information that powerful new tools are needed to manage and analyze the massive data stream. Scientists at the UW’s Data-Intensive Research in Astrophysics and Cosmology (DiRAC) Institute have spent the past six years developing those tools. Based in the UW Department of Astronomy, the institute brings together astronomers with expertise in computer science, statistics, machine learning, and more.

“We’re building software that will take images from the telescope’s 3.2 gigapixel camera and turn them into data and then analyze that data,” says James Davenport, research assistant professor of astronomy and associate director of DiRAC. “By checking the images against reference images, we can identify any changes — if a supernova goes off, an asteroid goes by, anything like that — and alert the astronomy community.”

When the Rubin Observatory was planned twenty years ago, with the UW as a founding institution, existing technology was woefully inadequate for managing the amount of information the telescope was expected to produce. It was unclear how researchers were going to manage terabytes of new data every night.

“We had to bank on Moore’s law — the idea that processing capability and bandwidth would just keep increasing over time,” says Davenport. “In fact, that’s what happened. The landscape has completely changed since we were dreaming this up. When this project began, we did not have cloud computing. Everything has come together in our favor.”

Still, it has taken dozens of talented scientists to develop the advanced datasets, algorithms, and tools needed to process, analyze, and understand the massive amount of astronomical data that will emerge. Until the observatory is operational, the researchers are working with data from other telescopes to refine their methods.

All this preparation should help scientists identify changes in the sky quickly. Speed is critical for answering long-standing questions in astronomy. Davenport offers the example of a supernova — an exploding star.

“Astronomers have long been able to detect supernovae about a week after they explode, but to understand the physics of a supernova and the star that created it and the gas that’s around it, you need to see it in the first day, before it reaches its maximum brightness,” Davenport says. “You want to see the thing actually exploding. You want to see that explosion front slamming into the gas around it, which means ideally catching it in the first twelve hours, before it’s barely gotten bright enough to detect. So there’s this race to develop a machine learning tool that can identify a supernova that’s just starting to go off.”

About two dozen faculty, postdoctoral, and graduate student researchers at the DiRAC Institute are involved in this work. This summer, three undergraduates were on board as well thanks to the DiRAC Summer Research Prize. The prize provided financial support so the students could dedicate their summer to research.

“This took so much pressure off my shoulders, and I couldn’t be more grateful for it,” says DiRAC prize recipient Josue Torres-Valdovinos. “The thing that excites me the most is being able to contribute to the Astro community and feeling like I’m part of something bigger.”

The prize was funded by donors through targeted gifts on Husky Giving Day, an annual philanthropy event at the University of Washington. “Seattle is chock full of people who love technology and outer space and science,” says Davenport. “We saw Husky Giving Day as a way to connect with our community of donors and astronomy enthusiasts.”

Several large gifts helped DiRAC reach its funding goal. David Brooks, a member of the DiRAC Advisory Board with a lifelong fascination with astronomy, was one of the donors. “The work DiRAC is doing is crucial,” says Brooks. “It’s very gratifying to see the UW playing this important role. So the idea of supporting these students was really appealing. Hopefully the gift gave them a sense that what they were doing mattered.”

The prize recipients, all astronomy majors (though that was not required), worked independently with guidance from a faculty or postdoctoral research mentor. Ishan Ghosh-Continho continued an ongoing project concerning the spectroscopy of high mass stars, which involves gleaning detailed information about a star from its spectrum of colors. Torres began a new project, developing updated software for image analysis. Arif Chu worked on an efficient way to detect low-luminosity variable stars or short-lived phenomena like supernovae. 

“This was a new field of research for me, but I felt that a lot of the skills from courses such as ASTR 324 (Introduction to Statistics and Machine Learning in Astronomy) greatly aided me in doing this cutting-edge astronomy research,” says Chu.

Not all of the summer projects achieved the desired results. That’s to be expected, says Davenport. In fact, knowing how to regroup after failure is an important skill for any researcher. “I tell students, ‘You have to be okay with the answer to a research question being ‘no,’” says Davenport. “You can’t dictate what nature is going to decide is the right answer. That’s something you relearn again and again and again as a scientist.”

The Rubin Observatory is sure to provide plenty more opportunities to learn that lesson — but also to discover much more about our mysterious Universe. Davenport can’t wait.

“We think this telescope will study something like 17 billion stars in our galaxy,” he says. “Right now our biggest catalog is about 2 billion stars. It will be the first time that we have more stars cataloged than people on the planet. We think this will be a game changer.”