The Mission to Improve Social Science Data Collection

The problem

Meteorologists are quite literally bathed in data; we have such frequent, dense observations of the atmosphere, that often the concern is the best way to store, transmit, and make use of everything that’s available. It can be easy to take these observations for granted, or even with everything that we have, to still feel like it’s not enough - who among us hasn’t wished for more frequent soundings, or the return of the profiler network, if just to satisfy our desire for one additional data point?

Imagine trying to conduct the business of meteorology without all of that. Imagine if all you had to go off of were the primitive equations, and to then diagnose what was going on in the atmosphere, all you had to go off of were the eventual hazards themselves: reports of tornadoes, reports of floods, and fires. You knew from theory that some set of things was likely going on in that rotating fluid to generate those outcomes, but you wouldn’t know exactly what that was.

When weather events occur and people are impacted, this is often the situation social scientists find themselves in. We have a lot of theories that describe human behavior in many ways: from the level of individual cognition to social interactions and even cultural influences, we have numerous theories that link situations characterized by risk and uncertainty to decision-making processes for responding to those conditions. However, unlike the field of meteorology, we don’t get the opportunity to observe people as events unfold, or - much of the time - to even hear from them afterward to get a post-mortem of what their decision-making was like. Which leaves us in a difficult place when meteorologists invariably work events with tragic outcomes, and they turn to us asking, “Why? Why did this happen? What could I have done differently?”

This is why we need a data revolution in social science data collection. It is amazing that science organizations like NOAA and NSF have increasingly supported fundamental and applied research in the social sciences for weather. We’ve learned a lot about the kinds of factors we should be looking at when events occur. We’ve come a long way toward developing those primitive equations. Now we need the observations to take it further and understand the machinations of specific events.

The first step forward

The first significant effort to produce observations of the ways people in the U.S. receive, understand, and respond to weather information has come from the University of Oklahoma Center for Risk and Crisis Management. In a project led by Dr. Joe Ripberger, the Center conducts annual surveys of the U.S. population. These surveys capture the ways people think they might receive weather information when real events occur, and the ways they think they would respond under different kinds of conditions. This information is extremely valuable, and gives us a useful baseline for exploring differences in foundational attitudes/beliefs across the country.

That said, we also know that individuals are terrible forecasters of their future decisions - especially when those future decision settings are very different from the kinds of settings they’re in while answering our questions. While calm, collected, and sitting alone in front of a computer, an individual is likely to have more optimistic views of their future capabilities, and less recognition of the constraints they might face. Being in a very different emotional state, they are also likely to give different answers than they would during the heat of the moment, under great time pressure, when faced with a real, significant, life-threatening hazard. The annual surveys give us a feel for the climatology of beliefs across the country. To move forward and give specific insight into particular events, we need the observations from real events. We need to sample something of what people really think about and do when all of the complexity of the event is upon them. This is especially critical if we want to assess the contribution of weather information to the decision process.

The tornado post-event survey

A consortium of organizations is working to make this dream a reality - at least, for events where tornadoes occur. Today, post-tornado public studies are conducted for fewer than 1% of all tornadoes that occur; people really only go out into the field when generational outbreaks or single tornado events happen and we see a large loss of life. That gives us a very biased, partial view into the ways people receive and respond to weather information. In particular, it probably gives us an overly optimistic view, as those events also tend to garner the most public attention. Over 1,000 tornadoes happen each year in the U.S. What might we find if we open the door to getting data from more of these events? The consortium is eager to explore that very question.

To begin the process, a standardized post-event survey was developed by a team of social science experts who have experience evaluating public response after tornadoes. Similar to the importance of having well-calibrated equipment in the atmospheric sciences, this standardized instrument is key for collecting data that will make sense together after sampling from multiple events. This survey then went through a federal human subjects approval process through, of all organizations, the Office of Management and Budget (the reason for this would be its own blog post… but if you’re curious, look up something called the Paperwork Reduction Act); this process was time-consuming (it took over a year), but ultimately rewarding, as now this tool can be used by many partners.

The consortium

The post-event survey is in its pilot stage now, and is being utilized by several different organizations:

National Severe Storms Laboratory “Tornado Touchdown” Citizen Science App - This past year, NSSL launched a citizen science app that any member of the U.S. public can use to report their experiences with tornado events.

National Weather Service - When forecasters go into the field after tornado events, they often encounter survivors wishing to tell their story. Soon, NWS forecasters will be able to point these individuals to the Tornado Tales app and greatly increase the representativeness of the data coming in.

National Hazards Center Quick Response Grants Program/Weather Program Office - The Natural Hazards Center has long run a QRG program that offers researchers funding to get into the field shortly after hazards occur to collect perishable data, such as response data. This year, with support from the NOAA WPO, they are providing a special set of funding for individuals to use the standardized tornado survey instrument and go into the field to collect post-event information.

What we’re trying to do is harness the resources that already exist in the field, such as NWS forecasters and QRG researchers, and the power of citizen science, to dramatically improve the volume and comparability of post-event observations. It is our hope that this endeavor will give us unprecedented insights into the questions meteorologists are always asking, and bring our science closer to the operational realities and concerns of the weather enterprise.

Keep us in mind when tornadoes occur in your area - advertise this link widely and help us better understand the effects of our weather forecast & warning system! https://inside.nssl.noaa.gov/tornado-tales/