Measuring the Success of Conservation Programs

Defining and measuring success is easy—if you are Rube Goldberg. A widely acclaimed 20th century cartoonist, Goldberg depicted outlandish inventions that accomplished simple tasks through an intricate series of linked steps, each one triggering another until a desired outcome was reached. Success, in Goldberg’s world, was clearly defined and could be attributed directly to the completion of several sequential, though highly improbable, cause-and-effect actions. Success, in the real world, even when it is clearly defined, is not so easily measured. Gauging the success of government programs, in particular, can be downright complicated, even when the principles used in designing them are rather simple.

Most conservation programs, for example, are designed to improve the environment by offering incentive payments to farmers, who are thereby induced to change their farming practices. Those changes in farmers’ practices—be they reducing pesticide use, adopting conservation tillage, or constructing a riparian buffer—should then lead to enhanced environmental quality. But, unlike the chain of events in a Goldberg invention, the actions involved in a conservation program take place not in isolation, but, rather, within a larger set of complex interactions, making it difficult to link programs to actions to outcomes.

The first step in measuring the success of agricultural conservation programs—and other programs designed to address agri-environmental issues—is linking a change in farmers’ stewardship behavior to the program being evaluated. Because many other factors (including other government programs) influence farmers’ choices, it is critical to determine the extent to which it was a given conservation program incentive that stimulated some farmers to do something that they would not otherwise have done. A second step requires assessment of how the portion of observed stewardship behavior that can be linked back to conservation program incentives then affects environmental quality—given that other factors also affect the environment.

Gauging Farm Operators’ Responses to Program Incentives

Farm operators are the target of conservation program incentives, even though the program itself aims to target one or more environmental enhancements. Thus, to evaluate the program, one must determine exactly how program incentives induced operators of farms of various types, sizes, or features to “sign up” as program participants. Then, for those who become program participants, it is important to find out how the type and extent of conservation practices they adopted relate to the levels of incentives provided through the program. Only by separating the influence of program incentives from other factors that affect farmers’ conservation choices can the program evaluator be confident that it was the program being evaluated that had an effect, not other circumstances.

A farmer may adopt conservation practices for a myriad of reasons. He or she may be an ardent environmental steward who would implement a particular practice (like maintaining grassed buffers between cropland and water sources) regardless of program incentives. Alternatively, a farmer may adopt an environmentally friendly practice wholly or partly in order to increase profits. ERS research on conservation tillage, for example, demonstrates that good stewardship can also be good business. Policy incentives aren’t usually required to induce a farmer to adopt what he or she views as good business practice; market forces should do the trick in this regard.

In evaluating the effectiveness of incentives to induce farmers to participate in conservation programs, it is important to note that conservation programs are not implemented in a policy vacuum. Both the costs and benefits of participating in a given program will vary as a direct result of the confluence with other government programs. For example, commodity programs influence some crop prices, making it more or less economically advantageous to manage the crops in ways that enhance environmental quality. Input use is sometimes controlled through quantity restrictions and use regulations. Input prices may also be influenced by policies—including labor laws, pesticide regulation, and subsidization of irrigation water—that influence relative input prices and, thus, the financial costs or benefits of conservation practices that shift input use patterns. Finally, technological change, economy-wide variables (such as interest rates and unemployment rates), and farm household constraints (such as the role of off-farm work in farm household income) are also likely to influence farmers’ decisions about farming practices—whether or not a conservation program incentive is added to the mix.

Because farmers may adopt conservation practices for reasons unrelated to the conservation program, simply identifying changes in farmers’ practices (let alone environmental quality) is an insufficient basis for judging the success of a conservation program. One has to be able to determine what proportion of farmers’ practices can be attributed to a particular program before the success of the program can be assessed.

Isolating the effects of program incentives from the effects of other factors potentially influencing farmers’ observed conservation practices demands a lot of data of particular sorts. A necessary requirement is the collection of data that enable statistically reliable comparisions of farming practices by farmers before and after program implementation, or by farmers who did and did not participate in the program in a given year or years. Statistical analysis of such data can support or refute a correlation between farm practices and conservation program provisions.

However, supporting or refuting simple correlation is not sufficient because that correlation may be spurious and because it does not prove causality. A “before-and-after” comparison, for example, might miss the strong influence of a new program on participants’ behavior if other factors, such as unusual weather conditions, prevented a large number of the participants from following through on their program-induced good intentions. Similarly, a “with and without” comparison could falsely attribute observed conservation practices to the conservation program if all farmer participants in the program were pre-inclined toward voluntary environmental stewardship even without the program, and nonparticipants were disinclined. More information is needed than simply who participated and what practices they employed if a strong case is to be made that the program was the stimulus for farmers’ adoption of observed practices.

Additional data are necessary to separate the effect of a conservation program incentive from the effects of concurrent changes in market prices, weather, other policies, and technology. Identifying the farmers for whom program incentives induced adoption of conservation practices requires data on the characteristics—types and locations—of both participating and nonparticipating farmers, the circumstances under which they made a participation decision, the amount of the incentive to which they did or did not respond, and regional and other variables.

A close look at outcomes associated with the Conservation Compliance provision of the 1985 Food Security Act reveals the importance of isolating the effects of the program in order to measure its success. The provision requires agricultural producers to implement soil conservation systems on highly erodible (HEL) cropland to remain eligible for farm program payments. Annual soil erosion on U.S. cropland declined by 40 percent between 1982 and 1997, suggesting that compliance mechanisms encouraged greater conservation effort. However, erosion also declined on cropland not subject to compliance requirements, demonstrating that other factors must also have played a role in reducing soil erosion. On farms for which conservation practices could have increased net returns to farming, for example, adoption may have eventually occurred regardless of effects on soil erosion. In fact, after accounting for other factors, such as erodibility, commodity program payments, and land use changes, ERS research shows that only about 25 percent of overall erosion reduction between 1982 and 1997 could be directly attributed to Conservation Compliance. Even on the HEL lands targeted by the provision, about 11 percent of erosion reduction during that period was due to factors other than Conservation Compliance.

Linking Farmers’ Choices to Environmental Quality

Measuring changes in farmers’ practices that result directly from conservation program changes tells only part of the story. Conservation programs are not designed simply to induce a change in conservation practices, but to change those practices in order to improve water quality, air quality, wildlife habitat, or a host of other environmental attributes. More and more frequently, conservation programs aim to improve all of those environmental attributes at once.

Connecting the dots that link a program’s incentives to success in achieving that program’s environmental goal(s) is difficult in general, but can be especially challenging when evaluating conservation programs. Most of these programs address “nonpoint” sources of pollution, such as the nutrients, sediments, pesticides, and salts that enter water diffusely in runoff. In comparison to “point” sources, such as factories and municipal plants, which discharge through a pipe, ditch, or smokestack on which a meter can be installed, nonpoint sources are not so easily measurable and have an environmental effect only in the aggregate.

For example, the goal of a particular conservation program might be to address water quality problems caused by agricultural production. Evaluating a program based on that objective would require data on the entire set of actions and outcomes associated with agricultural production. Farmers control their inputs and crop production practices. Their management decisions, including which crop is produced on which field and with what combination of inputs, can affect water quality, but gauging whether or not and how much they actually do affect water quality is a difficult task. Farmers’ decisions may lead to field-level emissions (through runoff or leaching) of potential pollutants, such as sediments, nutrients, and chemicals, which are difficult to monitor. Depending on the location of the field and other physical and environmental factors, an emission may or may not find its way to the target water body.

But even that sequence of events is only part of the story. The last piece involves the underlying objective: What is it about water quality that concerns us? Is the goal to reduce nutrient concentrations in drinking water? Is it to provide improved fish habitat, perhaps to increase recreational fishing benefits? Once a (potential) pollutant reaches an environmental sink, such as a river or aquifer, it may or may not have ecological or human health implications, depending upon its toxicity, the number of other sources emitting the same pollutant, interactions with other pollutants, and the total emissions simultaneously reaching the environmental sink. While scientists know much about the relationship between nitrogen runoff and tillage practices, and the effects of nitrogen levels on biological functions, less is known about how nitrogen is transported from a myriad of individual fields to specific water bodies or other sinks.

In evaluating the effects of a conservation program on environmental quality, the nonpoint source issue is compounded by the exceptional site specificity of many agri-environmental events. Soil losses (or other pollutants) at one location may have a different effect on the environment than an identical level and type of soil loss in another location. Furthermore, similar levels of environmental effects vary in value among locations depending upon the proximity of human populations or economic activity to the site of the damage. For example, if a program objective is to help restore a recreational fishery, water quality improvements that increase fish populations closer to cities and where interest in fishing is particularly high will be higher valued than equivalent changes in fish populations in regions of the country that are sparsely populated or where interest in fishing is low. Estimating monetary-equivalent values for environmental improvement is a particularly difficult task that, while not necessary for judging whether or not a conservation program met its goals, is essential to determining how efficiently those goals were met.

Models Simulate What We Cannot Observe

Environmental process models can help overcome the nonpoint source and site specificity complications of conservation program evaluation by substituting predictions from models for direct observations of effects. For example, site-specific changes in (in-field) soil erosion due to particular erosion control practices can be estimated using the Universal Soil Loss Equation and the Wind Erosion Equation. Both models provide reasonably accurate results and require only minimal data (a total of six variables) describing climate, topography, soil, and cropping information at the field level. In contrast, models of nutrient and pesticide runoff are far more complex, simulating multiple environmental effects from the transport and fate of multiple pollutants into environmental sinks. These “fate and transport” models require a lot of data, often necessitating the use of dozens of variables.

Any one process model has unique advantages and disadvantages, depending on the indicator of interest, but relatively few are capable of simulating the environmental effects of changes in agricultural practices on a national scale. (See box, “Some Agri-Environmental Process Models.”)

A final complication: Model results are unlikely to match real world observations because farming practices aren’t the only things that affect environmental quality. Floods or drought can damage the environment even under the very best management practices. A given level of runoff may cause no environmental damage in a wet year but may significantly harm fish and wildlife in a dry year when streams have insufficient flows to dilute the runoff to nonharmful levels. Likewise, a single watershed may well experience pollutant discharges not only from agriculture, but also from industrial sources, municipal water treatment plants, urban runoff, aerial deposition, and even natural seepage. Thus, the influence of unmodeled events needs to be extracted to reconcile simulation results with measurements made on the ground.

Identifying Appropriate Environmental Indicators

Just what is the best indicator by which to measure environmental quality change in the policy evaluation context? Regardless of whether it will be measured directly or simulated with an agri-environmental process model, the indicator(s) by which a given program will be evaluated must be carefully selected. Reflecting broadened public concerns, conservation programs increasingly target multiple environmental quality goals. Along with reductions in soil erosion, potentially measurable goals have expanded to include improved water quality and conservation of wetlands and wildlife habitat. Newer program objectives may include preserving open space, managing nutrients from fertilizers and livestock waste, reducing pesticide runoff, improving air quality, reducing greenhouse gas emissions, or sequestering carbon in soil.

The appropriate indicator for evaluating a program’s success must map to an aspect of environmental quality that the program aims to address. But that’s not enough. It must also link directly to those changes in conservation practices induced by the program. For example, a measure of ambient downstream water quality, such as nitrogen concentration, may appear to be an ideal indicator of the success of a conservation program that aims to improve water quality. But if agriculture is only a small part of the aggregate water quality problem, ambient water quality may be getting worse, even with a wildly successful conservation program in place. The ambient water quality indicator may not measure the factor of interest, which, in this example, is agriculture’s contribution to water quality, and thus is not a good choice for evaluating this agri-environmentally oriented program. In this case, a less direct measure of water quality, such as pounds of nitrogen discharged into the water body from farm fields, may actually be a better indicator.

Appropriate indicators are:

• Policy relevant—provide a direct link to both the environmental attributes of concern and the behavioral changes associated with the evaluated program incentives;
• Measurable—based on sound science and make use of data that are available or could feasibly be collected;
• Reasonably priced—cost-effective in terms of data collection, processing, and dissemination; and,
• Easy to interpret—communicate essential information to policymakers and other stakeholders.

Putting It All Together

The voluntary nature of most U.S. conservation programs, the human factors involved in farmers’ decisions to participate (and to what extent), the complexity of farm household decisionmaking, and the nonpoint source and site-specific nature of agri-environmental problems combine to make evaluation of conservation programs a data-intensive and technically challenging process. To be successful, program evaluations must answer both of the following questions explicitly, through estimated, simulated, or directly measured means.

• How do different farm operators in different circumstances decide what production and conservation practices to implement, in the presence and absence of the conservation program being evaluated, at different levels of incentives provided by that program?

Isolating the unique effect of conservation program incentives on farmers’ practices requires analysis to extract the influence of other (policy, household, general economic, etc.) factors that affect farm-level decisionmaking. This, in turn, requires evaluators to collect data on the full set of factors potentially affecting farmers’ decisions, in sufficient volume and across diverse farm and land types and locations, to allow statistical segregation of program-related effects from those of other influential factors.

• How do the farm practices attributable to conservation program incentives affect environmental quality?

Isolating the unique effect of farm practices on environmental quality requires program evaluators to determine where, and under what resource conditions, practices implemented in response to the program are located, and to designate appropriate agri-environmental indicators for measuring program success. Process models that simulate the complexities involved in the transport of agricultural runoff from multiple fields to environmental sinks may help link environmental performance with farm practices. But even then, additional analysis is required to reconcile model predictions with real world observations.

The complicated series of cause-and-effect relationships associated with conservation program evaluation seem beyond even the imagination of Rube Goldberg. Many factors must be accounted for to determine the portion of environmental enhancements directly attributable to program incentive-induced changes in farmers’ practices. Still, carefully designed survey and monitoring programs encompassing each of those relationships in a coordinated fashion make such evaluation not only feasible, but well within reach.