Thrifty Food Plan: Better Planning and Accountability Could Help Ensure Quality of Future Reevaluations
Fast Facts
Over 40 million people rely on the Supplemental Nutrition Assistance Program (SNAP), formerly known as "food stamps," to buy food. Benefits are based on the Thrifty Food Plan, which estimates how much it would cost a family of 4 to eat a healthy diet on a budget.
In response to COVID-19, the U.S. Department of Agriculture expedited a planned update to the Thrifty Food Plan. USDA allowed plan costs to increase beyond inflation for the first time—resulting in a 21% increase in SNAP benefits.
But officials made this update without key project management and quality assurance practices in place. We recommended ways USDA can improve future updates.
EBT accepted here
Highlights
What GAO Found
The Thrifty Food Plan (TFP) describes how much it costs to eat a healthy diet on a limited budget, and is the basis for maximum Supplemental Nutrition Assistance Program (SNAP) benefits. In 2021, the U.S. Department of Agriculture (USDA) reevaluated the Thrifty Food Plan and made decisions that resulted in increased costs and risks for the reevaluated TFP. Specifically, the agency (1) allowed the cost of the TFP—and thus SNAP benefits—to increase beyond inflation for the first time in 45 years, and (2) accelerated the timeline of the reevaluation by 6 months in order to respond to the COVID-19 emergency. The reevaluation resulted in a 21 percent increase in the cost of the TFP and the maximum SNAP benefit. The reevaluation was complex and involved several USDA offices. However, USDA began the reevaluation without three key project management elements in place. First, without a charter, USDA missed an opportunity to identify ways to measure project success and to set clear expectations for stakeholders. Second, USDA developed a project schedule but not a comprehensive project management plan that included certain elements, such as a plan for ensuring quality throughout the process. Third, the agency did not employ a dedicated project manager to ensure that key practices in project management were generally followed.
USDA's Thrifty Food Plan Reevaluation Lacked Key Project Management Elements
USDA gathered external input, but given time constraints, did not fully incorporate this input in its reevaluation. Specifically, USDA substituted a limited internal review of the TFP report for the formal peer review it had initially planned. This review was conducted by USDA officials who had been involved in the TFP reevaluation, and therefore were not independent. The TFP report lacked a comprehensive, external peer review to assess the transparency, clarity, or interpretation of the results. As a result, the review also did not meet relevant Office of Management and Budget (OMB) and USDA guidelines, such as the requirement to publish a report with the results of a peer review.
The complexity of the economic model USDA uses to calculate the TFP led officials to make numerous methodological and policy decisions during the 2021 reevaluation, as they had in past reevaluations. However, GAO found that key decisions did not fully meet standards for economic analysis, primarily due to failure to fully disclose the rationale for decisions, insufficient analysis of the effects of decisions, and lack of documentation. As a result, members of the public and policymakers reading the TFP report may not understand the rationale for decisions made by USDA officials, and external parties would face difficulties reproducing the TFP, which further decreases transparency and accountability.
Why GAO Did This Study
SNAP supplemented the food budgets of more than 41 million people in 2021. Following a provision in the Agriculture Improvement Act of 2018 to reevaluate the TFP by 2022, USDA completed a reevaluation of the TFP in 2021.
GAO was asked to review the reevaluation. This report examines (1) USDA's administrative process for reevaluating the TFP in 2021 and the extent to which the process employed leading project management practices; (2) the extent to which USDA gathered and analyzed external input to inform the reevaluation; and (3) how the methodology and results of the reevaluation compared to methodological standards.
GAO reviewed USDA documents, interviewed officials and external experts, and compared the information collected to key project planning practices; relevant OMB and USDA guidelines for information quality, peer review, and scientific integrity; federal standards for internal control; and GAO's assessment methodology for economic analysis.
Recommendations
GAO is making eight recommendations, including that USDA develop policies to ensure TFP reevaluations follow key project management practices, peer review guidelines, and quality standards; and publish information to allow external parties to reproduce results. USDA did not explicitly agree or disagree with the recommendations but disagreed with GAO's selection and application of certain criteria. GAO believes the criteria were appropriate and stands by the findings, conclusions, and recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Agriculture |
Priority Rec.
The Secretary of Agriculture should develop and document a process to ensure that TFP reevaluations follow the project management practice of establishing a key document at the start of a project, such as a project charter, that includes an overall assessment of risk and measurable objectives and metrics for success related to project requirements and other expectations for the project. (Recommendation 1) |
As of December 2023, the Food and Nutrition Service (FNS) developed and documented a more formal project management guidance for use by agency staff, including a template for a project charter. The charter template includes sections prompting the project team to describe the high-level requirements of a project, as well as measurable objectives, criteria for success, and an assessment of overall project risk. FNS stated that the template will be used in the 2026 Thrifty Food Plan reevaluation. In addition, FNS stated that its Office of the Administrator provided a brown bag training to staff on the templates and tools and created a project management matrix to help teams determine when to use the templates. Projects concluding with a product including extensive research, service, or results--such as the Thrifty Food Plan reevaluation--are required to have project documentation, including a charter. Additionally, the agency re-initiated its Project Management Community of Practice to further equip project managers with tools and best practices to manage projects agency-wide.
|
| Department of Agriculture | The Secretary of Agriculture should develop and document a process to ensure that TFP reevaluations follow the project management practice of creating a comprehensive project management plan that describes how the project will be executed, monitored, controlled, and closed and, in addition to a project schedule, includes:
|
USDA did not explicitly agree or disagree with this recommendation. As of December 2023, the Food and Nutrition Service (FNS) developed and documented a more formal project management guidance for use by agency staff, including a project management plan template. The project management plan template includes sections prompting the project team to describe and reference the project schedule, a strategy for risk management (or a full risk management plan for complex projects), and the general approach to managing project quality. The template also includes a section prompting the team to list all internal and external stakeholders, and a section to describe the project's approach to conveying clear, consistent, and timely information to stakeholders. FNS officials stated that the section in the template addressing the project's scope outlines the project requirements, and a supplemental "requirements" template, provided by FNS in February 2024, focuses on the analysis, documentation, and management of those requirements. However, the section of the template on quality does not prompt the project team to describe how the project will be monitored and controlled based on applicable USDA and other federal quality standards, as we recommended. Beyond the templates, FNS stated that its Office of the Administrator provided a brown bag training to staff on the templates and tools and created a project management matrix to help project managers determine when to use the templates. Projects concluding with a product including extensive research, service, or results--such as the Thrifty Food Plan reevaluation--are required to have project documentation, including a project management plan. Finally, the agency re-initiated its Project Management Community of Practice to further equip project managers with tools and best practices to manage projects agency-wide. We will continue to review FNS's actions with respect to this recommendation and will consider it addressed when we receive documentation of additional guidance to project teams to describe how the project will be monitored and controlled based on applicable USDA and other federal quality standards.
|
| Department of Agriculture | The Secretary of Agriculture should develop and document a process to ensure that TFP reevaluations follow the project management practice of designating a project manager, or another member of the project team with project management expertise, to ensure that TFP reevaluations apply generally recognized project management practices, including creating key project documentation. (Recommendation 3) |
As of December 2023, the Food and Nutrition Service (FNS) developed and documented a more formal project management guidance for use by agency staff, including a template to officially designate a project manager. The template memo designating a project manager delineates specific responsibilities, including preparing project documents--at a minimum, to include a project charter and project plan that includes high-level requirements, cost, schedule, stakeholders, roles and responsibilities, risk management, communications, and quality management. FNS stated that this template will be used in the 2026 Thrifty Food Plan reevaluation.
|
| Department of Agriculture | The Secretary of Agriculture should develop and document a process to ensure that TFP reevaluations are subject to formal, comprehensive, and independent peer reviews before publication. (Recommendation 4) |
USDA has taken several actions to implement this recommendation. Specifically, in April 2023, USDA published an Evaluation Policy specific to the Food and Nutrition Service (FNS) that includes guidelines for publishing formal peer review plans for studies that are expected to yield influential scientific information (a designation that FNS previously indicated applies to Thrifty Food Plan reevaluations). On January 31, 2024, USDA published "Clearance Procedures for the Release of Scientific Products at USDA's Food and Nutrition Service." This document establishes the process to be used for Thrifty Food Plan reevaluations and all scientific products produced by FNS. This policy explicitly states that the Thrifty Food Plan completes federal and external peer review prior to Agency and Departmental clearance. The included guidance for peer review: states that project leads should develop a plan for peer review and should plan to ensure appropriate time for the review, including time to identify reviewers, time to review, and time to respond to reviews, if needed; provides considerations for the selection of peer reviewers; states that products that are either influential scientific information or highly influential scientific assessments require peer review, and the Peer Review Plan Form will be publicly posted; and provides a table with differences in peer review activities between influential scientific information and highly influential scientific assessments, which notes that in both cases a peer review report and the disclosure of peer reviewers' names, affiliations, and expertise is required.
|
| Department of Agriculture | The Secretary of Agriculture should consult with the Office of Information and Regulatory Affairs and designate the TFP as a "highly influential scientific assessment" subject to more stringent OMB guidance concerning peer review. (Recommendation 5) |
USDA did not explicitly agree or disagree with this recommendation. In December 2023, USDA stated that the agency consulted with the Office of Information and Regulatory Affairs to determine if the Thrifty Food Plan (TFP) should be designated a "highly influential scientific assessment" subject to more stringent OMB guidance concerning peer review. According to FNS, OIRA confirmed that FNS must peer review the 2026 TFP reevaluation per OMB guidance for a highly influential scientific assessment. However, according to FNS, OIRA did not require that TFP revaluations be deemed highly influential in perpetuity. Rather, OIRA indicated it would designate future TFP reevaluations as highly influential if the criteria for highly influential scientific assessments are met. FNS has not provided documentation of its consultation with OIRA, as we requested in March 2024, but the agency provided an advance copy of the peer review plan for the 2026 TFP reevaluation, indicating the plan to consider the reevaluation highly influential. We will close this recommendation when USDA provides documentation of its consultation with OIRA.
|
| Department of Agriculture |
Priority Rec.
The Secretary of Agriculture should establish specific quality assurance guidelines for TFP reevaluations that will ensure methodological decisions meet key quality standards for an analysis that will affect public policy and inform policy makers. These guidelines should summarize applicable USDA and other federal quality standards and should describe how such standards will be embedded in future TFPs. These guidelines should ensure that future TFP reports have:
|
USDA did not explicitly agree or disagree with this recommendation. In February 2024, the Food and Nutrition Service (FNS) stated that the agency would follow the same quality assurance guidelines for the 2026 reevaluation as it did for the "Thrifty Food Plan Cost Estimates for Alaska and Hawaii" report. For example, the report included information on the objectives of the analyses, alternatives considered, and sensitivity analyses. In addition, FNS published an "Initial Study Plan: Thrifty Food Plan, 2026." This document outlines several quality assurance commitments for the 2026 reevaluation, such as publishing the initial and a final study plan and the project management plan on FNS's website; conducting the TFP reevaluation in alignment with USDA's Scientific Integrity Policy; following analytic best practices, collaboration with experts, and external peer review; and ensuring transparency by disclosing key information in the final TFP report. However, this initial study plan does not lay out specific quality assurance guidelines or controls, such as a checklist, to ensure the reevaluation team is aware of and held accountable for key quality assurance steps and standards. By contrast, as we noted in our report, USDA's Economic Research Service (ERS) has published specific quality guidelines for its data products. For example, these standards state that all products are thoroughly reviewed by knowledgeable staff prior to dissemination to verify the accuracy and validity of the data, and that the procedures used to conduct this review must be documented and available upon request. ERS also has turned these guidelines into a checklist for product quality review. We will close this recommendation when USDA provides documentation of specific quality assurance standards that would apply to future TFP reevaluations.
|
| Department of Agriculture | The Secretary of Agriculture should ensure that FNS designs, documents, and implements key internal controls related to data processing, including standards and procedures for review of the computer code used in generating the TFP Market Baskets. (Recommendation 7) |
USDA did not explicitly agree or disagree with this recommendation. In June 2023, USDA stated that the Food and Nutrition Service (FNS) has designed, documented, and implemented expanded internal controls related to data processing, including review of the computer code, and said they were described in the "Thrifty Food Plan Cost Estimates for Alaska and Hawaii" report. For example, according to FNS, the calculations were performed independently by two economists and results were compared at each step to ensure the final analyses yielded consistent results. In February 2024, FNS stated they will follow these same steps for the 2026 Thrifty Food Plan (TFP) reevaluation. The agency also provided us with a USDA departmental regulation and an FNS Implementation Guide for Public Access to Scholarly Publications and Digital Scientific Research Data. However, these documents do not specifically cover the TFP and do not document any specific controls, standards, or procedures related to code review. Further, the documents state that USDA-published reports, software code, and computer models are not covered by the departmental regulation, although making this type of information publicly available is encouraged. We will close this recommendation when we receive documentation of the specific controls, including standards and procedures, for review of the computer code used in generating the TFP Market Baskets.
|
| Department of Agriculture | The Secretary of Agriculture should publish the computer code and raw data used to generate the TFP Market Baskets, to the extent allowable, along with all of the final equations used to create the model, in order to ensure qualified external parties can reproduce and replicate the TFP. (Recommendation 8) |
The agency agreed with and has implemented this recommendation. In November 2022, USDA published the data and code necessary to reproduce the standard Thrifty Food Plan. In addition, in August 2023, they published the data and code necessary to reproduce the Alaska and Hawaii Thrifty Food Plans.
|