Developing Meaningful Metrics for EO Solutions — Solution Co-Development Toolkit

Purpose:

This tool is an overview of how to develop and use different types of metrics to monitor and measure impact-driven Earth Science solutions and programs.

How and When to Use This Tool:

Use this tool during Phase 2 when developing a Solution Implementation and Impact Monitoring Plan (SIIMP). It can also be used to inform broader program and project plans as part of requirements to measure project success criteria. The SIIMP plan template includes a fillable sample table for metrics.

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 105

What Are Meaningful Metrics?

Metrics are quantitative measures that become meaningful when designed specifically to monitor and validate whether a solution is achieving what it is designed to achieve.

A metric is a specific, measurable variable that provides systematic evidence about solution progress, performance, effectiveness, and impact. Metrics are embedded in the solution results framework, which acts as the blueprint for how and why a program is expected to work. More information on developing a results framework can be found in the tool Designing for Impact 101. Metric data is collected as part of a Solution Implementation and Impact Monitoring Plan associated with the solution and also built around the results framework.

Why Are Metrics Used?

Metrics reframe routine project reporting and information collection into actionable intelligence that enables the solution development team, as well as broader internal (such as NASA leadership) and external stakeholders to make informed decisions related to program performance.

A well-designed suite of metrics helps solution teams track deliverables and reporting requirements, and also understand whether outcomes are achieved and intended impact can be realized. To that end, metrics are used to:

Track progress toward goals
Validate steps in the results framework
Support solution management at all levels
Demonstrate value to stakeholders

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 106

Types of Metrics

A four-tiered system can be used to categorize metrics by the level of insight they provide. This tiered approach aligns with the solution results framework, and ensures that metrics serve multiple management and reporting needs. Institutional reporting requirements should be considered at each level and relevant metrics included at solution inception.

Level 1 - Input Metrics: Track necessary steps in solution implementation and delivery requirements Examples: Trainings conducted; individuals participating in trainings; needs identified; programming codes developed; stakeholders consulted, partnerships developed

Level 2 - Output Metrics: Track solution deliverables and anything generated from solution team inputs (Level 1) Examples: Solutions developed; users trained and able to adopt and sustain solutions, institutions adopting solutions, publications, information products, etc.

Level 3 - Outcome Metrics: Measure immediate results of the solution and whether intended objectives have been met; measure change in behavior, capacity or conditions as a result of adopting the solution. Validate whether assumptions held true and/or risks were successfully mitigated. (For more on assumptions and risks, see Needs Assessment Tool, Guidance for Analyzing User Inputs, and Designing for Impact 101). Outcome metrics are the first level that demonstrate a result of the solution and what a solution has achieved. Examples: Solutions adopted; successful uptake confirmed; decision-makers and beneficiaries have greater access to timely, trusted, quality data, information and analyses; applications being used in decision-making.

Level 4 - Impact Metrics: Designed to measure deeper or longer-term results arising from solution adoption and use or changes in behavior as a result. Impact metrics can address both the breadth of impact (for example, how many people are affected) and the depth of impact a solution can have on one or more individuals. Collecting information on impact metrics may not be within the solutions team's capacity to complete based on availability of resources, access to information, and project timeline. Metrics associated with Economic Impact Assessment are often impact level metrics. Examples: Improved resilience and sustainable resource management through the ability to take early and informed action, as a result of increased use of EO and EO-based solutions. Number of people evacuated prior to floods, how much saved by harvesting crops prior to floods.

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 107

What Makes a Good Metric?

A good metric is SMART:

Specific: Clearly defines what is being measured
Measurable: Quantifiable or qualitatively assessable; information is available, and capacity exists to collect and process
Achievable: Realistic within the program's scope and timeframe
Relevant: Aligned with solution and broader strategic objectives
Time-bound: Includes a timeframe for measurement

The number of metrics developed by a solution team will depend on a variety of factors including: 1) the complexity of the solution; 2) the level of effort and inputs required; 3) the number of and types of outputs and outcomes anticipated; and 4) any required reporting that involves data collection methods. For example, some programs or offices have program-wide objectives for which metrics that capture progress towards those priorities can be included at the solution level MIL plans.

Metrics should serve a purpose, specifically to validate or track steps in your logic chain and results framework or to fulfill reporting requirements. Don't count only for the sake of counting. A useful suite of metrics to best fit your solution should ensure the following:

You are measuring what is needed for funding reporting requirements.
What you are measuring is insightful, and not just easy/convenient to measure.
The insight provided by the metric is worth the time spent pursuing it.
A balance of metrics that address both breadth and depth of a solution's results.
Metrics information source is accessible and can be collected with a reasonable level of effort.

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 108

Fig. 1 — Sample Results Framework with Example Metrics

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 109

Examples of Meaning Metrics: Outputs, Outcomes, and Impact

Output: "Number of individuals trained" Outcome: "Percentage of trained individuals demonstrating sustained skill application in professional contexts" (measured three-six months post-training) Outcome: "Percentage of trained individuals with a self-reported acquisition of a new skill." (measured immediately post-training)
Output: "% of trainees/partners completing pre- and post-activity surveys" Outcome: "Number of feedback/needs assessment documents produced identifying institutional or individual partner needs, best practices and/or actionable lessons learned" Outcome: "% increase in trainee/partner ability to apply geospatial tools for modeling water quality, as indicated by pre- and post-training testing."
Output: "Number of solutions achieving ARL goal" Outcome: "Number of solutions operationalized and showing sustained use in decision making (Measured one year after project solution is put into practice.) Impact: "Number of solutions leading to intended benefits" (Measured one year after project solution is put into practice.)
Output: "Number of products developed (incl. publications, training materials, tools/applications)." Outcome: "Number of training materials developed under Training-of-trainers model and then successfully used by end users/beneficiaries" Outcome: "Number of co-developed products adopted by end-users for early warning alerts."

NASA Earth Action Solutions Co-Development Toolkit, v0.1 | 110