Part 1: AI Project Evaluation with Human-in-the-Loop Focus

1. Title

Evaluating AI Impact: Developing and Tracking KPIs with Human-in-the-Loop Oversight

2. Objective

The objective of this assignment is to enable students to design a set of Key Performance Indicators (KPIs) to measure the effectiveness of an AI-driven improvement to a business process, while emphasizing the critical role of human-in-the-loop (HITL) evaluation in assessing outcome success. Students will develop skills in defining measurable outcomes, aligning KPIs with business goals, and integrating human oversight into AI processes to ensure accuracy, accountability, and ethical implementation.

3. Tools/Resources Needed

• Provided Process Map: A workflow diagram (e.g., from a previous student submission or their own work) in Mermaid or Lucidchart format, depicting the original process and optionally an updated AI-improved version.
• Research Resources: Access to articles, case studies, or industry benchmarks on KPIs, AI evaluation, and human-in-the-loop practices (e.g., via web search or course materials).
• Table Tool: Word, Excel, or similar software to present the KPI framework.
• AI Improvement Context: Knowledge of the proposed AI enhancement (e.g., from prior assignments or a provided scenario).

4. Description of Assignment

Students will use a provided process map (either their own or another student’s from a prior submission) depicting a business process (e.g., customer onboarding, order fulfillment) and an associated AI improvement (e.g., from a previous assignment or a hypothetical enhancement). They will develop a set of at least 5 KPIs to evaluate the process before and after the AI project. Each KPI must:

• Be specific, measurable, and tied to the process (e.g., time to complete onboarding, error rate in order fulfillment).
• Include a rationale explaining its relevance to the AI improvement and business goals (e.g., efficiency, customer satisfaction, cost reduction).
• Describe how it would be measured (e.g., data source, frequency) and provide hypothetical “before” and “after” values. As a key deliverable, students will also address the importance of human-in-the-loop (HITL) evaluation in assessing the success of the AI project. This includes explaining why human oversight is essential (e.g., to catch AI errors, ensure ethical outcomes) and suggesting specific modifications to the AI-improved process to incorporate human review of AI-produced tasks (e.g., a manual check on AI decisions). These suggestions may involve updating the process map to explicitly include HITL steps. The assignment emphasizes aligning KPIs with business objectives while ensuring human accountability in AI deployment.

5. Deliverables

1. KPI Framework: A table or chart listing at least 5 KPIs, including:
   • KPI name and description.
   • Rationale for selection (1-2 sentences).
   • Measurement method (e.g., data source, tool, frequency).
   • Hypothetical “before” and “after” values (e.g., numbers or percentages).
2. Human-in-the-Loop Analysis: A written section (1-2 pages) that includes:
   • Explanation of the importance of human-in-the-loop evaluation for assessing AI outcome success (e.g., accuracy, bias mitigation, trust).
   • Specific suggestions for modifying the AI-improved process to include human review of AI-produced tasks (e.g., adding a human approval step), with justification.
   • Optional: An updated process map (e.g., Mermaid code or screenshot) reflecting these HITL modifications, if applicable.
3. Analysis Summary: A written explanation (1-2 pages) detailing:
   • How the KPIs reflect the AI project’s impact on the process.
   • How they align with broader business goals (e.g., cost efficiency, customer experience).
   • Challenges or limitations in tracking these KPIs (e.g., data availability, human resource constraints).