PineCones

An online starred problemset for saving and reviewing mistaken problems.

PineCones Project Image

Overview

Time
Jan 2020 - Aug 2020
Client
Squirrel AI by Yixue Group
My Role
Product Management, Research Lead
Keywords
EdTech Tool Design, Online Learning, Product Management
Teammates
Miaojun Xu (Project Lead), Yuchen Xie (Design Lead)
Jiachen Gong (Development Lead)

My Contribution

Led the team in identifying possible problems through qualitative user interviews and quantitative metrics analysis.

Collaborated with design lead for ideating possible solutions and prioritizing solution features. Evaluated and iterated design solutions based on user testing and surveys.

Defined solution features and user stories for handoff to our client’s dev team.

Problem

How Might We help students learn from their most important mistakes by categorizing, organizing, and analyzing them?

{{ projectBackgroundButton }}

Our client, Squirrel AI, offers an online learning platform powered by AI in China. The Squirrel AI online learning platform has a feature called Mistake Analysis System where the students can do Mistake Analysis Related Study (MARS) like correcting and analyzing mistakes and reviewing related knowledge components (KC’s). Our project aims to improve the learning experience of the Mistake analysis system.

Target Users

Students in grades 7-12 who take part in the after-school learning program at Squirrel AI to review learning concepts and practice additional problems on the online adaptive learning platform.

Solution

Concept Video

A personalized collection of important mistaken problems with intelligent support for reflection and review.

The solution we proposed is the Starred Problem Set feature to be hosted within Squirrel AI's MARS system. Students can filter the mistaken problems that are important to them and annotate them with procedural and content scaffolds for future review.

User Story 1: Adding Problems to the Starred Problem Set

As a learner, after each lesson, I want to categorize and annotate my most important mistaken problems, so that I can review them later.

User Story 1 Flow

Add problems and categorize them by importance and difficulty from the MARS system

Highlight important concepts in the problem, add comments with concept definitions autofilled, and ask teachers

Reflect on problem solving strategies by annotating on images

Submit annotated problems for future review

User Story 2: Reviewing Problems in the Starred Problem Set

As a learner, before exams, I want to filter out the relevant problems with my notes, so that I can review them and practice for the exam.

User Story 2 Flow

Toggle On/Off annotations for different review purposes

Interact with the teacher for feedback synchronously or asynchronously

Evaluation

To measure the success of the design, we conducted surveys with our users regarding the following aspects:

  • Benefits towards learning
  • Efficiency during the annotation process
  • Communication with teachers
  • Difficulty level in reviewing problems
Survey Result

Design Process

Design Process

Research Phase

Preliminary Research

At the beginning of our project, we conducted a series of preliminary research, including product analysis, stakeholder mapping, and competitive analysis, to find the focus of our project and identify potential stakeholders for interviews.

{{ preliminaryButton }}

Based on the results of the preliminary research, we identified Mistake-Analysis Related Study (MARS) as our main focus. Because it has a relatively low opportunity cost and little interference with the main system, but it can have a high positive impact on the students’ learning experience and their user experience.

Focusing Direction
Understanding the MARS Learning Process

To understand the essential steps of the MARS learning process, we conducted semi-structured interviews with 4 expert students and 6 learning experts (SMEs).

Based on the interviews, we identified the four main steps of the MARS learning process:

MARS learning flow

{{ flowButton }}

We found that the current online MARS system does not support the full MARS learning process - the selection and annotation/analysis steps are missing.

Understanding User Needs and Challenges

To understand the needs and challenges of the primary users of the online learning platform, the students and teachers, we interviewed 32 students and 7 teachers. We also examined the feature usage metrics in the MARS system.

Based on synthesized insights from affinity diagrams, we built identity models and experience maps to illustrate the user needs and challenges, which point to the potential areas of improvement for the online MARS system.

Student Model
Teacher Model

{{ teacherFlowButton }}

Synthesizing the Main Problems

From feature usage metrics and user interviews with students and teachers, we can synthesize the results into the following main problems:

* Hover over card to see evidence

The current system does not support the full MARS learning flow.

From SME Interviews:
The current system misses the critical steps of selection and annotation/analysis in MARS learning.

Students lack internal & external motivation for doing MARS

From Student Interviews:
Using MARS as a formality of homework can easily lead to a lack of motivation for the students.

Students can’t see the value of MARS features due to limited metacognitive abilities

From Usage Metrics and Teacher Interviews:
Students don’t know how to categorize and analyze their mistakes without scaffolds.

The current system cannot provide good online-offline integration

From Student and Teacher Interviews:
Students have difficulty accessing the system at school. Thus, they cannot take full advantage of the system.

Design Phase

Based on the main problems we've identified, we came up with the following "How Might We" statements to guide our ideation.

How Might We

  • Implement the steps of selection and annotation in an online environment?
  • Provide additional values other than homework practice in MARS?
  • Scaffold students’ metacognitive processes of categorization and analysis?
  • Enable students to take advantage of the system both online and offline?

Based on our HMW statements, we generated over 40 initial ideas through Brainstorming. We then evaluated those ideas in a difficulty & importance matrix, speed-dated them with 11 users and clients to get feedback, and finalized our idea considering design, business, and implementation constraints.

Difficulty & Importance Matrix
Narrow down ideas

TO BE CONTINUED