PineCones
An online starred problemset for saving and reviewing mistaken problems.
Overview
Time
Jan 2020 - Aug 2020
Client
Squirrel AI by Yixue Group
My Role
Product Management, Research Lead
Keywords
EdTech Tool Design, Online Learning, Product Management
Teammates
Miaojun Xu (Project Lead), Yuchen Xie (Design Lead), Jiachen Gong (Development Lead)
My Contribution
Led the team in identifying possible problems through qualitative user interviews and quantitative metrics analysis.
Collaborated with design lead for ideating possible solutions and prioritizing solution features. Evaluated and iterated design solutions based on user testing and surveys.
Defined solution features and user stories for handoff to our client's dev team.
Problem
How Might We help students learn from their most important mistakes by categorizing, organizing, and analyzing them?
Our client, Squirrel AI, offers an online learning platform powered by AI in China. The Squirrel AI online learning platform has a feature called Mistake Analysis System where the students can do Mistake Analysis Related Study (MARS) like correcting and analyzing mistakes and reviewing related knowledge components (KC's). Our project aims to improve the learning experience of the Mistake analysis system.
Target Users
Students in grades 7-12 who take part in the after-school learning program at Squirrel AI to review learning concepts and practice additional problems on the online adaptive learning platform.
Solution
A personalized collection of important mistaken problems with intelligent support for reflection and review.
The solution we proposed is the Starred Problem Set feature to be hosted within Squirrel AI's MARS system. Students can filter the mistaken problems that are important to them and annotate them with procedural and content scaffolds for future review.
■ User Story 1: Adding Problems to the Starred Problem Set
As a learner, after each lesson, I want to categorize and annotate my most important mistaken problems, so that I can review them later.
Add problems and categorize them by importance and difficulty from the MARS system
Highlight important concepts in the problem, add comments with concept definitions autofilled, and ask teachers
Reflect on problem solving strategies by annotating on images
Submit annotated problems for future review
■ User Story 2: Reviewing Problems in the Starred Problem Set
As a learner, before exams, I want to filter out the relevant problems with my notes, so that I can review them and practice for the exam.
Toggle On/Off annotations for different review purposes
Interact with the teacher for feedback synchronously or asynchronously
Evaluation
To measure the success of the design, we conducted surveys with our users regarding the following aspects:
- Benefits towards learning
- Efficiency during the annotation process
- Communication with teachers
- Difficulty level in reviewing problems
Design Process
Research Phase
■ Preliminary Research
At the beginning of our project, we conducted a series of preliminary research, including product analysis, stakeholder mapping, and competitive analysis, to find the focus of our project and identify potential stakeholders for interviews.
Based on the results of the preliminary research, we identified Mistake-Analysis Related Study (MARS) as our main focus. Because it has a relatively low opportunity cost and little interference with the main system, but it can have a high positive impact on the students' learning experience and their user experience.
■ Understanding the MARS Learning Process
To understand the essential steps of the MARS learning process, we conducted semi-structured interviews with 4 expert students and 6 learning experts (SMEs).
Based on the interviews, we identified the four main steps of the MARS learning process:
We found that the current online MARS system does not support the full MARS learning process - the selection and annotation/analysis steps are missing.
■ Understanding User Needs and Challenges
To understand the needs and challenges of the primary users of the online learning platform, the students and teachers, we interviewed 32 students and 7 teachers. We also examined the feature usage metrics in the MARS system.
Based on synthesized insights from affinity diagrams, we built identity models and experience maps to illustrate the user needs and challenges, which point to the potential areas of improvement for the online MARS system.
■ Synthesizing the Main Problems
From feature usage metrics and user interviews with students and teachers, we can synthesize the results into the following main problems:
* Hover over card to see evidence
The current system does not support the full MARS learning flow
From SME Interviews:
The current system misses the critical steps of selection and annotation/analysis in MARS learning.
Students lack internal & external motivation for doing MARS
From Student Interviews:
Using MARS as a formality of homework can easily lead to a lack of motivation for the students.
Students can't see the value of MARS features due to limited metacognitive abilities
From Usage Metrics and Teacher Interviews:
Students don't know how to categorize and analyze their mistakes without scaffolds.
The current system cannot provide good online-offline integration
From Student and Teacher Interviews:
Students have difficulty accessing the system at school. Thus, they cannot take full advantage of the system.
Design Phase
■ Ideation
Based on the main problems we've identified, we came up with the following "How Might We" statements to guide our ideation.
How Might We
- Implement the steps of selection and annotation in an online environment?
- Provide additional values other than homework practice in MARS?
- Scaffold students' metacognitive processes of categorization and analysis?
- Enable students to take advantage of the system both online and offline?
Based on our HMW statements, we generated over 40 initial ideas through Brainstorming. We then evaluated those ideas in , speed-dated them with 11 users and clients to get feedback, and finalized our idea considering design, business, and implementation constraints.
Our final design direction is to create a new feature within the online learning platform to support the students in
- selecting their most important mistaken problems using difficulty and importance levels and custom labels
- analyzing their selected problems with highlights and annotations
- receiving feedback from teachers through text comments
- reviewing and practicing problems online or in printed formats
■ Prototyping
Using wireframes and low-fidelity prototypes, we tested our idea in 3 rounds of user testing with 16 users to further flesh out the specific features for the Starred Problem Set and iterate on the learning experience and user experience.
Based on the testing of our prototype, , the final design has the following user stories for the sub-features:
■ Final Design
Here is a prototype of our final design, feel free to try it out. (It's better to be viewed fullscreen)