Zipper

An assessment tracking tool for teachers in Project-Based Learning (PBL) classes.

Design for Education

iPad App

Client: Winchester Thurston High School
Duration: 3 Months | Sep.2018 - Dec.2018
Team: Angie Wang, Greg Bunyea, Zeqian Peng (UX Research, UI Design and Interaction Design)

Problem Space

How to overcome the difficulty of assessing students in PBL (Project-Based Learning) classes due to insufficient evidence and the dilemma of balancing between traditional assessment and mastery learning?

Client Background: Winchester Thurston School (WT), an independent school in Pittsburgh, is in a unique position to facilitate Project-Based Learning (PBL) courses. City as Our Campus is an initiative that brings community-based learning to provide authentic learning experiences for students from diverse backgrounds and academic interests.

Solution

From our user study we identified following problems:

  • Difficulty of assessing students' competencies in an open learning environment
  • Difficulty of keeping evidence of students' performances
  • Difficulty of prompting students to reflect

To Tackle with the challenges, we designed a native iPad App featured with:    

  • Sliders to assess customizable competencies  
  • Multimedia note-taking for instructors to document students’ performances 
  • Shareable and editable notes
  • Evidence tracking calendar
  • Bi-weekly student and class report for reflection

Our solution was presented to the principal and faculties in WT high school and was highly recognized.

01

User Study

To better understand students’ experience, expectations, instructors’ pain points and also the interaction between students and teachers, we conducted 3 contextual inquiries with PBL “experts”, 2 stakeholder interviews with the director of City as Our Campus program and an instructor from “Research Science” class, and one class observation.

What is the current experience for students in PBL classes?

Contextual Inquiries

  with 3 PBL "expert" students

In order to learn students’ experience, expectation and behavior in PBL classes, we targeted at 3 PBL experts and conducted contextual inquiries with them to observe their work style and inquire information.  We created user profile boards, used affinity diagram and identity model (see them here) to synthesize our interview data and find some common mentality of expert students:

What are the current challenges in implementing PBL classes?

Stakeholder Interviews

  with the director of the program and a PBL class teacher

Interview 1

Adam Nye, 

Director of City as Our Campus

We discussed the implementation challenges of PBL in formal education:

  • Formative and summative assessments are often not standardized
  • Grading inconsistencies among groups
  • Difficulties of demonstrating student performance to outsiders
  • Insufficient impact on the community
  • "We are also working on mastery transcript to address this issue"

Interview 2

Graig Marx, 

Instructor of "Research Science" class

We learned a lot about the instructor’s role in Research Science, which is incredibly central. In order to foster a more open PBL environment which is highly attracted to students, the instructor has no syllabus, seldom use of rubrics, and few assessments without specific criteria. Thus he has difficulty determining the final grades for students due to lack of evidence, problems anticipating which student groups will get farther in the design process, insufficient formal feedback and no systematic process for evaluating student learning.  

We consolidated two interviews and identified the current problems, causes, consequences and potential solutions in PBL assessment by using diagnostic mapping technique:

How do teachers and students interact during PBL classes?

Class Observation

  in "Research Science" class

To understand the atmosphere in PBL classes, the way student groups work together, interactions between the teacher and students, and also how teacher observes, documents students' behavior, we conducted a class observation in the "Research Science" class taught by Mr. Graig Marx. To analyze the data we observed in class, we created a simple graph to manifest what we observed:

Observations

  • The attention given to each group by the instructor is uneven
  • The current way that the instructor keeps evidence is by writing on a notepad which in later talk we learned that it is hard to keep track of and hard to refer back
  • The instructor noticed some of behaviors but did not take it down on the notepad
  • The time that the instructor took notes while interacting with students is very limited
  • The instructor intended to send the students some information but forgot later

What we gained from the user study...

Open Nature

 vs

 Structure

One of the core values of PBL classes is the open nature of it: students can select their project topic, most of the design decisions are open-ended, there is no fixed schedule etc. Through our user study, we found that the tension between open nature vs structure is a common case and cause for most of the assessment problems:

02

Ideation

We learned that the open nature of the class makes it hard for the instructor to use traditional structured pedagogy to do assessment. In order to help teachers in this class, how can we better facilitate such open-ended project-based class to make it somehow more structured but also not to break the balance and openness of the class?

How did we generate and convey our ideas?

Brainstorming and Storyboarding

To address the problems identified in the first phase, we conducted brainstorm session to create How Might We (HMW) notes, to better understand current challenges in the class and also what are the possible ways to solve them. After that, we discussed and voted to narrow down our potential solutions into 15 scenarios and made them into storyboards.

How did we validate users' needs and consolidate our solution?

Speed-Dating

After creating storyboards, we conducted Speed Dating sessions with instructors to validate whether the needs depicted in the storyboards are real needs of them. The validated needs and potential solutions are listed as follow:

A New Way (Sliders)
to assess students’ competencies

Paper Trail
to document students’ performance

Bi-weekly Student Report
to help students reflect

Calendar-based Evidence Tracking
to help trace back past notes

Final Grade Suggestion
based on students' performance

Setting Milestones
to add more structure to the class

Consolidating ideas into a journey!

User Journey Map

According to the feedback we got from the Speed Dating session, we synthesized potential needs and feature into a more integrated solution:

  • A native iPad app featured with Apple Pencil to replace the notepad that teachers can use to take notes
  • The instructor can use sliders to assess on students or groups' performances, competencies and mastery learning progress
  • Student will receive report of their learning path periodically
  • Instructor can use the app to track back to any of the notes that was taken down before

03

Prototyping

Based on what we learned from user study, speed dating session and the final user journey map, the idea of making a native iPad app to replace the notepad had come to a stage that can be prototype to better validate the solution. In order to do that, we used both paper and digital prototype technique to conduct two filed tests with the instructor.

How did we make our ideas come true and test them?

Paper Prototyping & Wizard of Oz Testing

In order to test our idea in a fast way, we used pen and paper to build our prototype and conducted Wizard of Oz testing to get feedback:

Field Test 1

What we learned from the first round of field test:

  • Multimedia should be supported in note taking
  • Competencies should be customizable for different contexts
  • Student should reflect on current performance instead of past performance
  • Consider Traditional assessment incorporation

Digital Prototyping & Scenario Walkthrough

According to the feedback we received from the first field test, we put our design into a digital format including a digital wireframe and a dynamic inVision prototype. To test on how the instructor would use the product in his daily scenarios, we designed some situations for the field test 2 to observe how the app can be helpful and what are the things that need to be improved.

Field Test 2

During field test two, we asked Mr. Marx, as the participant, used our digital prototype to test some of the scenarios we created for him, including:

Scenarios

1. Answering questions from parents about student behavior.
2. Explaining grades to students
3. Dealing with student who has not been assessed for over a week
4. Students explained the behavior to the instructor for some misunderstanding

After the scenario walkthrough, we validated most of the features we ideated but also found following elements to be improved:

  • Share notes with students and parents
  • Ability to edit past notes and assessment
  • Using color to indicate performance and assessment

04

Final Design

After user study, ideation sessions and user testing, we selected several core features to focus on, while other needs are still be potentially considered in future development. The final design was presented to the principal and faculties in the WT high school and was highly appreciated for our right-on-spot, light-weighted and user-friendly design.

  

Smart Group Order

  • Clear and simple group layout
  • Smart and customizable group order

  

Slider to Assess Compentencies

  • Easy-to-use sliders to assess competencies
  • Color implication for performance

  

Multimedia Note Taking and Sharing

  • Shareable and editable notes
  • Multimedia input

  

Evidence Tracking Calendar

  • Icons and colors indicate changes
  • Editable notes and sliders

05

Post Mortem

Throughout the whole process of designing the product, we all learned a lot about the challenges in PBL education and how schools are making efforts to cast impact on the community. After the project, we also reflected on ourselves about what did and did not go well and what is the future work as our post mortem.

What went right?

  • Teacher Centric approach is a strong point of this design. We found a way to avoid student buy-in and allow students to continue working without interruption while scaffolding the teacher to provide better evidence collecting.
  • We did some great need-finding and narrowing which helped improve our design and reduce feature-bloat.
  • Our co-design approach facilitated an in-depth understanding of the problems, opportunities and possible solutions for Mr. Marx’s class.
  • Some of our ideas were novel and meshed well with the opportunities we identified. We received a lot of support for our sliders and how they balance subjectivity and transparency.

What could be better?

  • Improvements to our research process, planning, and organization. We came into most user research sessions with Mr. Marx with a plan to test and get feedback, but we often did not have a research model in mind ahead of time. As a result, the data required more involvement to synthesize.
  • In future work, we would need to revisit some models and spend more time consolidating the data we do have. Our identity model in particular needs more synthesis to become something really solid. Overally, more organize and tighter research plans would have been the biggest change to our process we would make. Doing rigorous background research throughout the process would also have benefited us.
  • One of our biggest challenges was subjectivity in grading for competencies. Part of the challenge was how Mr. Marx felt about his course in that he feels rubrics do not really apply.

Thank you for reading!

You might also like:

ScotBot

Your personal transit buddy at CMU

Pet-ential

Matching platform for pet adoption

Interested in working with me?

pzq0127@gmail.com

   ⫶      ⫶