Efficiently documenting and sharing research insights at ecobee by building an insight database.

Background

As the demand for user research increases at ecobee, the research team found it challenging to consistently document, find, and share research insights across the growing product team. When PMs, designers, or engineers asked “what do we know about X?”, it is difficult for researchers to quickly find relevant research on existing knowledge management platform.

Therefore, I’m tasked with conducting research to understand user needs and creating a research database (named “Insight Portal”) to help my team quickly find research insights.

Process & Timeline

In 4 months, I led the design and research, from creating research plan, interviewing users, designing information architecture, creating prototypes, and conducting two rounds of usability tests to guide design iteration for the Insight Portal.

Let’s zoom in to the final round of rapid evaluative research

Among multiple rounds of foundational and evaluative research, the last round of usability testing generated the most insights to guide the design decisions for Insight Portal because I summarized a set of quick usability fixes for the design, and more importantly, some long-term considerations that would guide future product direction and iteration.

Planning

Research goal

Identify a list of actionable usability issues in the Insight Portal data entry and retrieval experience for different roles (i.e. researcher, designer, product manager). We want to understand these key questions: (a) Can product managers, designers, and researchers easily find existing research information? and (b) Can researchers easily store research findings, session logistics, and participant information?

What are the usability metrics?

Based on Google’s HEART framework of user-centered metrics and Sharon’s concept of Key Experience Indicators (KEIs), which provides a quantitative score of a specific, important, and actionable phenomenon related to using a product or service, it will be valuable to focus on measuring the happiness, adoption, and task success dimensions of the Insight Portal’s user experience in this case.

Participant Sampling

I recruited 5 product team members who need to document and retrieve research insights on a weekly basis. The sample included a mix of participants across product lines, job roles, gender, and familiarity with Insight Portal (previous exposure vs. first time users).

Method

In-person moderated usability tests with 5 team members, in sessions lasting 30 minutes. All had access to the Insight Portal in Airtable.

Hypothesis

1. Users will be curious about different insights across product lines and are eager to explore what insights other teams have discovered.

2. Users would like to find out which specific participant and session contributes to a particular insight.

3. Users would like to explore through the Top Keywords page as a starting point if they don’t have a clear idea of what to search for.

Synthesis

After data collection, I synthesized the data using a rainbow spreadsheet, which is a visual method for data analysis proposed by Tomer Sharon. The rainbow spreadsheet is particularly helpful to synthesize usability findings because its colorful visualization makes it easy to scan behavioral patterns and identify issues. In general, the observations with most colored cells, presented a potential priority item.

In the ‘Category’ column, I’ve assigned different categories to observations, such as ‘Wish List,’ ‘Pain points,’ ‘Pleasure points,’ and ‘Long-term considerations’ to easily scan major themes. I’ve also marked some cells in red to identify key pain points.

Key themes and insights

At the end, there are 4 key themes that stood out from the data.

01

Different roles have diverse needs for content discovery

02

The key is to socialize and educate users about the portal

03

Spreadsheet format is not ideal for insight readability

04

Burden on UXR to maintain the system

Recommendations

Based on the research findings, I grouped a set of issues that can be easily fix (e.g. add different access accounts), have clear design solution (e.g. specific redesign) or can be solved by conducting further research (e.g. card sort). However, there are also some issues that are more complex and needed cross-functional discussions to truly address, which I grouped as ‘Long term considerations.’

Quick usability issues to fix

Bonus feedback with Tomer Sharon

After the Insight Portal has been created, I reached out to Tomer Sharon, the creator of the Insight Nuggets and Airtable research database concepts, and hope to consult him with some of the trickier problems identified from the research.

Luckily, I was able to scheduled a call with him and he kindly gave additional recommendations and feedback for how ecobee can further refine the Insight Portal.

✉️ Contact me to learn more about my experience at ecobee