Appen – web
2022
Contributors management
CONTEXT
Offering precision and flexibility for Data Scientists to manage 700K+ contributors working on AI odels datasets.
Appen Limited specializes in generating and refining data for machine learning and artificial intelligence development. High-quality data annotation is essential for training effective machine learning models.
The company's latest initiative, Quality Flow, is a comprehensive suite of quality management tools and platform enhancements designed to improve usability on the Appen platform. It introduces new management and quality control features to streamline project design and execution.
Quality Flow transforms the way data scientists and project managers manage data, projects, contributors, and performance metrics. As a designer, my primary focus was enhancing the Contributor Management experience to improve efficiency.
USER PROBLEM
Data scientists need labeled data before training a machine learning model, and this is where Appen comes in. The company provides a curated and well-trained pool of contributors to handle this data. However, despite their high level of skill, there must be quality assurance checkpoints to ensure the accuracy and validity of the generated data. Continuous monitoring of worker performance is essential to maintaining quality standards. Currently, the platform experience for managing contributors is time-consuming, inefficient, and lacks key resources.
Project Managers and Data Analysts lack the exactness and adaptability needed to handle a project's participants–the data labelers. How can we simplify contributor management, making it more comprehensive and less burdensome for them as they create the globe's leading Machine Learning Platforms?
Business Goals
Provide Data Scientists and Project Managers with flexibility and autonomy when it comes to crowd management.
Allow users to swiftly move around contributors, relocating them as needed.
Eliminate the use of third-party software, legacy systems, and endless spreadsheet reports.
IMPACT
795,287
Contributors now managed in-platform
~2,000
Projects migrated from legacy platform
98%
Increase in User Adoption
This project was a collaborative effort between me, two other UX designers, and a large engineering team. I was responsible for designing the Contributor Management Experience, leading User Research, overseeing Product Discovery, and delivering high-fidelity designs. I worked closely with PMs and SMEs to define objectives and kickstart the solution design process. Additionally, I led multiple discovery workshops and maintained direct communication with engineers to ensure the accurate implementation of my design decisions.
BACKGROUND
As a Data Scientist or Project Manager, it's essential to regularly monitor detailed statistics on Contributor performance. Let’s explore Lestat’s workflow to better understand how Appen operates.
Consider the scenario: as a PM, Lestat sets up a project to process audio clips from a doctor's appointment. His client, Amazon, requires transcripts with metadata indicating when the doctor and patient are speaking. Given the project's need for high-quality data, Lestat selects Appen’s Curated Crowd managed services. Our trained data labelers will generate the data the client requested. Throughout the project, the PM must continuously track data and evaluate Contributor performance. For instance, if a Contributor underperforms, it’s Lestat’s responsibility to identify the issue and take appropriate action.
Current experience
Currently, Lestat’s workflow would be tricky and inefficient. The project management team must rely on a legacy system to download multiple reports each week to monitor contributors. Additionally, they periodically assess and update manual Excel spreadsheets for each project, making the process even more time-consuming.
Another major issue is the tool’s lack of visibility into contributors working on a project. Previously, users could only assign workers to jobs using broad parameters, leaving them without an accurate list of active contributors. This limitation also restricted their ability to reassign contributors as needed, reducing overall flexibility.
Discovery
The UX design team was in charge of conducting user interviews with subject matter experts and stakeholders in order to understand all facets of the challenge. I worked directly with two managers and two AI professionals to find product requirements.
Interviews
We had a kick-off meeting to understand the following points:
How complex is the management of contributors.
What metrics do users expect to see on a contributor table?
What interactions we should cover in terms of filtering and managing individual workers?
What are the main pain points in their current experience?
How many contributors each user deals with at once?
From these conversations, we learned that:
"My need is to quickly scan contributor performance and be able to manage them according to that.”
“When it comes to management, I'd like to assign contributors to specific jobs. I usually do that when I already know that such a worker performs well.”
“I want to have less clutter in my computer and fewer excel spreadsheets to update. This is time-consuming and super prone to errors. I deal with thousands of contributors allocated in several projects, it's hard to keep up.”
"The platform misses more managing resources to cut down on time and make processes more efficient. I need to manually calculate performance to report back to the client."
DEFINE
Our users urgently needed a single platform to manage worker labeling data for Appen's projects. Since they rely on spreadsheets daily and appreciate their flexibility, we chose a table-based management experience—enhanced with powerful features to streamline their workflow.
How can we incorporate all needed funcionalities into a table without making it overwhelming?
Solution
We knew from the start that we were dealing with a complex feature with a broad range of requirements. The solution was to provide a step-by-step method in which users are guided through the process of creating their crowd and gradually exposed to the crowd management tab. Because this is the format our users are accustomed to, I chose to enhance and keep the spreadsheet experience in the design. To achieve the final product, I held multiple validation meetings with engineers, potential users, and design peers.
Guided experience
Appen’s platform serves different client types, from self-serve users to premium Curated Crowd services. To make this clearer, I used the empty space in the CC tab to highlight Appen’s service options. A bonus? That same space also works as a quick-start guide for new users.
Grouping Contributors
Another major goal of this project was to provide better management capabilities in order to optimize the work. I created a grouping mechanism that enables Project Managers and other users to create worker groups based on filters, performance, or any other criteria. PMs, for example, can cluster high-performing workers and simply assign them to a certain job by assigning the group.
Table Filtering
The Data Science team works with a vast number of workers, with some major projects managing thousands of people. With this in mind, filtering is critical to success. I collaborated closely with others in my team to ensure that the necessary resources were included.
We provided the option to hide or show columns so that users could choose the amount of complexity of information displayed.
Users can modify the width of a column as well as its position by dragging it around.
All columns offer sorting and additional filtering capabilities.
Assigning Contributors
The number one priority was to allow users to freely move contributors within a project. It was critical to have the option of allocating certain workers to specific jobs. With the provided solution:
Users can choose from a list of contributors and assign them to certain jobs inside the project.IIf necessary, they can also remove contributors from a job.
Users can completely remove a contributor from their project without relying on a manager or other superior access.
✹