.webp)
“Creativity is the unfiltered expression of who we are. There’s no workaround for true creative diversity in training data,” says Jeremy Tennison, an Operations Manager at Invisible.
In true form as a data-driven ops guy, Tennison has proof to back up his assertion. Early during his ongoing role at Invisible, AI researchers developing a sales chatbot set out to identify scenarios where their model was using unethical persuasive tactics. To run the study, they asked a small team of Invisible’s AI trainers to act as a number of different characters, each with their own motivations and priorities, and had the chatbot try to sell them a product.
The research team quickly found that this was an ineffective way to set up their experiment — each trainer tended to gravitate towards their own perspectives even when playing a character. Having a small group of humans attempt to mimic the vastness and diversity of people in the real world was simply ineffective.
Too often, AI researchers request a large human dataset from a training data provider, only to find that the dataset they paid for has little positive effect on their model. It’s not hard to understand why: most training data providers pay their trainers per task, so they’re incentivized to complete them as fast as possible, rather than prioritizing quality or creativity. These prompt and answer factories tend to reduce acts of creation such as writing, illustrating, design, etc. to rote, assembly line-style tasks, leading to uninspired and burned out creatives that generate lower quality human data.
Tennison, who heads a team of around four hundred creative writing data trainers at Invisible, strives to do the opposite. He has built a warm, kind, and encouraging team culture that inspires creatives to do their best work — even when they’re writing under tight deadlines and get tracked on everything from word counts to tasks completed.
In this article, we’ll uncover some of his tried and true practices for building and leading creative data training teams that deliver high quality work.
How we screen data trainers for creative projects
AI training data providers often get thousands of applications per project, and those requiring creatives have the additional challenge of ensuring that each individual can produce high volumes of high quality, creative, and diverse work.
To build a creative writing workforce at Invisible, Tennison’s team set up a screening questionnaire that prioritized passion for writing over typical interview questions about work experiences. The questionnaire asked each applicant to describe:
- What they liked about writing and why they were passionate about it
- What forms of writing they practice, either professionally or personally
The team then selected applicants whose responses were especially enthusiastic, resulting in a group of four hundred writers who can not only meet the volume demands for training data, but deliver on quality, creativity, and variety as well.
“I take a lot of pride in my work. Everything I touch, I want it to be good,” says Roksana Kasprzyk, an Editor on Tennison’s project.
Tennison also hired around fifty quality analysts to the team, choosing them for their ability to balance teaching with enforcing quality standards and inspire confidence even while giving constructive feedback.
“I make a point to acknowledge that everyone makes (mistakes), especially when you’re working on a deadline. I always reiterate how much the writer has already accomplished and try to alleviate some of the pressure that comes with the metrics they’re being tracked on by reminding them that their work is always judged holistically,” says Cole De Monico, a Quality Analyst on the team.
How we build and maintain a culture of creativity
About half of Invisible’s creative writing workforce has done data training work before at other data providers. Their experiences at these “content factories” left them burned out from days of writing spent prioritizing speed and efficiency over creativity.
At Invisible, things are noticeably, purposefully different. “There are a lot of people who assume that creating a culture of warmth and openness must mean that some compromise has been made in efficiency and output. In my experience, it’s been the opposite,” says Tennison.
Tennison has built a whole brand around his team and the creative project they’re working on, prioritizing kindness, resourcefulness, integrity, and fun. The team’s internal website, which houses everything from project details to writing inspiration, is space-themed, with each squad assigned a different star or planet.
Squad leaders, who each head a team of around thirty writers, are chosen specifically for their palpable kindness, their ability to energize and inspire others, and their enthusiasm for the project as a whole.
“When I started on this project, I was excited about what we were helping to build. My excitement for the product — the model we were creating data to train — came through,” says Tennison. He looks for the same passion for AI in his squad leaders, who are responsible for inspiring their teams as well as keeping them on track for quality and efficiency.
How we unblock creative work with collaboration and feedback loops
The openness and warmth of the team culture enables some of the most important aspects of creative work: overcoming creative blocks and improving data quality. Writers and editors on the team are comfortable enough with their leaders and quality analysts to ask for help when they need it.
“When I’m feeling burned out, I can turn to the team for help. I can always say, ‘I have writer’s block,’ and my squad lead or a QA will hop on a call with me, sometimes even working through a task with me if I need it,” says Kasprzyk.
On the opposite end, leaders are open to feedback. “Sometimes when they push back (on negative feedback), they’re right! And I accept that. Making sure the work is accurate and high quality is more important than being right,” says De Monico.
This collaborative and open environment extends not only to the data training team, but to the AI researchers on the client side as well. Over the course of months, Tennison has established a streamlined system to bubble up feedback and questions from data trainers and analysts, usually collected from team meetings and Slack conversations, to the AI researchers during regular reporting meetings.
“One of the biggest advantages we have in this project is the true collaboration we have with the researchers, engineers, and the Chief of Staff on our client’s team,” says Tennison.
The ability to ask for specific context from the researchers and contribute feedback on content and training tasks gives trainers a sense of pride and ownership over the project as a whole, further inspiring each individual to deliver their best work.
How we changed workflows to prioritize quality and efficiency
While his team of creative data trainers produce high quality work, Tennison keeps a careful eye on the data operations, looking for opportunities to increase efficiencies without impacting productivity.
For example, when the team was first put together, they operated on a system that assigned two trainers per data row. One trainer would draft content according to the prompt, and an editor (often a native or highly educated English speaker) would then edit the draft. After several months with this workflow, Tennison delved into the tracking data they had collected throughout the operation and found that those in the drafting role:
- Had become far better at their tasks with a few months’ experience on the project
- Felt an increased sense of ownership over their work than editors did
While it wouldn’t have been efficient to start off with a single-trainer-per-data-row process while trainers were still learning the task and deepening their (extremely comprehensive) style guide, the data showed that their jobs had clearly become easier over time. Tennison had his team move to a single-step process where all trainers were drafting content, boosting both efficiency and the team’s sense of pride in their work.
Whether your team is building generative AI to aid filmmakers in editing, help academics produce better research papers, or support sales teams to win over prospects with compelling content, you’ll need to train your model on human data generated by creative people.
High quality human data for language models starts with teams of experienced, inspired humans who are passionate about training AI on their work.
Talk to Invisible today to learn what our creative data trainers can produce for your AI project.
