Visual design by the very talented Sapna Patel.
watson orchestrate
Watson Orchestrate is a digital worker offered through IBM Business Automation. Powered by AI, it carries out business processes like a colleague, in partnership with human team members. It creates automations, offloads menial tasks, and connects to apps like SAP, Slack, and Workday.
The chat UI uses Constrained Natural Language (CNL) to match user requests to relevant skills, which fulfill those requests. CNL follows a strict grammar—the language engine was not yet able to parse natural human language.
My role: UX designer
My team: Sapna Patel, visual design; Hunter Hobbs, UX design
Skills honed: AI, conversational design, chat UI, user onboarding, UX writing, digital employees
Status: Handed off to development, later redone
Screens showing how a user can reassign an image approval task using Watson Orchestrate. This is an example of an invocation request.
Project Overview
What is the future of work?
From the original project presented at the IBM THINK conference. Visual design by Kylee Barnard.
Watson Orchestrate began as a presentation at the 2021 IBM THINK Festival, imagining how human workers interact with digital workers in the future. My team inherited the project to turn it into a reality.
project goals
Business goal: Demonstrate the power of AI and how it can integrate with any team’s existing workflows, positioning IBM as a key player both in automation and in the productivity tool space
User goal: Create an interface that empowers the user to work more efficiently, offloading menial tasks and saving time
project constraints
Watson Orchestrate can only communicate through Constrained Natural Language (CNL)
“Digital employees” are not commonplace, requiring a better definition of what Watson Orchestrate is and can do—and not do
Research
understanding the user and our competition
Meet Cassie.
Cassie is a low-code taskworker in HR at a medium-sized company. She deals with a dozen different tools like Slack, email, Salesforce, Workday, and Excel. Overwhelmed, she needs a tool that streamlines her work—but doesn’t require extensive training.
Often, her greatest enemy is time.
Chat UI research
The team spent weeks conducting researching on chat UIs and their advantages and disadvantages, as well as Slack and Microsoft Teams for future integrations.
The main disadvantages are:
a clumsy interface for larger projects involving multiple people
a “psychological pull” that interrupts productivity and, in some cases, can be addictive
increased cognitive load due to frequent context switching
The main advantages of chatbots and messaging interfaces are:
an ideal interface for quick, ad hoc interactions
a history-at-a-glance of previous conversations, especially in chronological order
a means of communication for remote work among teams
A snapshot of the pros and cons of messaging UIs, featuring work done by the IBM Research team. Click to see larger.
Users decide to automate or offload tasks based on five factors:
Consequences associated with the task. For tasks involving sensitive information (like SSNs, passwords, tax info, etc.), users would rather have more oversight, as errors can have legal consequences.
Familiarity with the task. If the user is less familiar with a task and how it works, they would prefer more updates and details at each step.
Collaboration involved. If a task involved bringing in another person, especially a customer, users preferred to do it themselves. This includes establishing relationships with clients or having face-to-face follow-ups.
Trust towards system. Users not yet comfortable with a system are more likely to monitor it closely and review its output before submitting any work.
Individual characteristics. Some people are excited about AI and will automate anything they can! Others, not so much.
Design
conversational ux and interactive UI
The user journey of how Cassie creates an approval, using the Approval skill.
The desktop user experience of Watson Orchestrate had three main areas to create: the conversational UX in the chat, the non-chat interactions in the UI, and user onboarding.
cONversational UX - the chat experience
To design the conversational UX of Watson Orchestrate, we broke down all the different types of requests and questions a user might input:
Next, we had to break each category of requests into specific patterns—what the user might say, how Watson Orchestrate (WO) would respond, what the user might say next, etc. After further research and iteration, we determined that all Watson Orchestrate chat interactions fall into five categories (click to see larger):
Welcome messaging: WO would greet the user or welcome them into the product, either for the first time or at the start of a session
Invocations: The user commands WO to do something, like setting a reminder or scheduling a meeting
Questions: The user asks for information from WO
Error handling: If WO is unable to carry out a request, how does it notify the user and correct the issue?
WO-initiated actions: WO reaches out to the user, such as when they receive a request from someone else or when a reminder is scheduled
Using IBM UX writing guidelines, we developed a “voice” and personality for the chat, using branding archetypes. You can find more on Watson Orchestrate’s voice here.
non-chat interactive UI - the editor panel
To enhance the experience—and increase user trust—we included an editor panel on the right that can display documents, information, and interactive forms. The editor panel allows the user to accomplish what can’t be done in the chat.
Low fidelity designs showing examples of what the user can accomplish in the editor panel
Low fidelity designs showing how a user creates an approval in the chat—then inputs details for the approval in the editor panel.
The right panel is fully interactive and can do everything from sending emails to launching approvals to creating automations.
Next, we faced are our biggest challenge: The conversational engine behind Watson Orchestrate had a limited grammar that it could understand. Through user onboarding, we had to teach users how to write requests for Watson Orchestrate.
User testing
validating an onboarding experience
Our first step was to determine an underlying grammar for requests using Constrained Natural Language (CNL for short)—a limited version of Natural Language. I looked at sites like Duolingo and Code Academy for effective ways to teach users and came up with this basic structure:
This basic structure was the result of our research and our slight panic that we were informed of this limitation only four weeks before our deadline.
We undertook a study on best methods for language guidance and sought to answer:
How quickly do users pick up CNL and use it correctly?
What method of guidance best teaches CNL?
What are users’ natural, intuitive syntax and phrasing? And how do we emulate that with our own CNL?
We made low-fidelity mockups and tested three methods: no guidance, implicit guidance (where the user could access help if needed), and explicit guidance.
Click to view larger. Which one do you think is the most helpful?
Users were asked to complete one of three tasks: schedule a meeting, create an approval, or set a watch (a type of automation). We conducted a total of 20 unmoderated, think-aloud tests (2 rounds of 10 tests) via UserTesting.com. The 20 users spanned 11 countries, ranged from 25 to 49 years old, and worked in a variety of industries.
Results from our user testing. A majority of users preferred the right side guidance (aka the explicit guidance).
Highlights of our findings:
Only 8 of 20 were able to use the CNL correctly after reading guidance
The explicit guidance, where the grammar and examples were shown on the right panel in the UI, had the best success rate
Despite 10 of 20 users stating a preference for the implicit guidance, 0 of 20 actually read it
Slack, followed by Microsoft Teams, are the most-used messaging apps
Not a single user used CNL correctly on the first try
My favorite insight: Users were polite to the chat! 10 of 20 users included a “please” in at least one of the requests
Screenshots of user responses during the test. 3 of 20 users suggested we include a voice command, like Siri.
Users were also asked what they would and wouldn’t use Watson Orchestrate for.
Tasks that users would not use WO include:
Any task that is customer-facing
Tasks of high importance with little room for error, or tasks involving confidential information
Higher-level creative tasks like building a marketing campaign
Tasks that users would use WO include:
Approvals and forms requiring submitting
Setting reminders
Simple, repetitive tasks
With this new research in mind, we set about adapting the Watson Orchestrate onboarding experience and chat interaction, with a focus on Day 1, first-time use.
Final Design
welcome to the future of work
Our final opening screen. “HiRo” is an HR-specific iteration of Watson Orchestrate. Visual design by Sapna Patel.
Through our research, we determined that task tracking and the “watch” feature (though the term watch is still under iteration) would be the most common things users might want to do with Watson Orchestrate. Over time, however, that welcome message would instead include the tasks the user does most often.
saving time
Cassie can use Watson Orchestrate to send approvals, create reports, integrate tools, and track her tasks. These are examples of invocation requests.
She can offload her menial tasks with her own digital business assistant!
everything you need, in one place
Her “digital worker” can integrate with her existing apps, such as Workplace, an IBM productivity tool.
workflow integrations
Watson Orchestrate offers a Slack integration, where users can communicate with the digital worker as they would a DM.
Note: The use case presented here is an insurance claims use case, as requested by product management.
Then came great news: Less than a month after our product release, we were informed that Watson Orchestrate would be using conversational patterns from a larger language base, and would be able to parse far more than our original limits—our concerns about CNL were no longer worries!
A week after that, we found out this entire project would be absorbed into a new version of Watson Orchestrate, but that’s another story.
A brief review
What i learned
Every project and team teaches me something new about design. Here’s what went well, what went wrong, and what I learned while working on Watson Orchestrate:
Nothing teaches you more about a process than doing it out of order. Up until this project, most design work I’ve done has followed a consistent pattern of Research → Iteration → Testing → Final Design. This project was a ready-made concept that instead needed to be validated, so we came up with more generalized concepts and tested them, gathering evaluative research in the process.
Conversational UX requires looking at a qualitative experience through a quantitative lens. How do you pull underlying patterns out of a two-person back-and-forth? How do you account for all the different ways a person can phrase a single idea? It is very, very difficult to think systematically about something as organic as human conversation. But developing chat UX requires understanding the structure of how people communicate.
You cannot box users in, no matter your product’s limitations. Our tango with Constrained Natural Language (CNL) was a prime example of how if a product does not match user expectations, people simply will not use it. CNL is tricky to grasp and not intuitive, which would have turned users like Cassie off of Watson Orchestrate. Thankfully, the language engine is getting an overhaul, and Cassie will be able to communicate with future Watson Orchestrate freely.