relationship building with an intelligent agent

overview

problem

It's increasingly common to have ongoing communications with non-human entities. However, "no one wants to have a relationship with a bot."  Or rather, as we approached the problem, under what conditions and circumstances would a human want to work with an intelligent agent?

client

Our confidential client was a global IT company, interested in expanding the capacity of its chat-enabled messaging platform.

concept

We focused on a relationship with constraints to bound our problem, choosing a use-case of an employee onboarding process.  Our HR chatbot, Riley, proactively and reactively helps new employees, building relationships over time by being trustworthy, demonstrating competence, and showing continuity behavior.

my role

On this team project for an Intelligent Interfaces course, I had primary responsibility for the WOZ testing environment, system architecture, and the final presentation.  I also actively contributed to research, personas, scenarios, WOZ testing, and the final prototype.

understanding the user and the problem space

research methods

The literature on bot-human relationships is still new, so we focused on investigating human-to-human relationship building.  We conducted an extensive literature review, but also conducted a focus group.  Adding in the latest research on designs and practices for virtual agents, I started an online affinity map to collect and sort our ideas.  We chose to focus on:

• progression of shared knowledge
• evolving levels of trust, and
• personality attunement over time. 

contextual analysis

Relationships occur in many settings and have complex parameters.  Through a series of sketches and brainstorming sessions, we explored different kinds of relationship roles before finally settling on a personal assistant.  We also mapped key relational stages to help define appropriate bot behavior.

personas - human and bot

To more deeply understand attunement, we reviewed several personality inventories.  We opted to base our default personas on both the DISC and Personality Insight Inventory, often used in corporate team building.  We created abbreviated people personas, then defined more detailed behavioral and conversational attributes for each corresponding "bot-style" persona.

task analysis

We conducted thorough task and goal analyses of the employee onboarding process, identifying ignition points, subtasks and focus points.  We mapped users' motivations, prior knowledge, and emotions.  We also mapped the desired corresponding behavior of the bot, identifying both proactive and reactive tasks for Riley.  Finally, we integrated the relationship stages into our model to mimic the progression of behavior over time.

Reactive tasks include:

• Introduction / capabilities defined
• Ask or answer questions
• Help gain skills or make connections

Proactive tasks include:

• Follow up on outstanding tasks or checklist items
• Assist with goal setting

design process

Scenario and Script Development

We sketched several scenarios where a user might interact with Riley, and then created ‘conversational snippets’ for the bot in both potential proactive and reactive situations.  These snippets were used to test the first prototype, which we conducted as Wizard of Oz sessions.

Wizard of Oz Testing

I set up an online chat environment, and created a user ‘account’ for Riley.  Participants were talked through scenarios by the moderator, and then Riley’s operator responded to their live questions and text using the conversational snippets.   

Prototype

For the prototype, we explored means of wireframing a conversational interface. It needed to illustrate the user’s inputs, system functions, and connections to databases and services. We chose to orient the timeline vertically to reflect the flow of a typical messaging conservation.

Final Architecture

Based on our testing and research, we determined the architecture for a relationship-building bot needs to include elements of context, language, sentiment and personality analysis.  Initial query results are combined to form user-specific variables and so on, eventually determining the style and content of the bot’s response.  The system would likely be a combination of simple rule-based engines as well as a more Bayesian approach to determine the 'most-probable-best' actions and responses as learned for each user over time.

evaluation

Challenges / Limitations

An element that was not easily prototyped was the over-time interaction that is an intrinsic part of relationship building. This is likely to require a test build, or a semi-longitudinal study through which the participants interact with scripted respondents over a period of time.

Next Steps

After collaborating with engineering to determine the feasibility of this approach and making necessary adjustments, the next step would be to tightly define the bot’s scope and functionality and create the robust language database.

Lessons Learned

One of the first things we learned from the WoZ prototype sessions was that the operator needed to be familiar with the sample lexicons as well as the selected communication style so that improvisational responses could be given to non-standard questions. This highlighted the need for a robust language database.

We did see positive responses to the concept, but learned that proactivity must be paired with usefulness in order to build trust and a willingness to continue future engagement.