1. Understanding user contexts with Digital Use Profiles

2. User interactions: Unguided prompting

The toolkit

HOW TO USE THIS!

4. Exploring futures of gender bias or its absence in localised LLM use cases : Speculative Cards

3. Gauging attitudes towards gender bias, and testing those against LLM applications: Predicting & Reflecting on Culturally-Specific LLM Interactions

2. Understanding prompt-generation and getting feedback from users on scenario-specific usage of LLM applications: Guided prompting with Life Journey Scenarios

1. Understanding user experiences with technology and chatbots: Unguided Prompting & Chatbot Interface “Skins” Testing

The Team

Research Deck Development Team: Meredith Stinger, Rimjhim Surana, Dhaval Kothari, Yuvraj Jha

See the larger project from Digital Futures Lab and the findings from this research at:

Our user group were low-resource users, non-English speakers, and had never interacted with an LLM chatbot from Mumbai, Bangalore and Goa. They varied across gender, profession, education level, age and digital literacy.

Context

The research aimed to uncover challenges and opportunities in deploying LLMs in diverse linguistic and cultural contexts, thereby contributing to the development of more inclusive and responsible AI.

From Code to Consequence: Interrogating Gender Biases in LLM…

These cards were developed as a part of an exploration into possible tools to explore gender and language in LLM development and use.

User Research Toolkit

A diary study approach with users’ own devices could yield more naturalistic insights. High levels of hesitancy were observed among some participants, particularly those with lower digital literacy levels.

The activity highlighted the sensitivity of LLMs to prompt quality, spelling and using context in prompts

The open-ended nature of the activity highlighted varying user experiences

The activity effectively gauged initial user comfort levels using LLM-powered chatbots.

REFLECTIONS

Understanding user experiences with technology and chatbots

INDEX:

Exploring futures of gender bias or its absence in localised LLM use cases

Mix and match the cards, draft new scenario prompts to suit your research needs, focusing on building better tech that meets the needs of those typically underserved in tech design

View these cards as a starting point or example. Review the cards and make adjustments for your user group, your technology and your intersecting focus area(s): Gender, class, caste, education, and more.

We offer our research reflections here as suggestions for what worked -- and what didn’t-- to guide adaptation of these cards.

This research was exploratory-- intentionally broad, serving as sensing activities towards broader sociocultural entanglements with technology, which we hope YOU will build on!

These tools are aimed towards participatory and equity-focused technology design and research.

How to use this toolkit

Gauging attitudes towards gender bias, and testing those against LLM applications

To gain additional insights, one could incorporate additional prompts or scenarios that explore intersectional aspects of gender, such as the intersection of gender with socioeconomic status, religion, or disability.

The research provided insights into how users negotiate societal expectations and alternative perspectives presented by the chatbot.

The activity served as an effective cultural probe, uncovering contextual dissonance between users’ expectations and the chatbot’s responses.

The structure of the activity worked well in gauging users’ expectations and satisfaction with the chatbot’s responses.

The activity effectively identified gender-related mindsets among users and the context behind them.

REFLECTIONS

Understanding user experiences with technology and chatbots

Understanding prompt-generation and getting feedback from users on scenario-specific usage of LLM applications

Environmental and logistical factors, such as the physical location and available space for the printed mock-ups impacted the ability to conduct the activity effectively.

Accessibility and inclusivity were concerns, as the printed interfaces and the significant reading requirements posed barriers for participants with limited literacy.

The activity was more effective for users with more experience wtih digital interfaces. Novice users struggled to articulate preferences.

This activity was helpful in understanding the preferences of users from the perspective of product and interface design. This could be particularly helpful for a niche user group or use case, such as in healthcare settings.

RESEARCH REFLECTIONS

1. Interface with a short response and follow up question

2. Interface with a long response

GROUP 2 - RESPONSE LENGTH / FORMAT

The exercise showed promise in understanding factors such as trust, motivation, and cultural relevance, suggesting the need for refined research methods.

Variability in comfort with Hindi and Hinglish influenced user interaction, indicating the need for local language speakers in research teams.

Since the activity involves engaging with an LLM based application it provides additional insight into user engagement, and the ability to create prompts when there is context already provided. With a scenario provided, users had more information in their prompts than in the open-ended prompting act

This activity gave a good sense of the mindset of users regarding LLM based applications for questions around personal life decisions. It also brought up conversations around trusting/not trusting tech and/or LLM based applications. Users were able to clearly point out whether they would like human

The activity could be reframed to focus on understanding prompt-generation and user feedback on scenario-specific LLM applications, since we were unable to test for the impact of language.