SESSION: Digital Crafting and Computational Affordances
Session details: Digital Crafting and Computational Affordances
Hand Spinning E-textile Yarns: Understanding the Craft Practices of Hand Spinners and Workshop Explorations with E-textile Fibers and Materials
The ‘material turn’ in Human Computer Interaction (HCI) is increasingly drawing attention to the computational affordances of materials and how we can craft with them. In this paper, we explore opportunities for combining the maker cultures of hand spinning with e-textile crafting. In our first study, we interviewed 32 hand spinners on their practices to better understand their motivations for spinning their own yarns and the techniques they use to do so. In our second study, we conducted workshops with 6 spinners at a local spinning guild, where participants worked with the conductive fibers and spun e-textile yarns. After the workshops, we conducted follow-up interviews with each participant to understand the opportunities and tensions of hand spinning e-textile yarns. Our findings show how spinners can blend local materials with conductive ones to develop their own custom interactive textiles, and the mismatch between how these fibers are sold and what information spinners require to inform their design decisions. Through these results, we hope to empower makers and inspire the design community to develop tools to support these DIY practices.
AnimaTo: Designing a Multimorphic Textile Artefact for Performativity
Multimorphic textile-forms, obtained through simultaneous thinking of material and form that change in design and/or use time, have the potential to elicit diverse performances in the use of textile artefacts, thereby extending their relevance in our everyday lives. We present AnimaTo, a multimorphic textile artefact designed for performativity that reacts to water exposure via the shrinking and dissolving of its fibres. Adopting a material-driven design approach, we engaged in material tinkering with these qualities to achieve changes in the texture, size, and shape of AnimaTo. Following this exploration, we conducted a pilot study to gain insights into AnimaTo’s temporal behaviour and performativity in use. In the further development of the artefact, we highlight the challenges that arise in producing high-fidelity prototypes. This work grants insights into how designers can tune material, form, and temporal qualities of textile artefacts towards multiplicity of use and prolonged user-textile relationships.
Design Bookkeeping: Making Practice Intelligible through a Managerial Lens
As DIS researchers increasingly describe design as an emergent and material engaged practice, many are embracing different approaches to design documentation that capture the breadth of these practices. This pictorial contributes to these efforts by shedding light on a kind of managerial work that emerged when creating a complex e-textile installation. Specifically, we reflect on our project through the lens of “design bookkeeping” to describe documents that embody managerial knowledge and describe what these documents make intelligible about our practice. We surface findings and cross-cutting themes that bring attention to these practices in relation to broader understandings of project documentation. We then speculate on how the DIS community could circulate this knowledge within, and beyond, academic publication venues.
Millipath: Bridging Materialist Theory and System Development for Surface Texture Fabrication
Proponents of digital fabrication argue that future technologies will fundamentally reshape manufacturing; however, we still have a limited understanding of the relationship between contemporary digital fabrication technologies and the values and labor of people who make things. Contemporary materialist theories can offer insights into how interaction modalities with machines and materials influence human production activities. We aim to implement these theoretical principles in technical system development. We focus on action as a bridging concept between abstract notions regarding human-machine-material relationships and concrete digital fabrication system features. We use CNC-milled surface texture production on wood as a case study. We follow a research-through-design process to develop Millipath, an action-oriented programming platform enabling the parametric design of machine toolpaths. Through the analysis of autobiographical data from fabricating artifacts, we investigate how digital fabrication systems informed by materialist theories support expressive modes of production and design decisions in response to material behaviors.
Resistive Threads: Electronic Streetwear as Social Movement Material
Informed by legacies of textile activism, we design Resistive Threads as a wearable probe to investigate potential roles and trajectories of electronic streetwear in US urban social movements. Resistive Threads is an interactive denim jacket that refashions the Anti-Eviction Mapping Project’s (Dis)location Black Exodus print zine. The jacket plays audio stories, poetry, and music from embedded speakers when interactive patches sewn with conductive thread are tapped upon. Examining the artifact with 10 community organizers and partners, we find that augmented streetwear may take on the role of a housing organizing instrument or speculative garment. In turn, we discuss how we might learn from textile histories and solidarities to recognize—not rehearse—damage-centered research. We close with a reflection on what makes the electronic aspect of e-textiles meaningful to social movement practice and performance.
SESSION: Voice-Interactions & Soundscaping
Session details: Voice-Interactions & Soundscaping
Does the Medium Matter? An Exploration of Voice-Interaction for Self-Explanations
This research evaluates voice-based self-explanations as a pedagogical tool in preparation for lectures, assesses user preferences between voice and text, and derives design insights. We report two studies: Study 1, a quasi-experimental field study, with 247 participants divided into voice-based (N = 83), text-based (N = 81), and choice (N = 83) conditions. Study 2 uses semi-structured interviews (N = 16) to explore perceptions of the interaction paradigms in-depth. Results from the first study revealed a general preference for text, though voice users produced longer responses and more topic-related keywords. Over time, the preference for voice increased among students, from 10% to 46%, when given a choice. Study 2 suggested that factors like social presence contribute to hesitance toward voice-based explanations, with a cognitive load, self-confidence, and performance anxiety also influencing medium preferences. Our findings highlight design recommendations and demonstrate the potential of voice-based self-explanations in educational settings, indicating that mixed interfaces might better meet diverse needs.
VOICON: Geometric Motion-Based Visual Feedback in Voice User Interface
Geometric motion-based visual feedback widely used in existing voice assistants (VAs) plays a significant role in voice interactions. It enables a realtime and simultaneous depiction of VAs’ conversational states, thereby mitigating the verbosity of speech and reducing unnecessary turn-taking. However, the study on VA visual feedback, which focuses on geometric motion graphics for different conversational states, has not been extensively explored. In this study, we systematically designed `Voicon’, shape-based visual feedback, and explored the expandability of its visual language. To achieve this, we determined 3 conversational states through a literature review. We then conducted a conjoint analysis using the visual attributes extracted from a participatory design workshop. Based on the results, we proposed the Voicon system and created video prototypes for qualitative evaluation. Our findings demonstrated that our Voicon system is intuitively understandable and perceived as useful by users. We believe this pictorial enriches and expands the design space of visual feedback for VAs, which facilitates voice interaction.
SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness
Mixed-reality (MR) soundscapes blend real-world sound with virtual audio from hearing devices, presenting intricate auditory information that is hard to discern and differentiate. This is particularly challenging for blind or visually impaired individuals, who rely on sounds and descriptions in their everyday lives. To understand how complex audio information is consumed, we analyzed online forum posts within the blind community, identifying prevailing challenges, needs, and desired solutions. We synthesized the results and propose SoundShift for increasing MR sound awareness, which includes six sound manipulations: Transparency Shift, Envelope Shift, Position Shift, Style Shift, Time Shift, and Sound Append. To evaluate the effectiveness of SoundShift, we conducted a user study with 18 blind participants across three simulated MR scenarios, where participants identified specific sounds within intricate soundscapes. We found that SoundShift increased MR sound awareness and minimized cognitive load. Finally, we developed three real-world example applications to demonstrate the practicality of SoundShift.
Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User Interfaces
With the progress in Large Language Models (LLMs) and rapid development of wearable smart devices like smart glasses, there is a growing opportunity for users to interact with on-device virtual assistants through voice and gestures with ease. Although voice user interfaces (VUIs) have been widely studied, the potential uses of full-body gestures in VUIs that can fully understand users’ surroundings and gestures are relatively unexplored. In this two-phase research using a Wizard-of-Oz approach, we aim to investigate the role of gestures in VUI interactions and explore their design space. In an initial exploratory user study with six participants, we identify influential factors for VUI gestures and establish an initial design space. In the second phase, we conducted a user study with 12 participants to validate and refine our initial findings. Our results showed that users are open and ready to adopt and utilize gestures to interact with multi-modal VUIs, especially in scenarios with poor voice capture quality. The study also highlighted three key categories of gesture functions for enhancing multi-modal VUI interactions: context reference, alternative input, and flow control. Finally, we present a design space for multi-modal VUI gestures along with demonstrations to enlighten future design for coupling multi-modal VUIs with gestures.
SESSION: Across Realities #2
Session details: Across Realities #2
GraV: Grasp Volume Data for the Design of One-Handed XR Interfaces
Everyday objects, like remote controls or electric toothbrushes, are crafted with hand-accessible interfaces. Expanding on this design principle, extended reality (XR) interfaces for physical tasks could facilitate interaction without necessitating the release of grasped tools, ensuring seamless workflow integration. While established data, such as hand anthropometric measurements, guide the design of handheld objects, XR currently lacks comparable data, regarding reachability, for single-hand interfaces while grasping objects. To address this, we identify critical design factors and a design space representing grasp-proximate interfaces and introduce a simulation tool for generating reachability and displacement cost data for designing these interfaces. Additionally, using the simulation tool, we generate a dataset based on grasp taxonomy and common household objects. Finally, we share insights from a design workshop that emphasizes the significance of reachability and motion cost data, empowering XR creators to develop bespoke interfaces tailored specifically to grasping hands.
Stick-To-XR: Understanding Stick-Based User Interface Design for Extended Reality
This work explores the design of stick-shaped Tangible User Interfaces (TUI) for Extended Reality (XR). While sticks are widely used in everyday objects, their applications as a TUI in XR have not been systematically studied. We conducted a participatory design session with twelve experts in XR and human-computer interaction to investigate the affordances of stick-based objects and how to utilize them in XR. The results led us to develop a taxonomy of stick-based objects’ affordances in terms of their functions and holding gestures. Following that, we proposed four types of stick-based XR controller forms and discussed their advantages and limitations. In the end, we juxtaposed twenty-six existing XR controllers against our proposed forms and identified Landed (Cane) Stick, Thin Stick’s flexible usages, and Modular Design as the major opportunities that remain unexamined yet for stick-based XR TUI design.
Behind the Scenes of CXR: Designing a Geo-Synchronized Communal eXtended Reality System
We have developed a Communal eXtended-Reality (CXR) system that enables groups of people in a shared moving vehicle to view a common geo-synchronized tour. This paper describes the geo-synchronized multi-user extended reality system we created to provide a situated and shared experience to promote community engagement. This paper describes: (a) the technical implementation of the CXR system, which geo-locates and orients the view of the participant within the moving vehicle; (b) the immersive digital twin tour, critically aligned with the real-life location; (c) our fallback system, which allows people who feel disoriented or motion-sick to continue along with the content of the tour. We validated the sense of communality, comfort, and effectiveness of the system through in-ride observation and post-ride surveys. Our intent is to enable development of similar systems to foster communal engagement in communities worldwide.
Exploring Human Values in Mixed Reality Futures
The rapid development of immersive technologies is heralding a shift from purely physical environments to one that seamlessly mixes the physical and the digital. As these Mixed Reality (MR) worlds develop quickly we need to reflect on how human values are incorporated into the design and deployment of the technology. Human values map our perception of the world, reflect attitudes, guide behaviours, and provide us with social and moral grounding. However, there is limited research on incorporating values in the design of MR technologies. This research has three contributions: (1) a playful values-driven workshop design, (2) insights into the values of different groups of people in diverse MR scenarios, and (3) recommendations for incorporating human values for future MR design and application. This work will contribute to improving the ethical and responsible development of current and future MR applications.
SESSION: Design, Ethics and AI Explorations
Session details: Design, Ethics and AI Explorations
The Power of Absence: Thinking with Archival Theory in Algorithmic Design
This paper explores the value of archival theory as a means of grappling with bias in algorithmic design. Rather than seek to mitigate biases perpetuated by datasets and algorithmic systems, archival theory offers a reframing of bias itself. Drawing on a range of archival theory from the fields of history, literary and cultural studies, Black studies, and feminist STS, we propose absence—as power, presence, and productive—as a concept that might more securely anchor investigations into the causes of algorithmic bias, and that can prompt more capacious, creative, and joyful future work. This essay, in turn, can intervene into the technical as well as the social, historical, and political structures that serve as sources of bias.
In Whose Voice?: Examining AI Agent Representation of People in Social Interaction through Generative Speech
As generative artificial intelligence (genAI) applications gain popularity, there is a dearth of research examining how applications may transform social interactions. One possible application set to transform social interactions is the use of generative speech to power AI agents that can realistically represent people. Our work examines the potential implications of AI agents representing individuals in human conversations (“agent representation”) as a way to begin filling this research gap. We take a multi-method approach, conducting formative interviews with developers, a co-design workshop with designers, a harm analysis among researchers, and interviews with the general public. Both technologists and potential users worry adopting agent representations might harm the quality, trust, and autonomy of human communication. Potential users are particularly concerned that agent representations could undermine the value of social interaction and threaten individuals’ ability to control their image. To avoid such potential consequences, future genAI-powered agents and speech applications should take into account user-defined red lines when considering applying these technologies in social settings.
A Steampunk Critique of Machine Learning Acceleration
The application of Machine Learning is driven by the techno–capitalist struggle for productivity across various domains, including the creative industry. Sociological research has demonstrated how technology–induced temporality introduces challenges at the individual and societal levels. Art creativity conflicts with speed and mass production. This paper describes Isotta, a critical artefact combining a Mignon typewriter and a Language Model to spark discussion about ML–induced acceleration. Fourteen artists evaluated Isotta in an interview study, and semiotics was used as the analytical lens. Results exposed ideological assumptions around the consequences of technology in the writing realm. We discuss these insights in the context of interactive design in times of techno–capitalistic acceleration. Our findings highlight the significance of temporal factors in designing generative writing interactions and underscore how complex societal challenges can be approached in design through the contrast–eliciting property that outdated technologies offer when juxtaposed with contemporary technologies.
Motivating Users to Attend to Privacy: A Theory-Driven Design Study
In modern technology environments, raising users’ privacy awareness is crucial. Existing efforts largely focused on privacy policy presentation and failed to systematically address a radical challenge of user motivation for initiating privacy awareness. Leveraging the Protection Motivation Theory (PMT), we proposed design ideas and categories dedicated to motivating users to engage with privacy-related information. Using these design ideas, we created a conceptual prototype, enhancing the current App Store product page. Results from an online experiment and follow-up interviews showed that our design effectively motivated participants to attend to privacy issues, raising both the threat appraisal and coping appraisal, two main factors in PMT. Our work indicated that effective design should consider combining PMT components, calibrating information content, and integrating other design elements, such as visual cues and user familiarity. Overall, our study contributes valuable design considerations driven by the PMT to amplify the motivational aspect of privacy communication.
SESSION: Future Imaginations and Speculative Design
Session details: Future Imaginations and Speculative Design
“It’s kind of weird talking to a sphere”: Exploring Children’s Hopes and Fears on Social Robot Morphology Using Speculative Research Methods
The integration of social robots into children’s environments is becoming increasingly pertinent, spanning from entertainment to health care. Although prior studies highlight the role of morphology in shaping children’s perceptions of these machines, there is little research to examine their perceptions of social robots in various contexts. Our research investigates how different morphologies (anthropomorphic, zoomorphic, and mechanomorphic) influence children’s emotional responses. We involved 36 children (9-11 years old) in a design fiction-based study examining how morphology impacts children’s hopes and fears of social robots in varying scenarios and contexts, categorising the results into four distinct themes that represent children’s general perceptions (autonomy, cognition, socio-emotional, and physical). From this, we identify two distinct design challenges, and discuss how these may impact both researchers general users of social robots with children. Further, we emphasise the need for careful and considerate deployment of social robots for children in both research and real-world applications.
Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivation
We ideate a future design workflow that involves AI technology. Drawing from activity and communication theory, we attempt to isolate the new value that large AI models can provide design compared to past technologies. We arrive at three affordances—dynamic grounding, constructive negotiation, and sustainable motivation—that summarize latent qualities of natural language-enabled foundation models that, if explicitly designed for, can support the process of design. Through design fiction, we then imagine a future interface as a diegetic prototype, the story of Squirrel Game, that demonstrates each of our three affordances in a realistic usage scenario. Our design process, terminology, and diagrams aim to contribute to future discussions about the relative affordances of AI technology with regard to collaborating with human designers.
Designing with Transactional Data: FTML and Money/Data Laundering
In a digital, and cashless economy, transactional data has become ubiquitous, telling and highly valuable. Yet, this data is rarely considered critically as a material for design. This pictorial presents two successive Research through Design projects exploring practically how we might design with transactional data. The first, ‘FTML: Financial Transaction Mark-up Language’ is a speculative design project and short film, which explores how value-laden ‘mark up’ of specific transactional data could underpin new services and digital applications. The second, ‘Money / Data Laundering’ adopted an approach of ‘designerly hacking’, with Point of Sale (PoS) payment card readers, to develop a web application to digitally ‘wash’ or launder specific values into an indivdual’s bank account via a symbolic transaction. Reflecting on both interventions, we demonstrate the need, and opportunities, for designers to engage critically with transactional data and financial infrastructures, to enable new forms of value(s) exchange.
Amazon Z to A: Speculative Design to Understand the Future of Labor-Intensive Workplaces
Understanding warehouse work is critical to “future of work’’ scholarship as warehouses are vital indicators for anticipating how work could be structured, controlled, and experienced in other data-driven workplaces in the future. However, researchers often face challenges in studying and designing interventions in such work environments, particularly ones where non-disclosure agreements and intensive, isolated, and precarious work conditions pose practical barriers to research access. By creating a set of speculative designs about warehouse work futures, we explore how speculative design techniques can be used to analyze and critically engage with on-going ethnographic research into warehouse work at Amazon fulfillment centers. These designs serve not only as a means for unpacking the logics of contemporary warehouse work but also as an approach to identify directions for worker-centered research and design in the future. This paper also provides sensibilities for using speculative design techniques to study hostile and labor-intensive work environments.
SESSION: Augmented Reality on the Go
Session details: Augmented Reality on the Go
“I’m not alone in that battle”: Designing Mobile AR for Mental Health Communication and Community Connectedness
For researchers at the intersection of health and human computer interaction, mobile AR presents a compelling platform for public health communication: it is increasingly available, highly customizable, and can present interactive visualizations of complex data. However, designers face challenges not only in adapting appropriate data and relevant public health metrics, but also in assessing their communicative potential and effectiveness for the target community. To contribute insight into this research area, we designed four mobile AR visualizations based on mental health issues and resources for our local university community. We then conducted a mixed-methods field experiment to investigate the impact of our AR visualizations on participants’ awareness and understanding of pressing health issues, and to document barriers to use in this context. We show that our visualizations increased participants’ sense of community connectedness and prompted them to reflect on their relationship with the university community. Based on these findings, we discuss opportunities for the field of human-computer interaction to further support public health communication.
Integrated Calculators: Moving Calculation into the World
Computing devices commonly act as tools, extending our abilities and shaping how we interact with the world. We investigate one such tool, the calculator, which helps with arithmetic, but also commonly offers specialized functions for conversions, formulas, or graphing. Through an analysis of calculator apps and use cases, we describe limitations of current calculators. Crucially, calculator apps remain detached from tasks, motivating us to explore how to more closely integrate calculation with the world through augmented reality (AR). AR calculators can directly use measurements and numbers from the world in calculations as well as display results of calculations in the world. We provide a conceptual account of calculation in AR, as well as video prototypes that concretize the concept across different scenarios. These examples demonstrate how moving tools like the calculator to AR offers tighter task integration and reduces the work required in translating between the world and computational tools.
Virtual Threads: A Systematic Literature Review of 10 Years of VR/AR/MR Adoption in Fashion Design
In the past decade, HCI researchers have developed virtual reality (VR), augmented reality (AR), and mixed reality (MR) solutions to support fashion designers. Unlike traditional technologies, VR/AR/MR technologies provide immersion and physical interactions with digital garments. To comprehend the transformative impact of these technologies in fashion design, we conducted a systematic review covering 2013 to 2023, assessing the impact of VR/AR/MR adoption in fashion design. After screening 126 articles, we identified 26 articles that introduce immersive systems centered on two main themes: advancing the fashion design toolkit and enhancing fashion design processes. We discuss potential opportunities and limitations to be studied in future research and reflect on how VR/AR/MR technologies can continue to support fashion designers in fulfilling their needs and overcoming obstacles.
GlassMail: Towards Personalised Wearable Assistant for On-the-Go Email Creation on Smart Glasses
Optical See-through Head-Mounted Displays (OHMDs) offer new opportunities for completing complex information processing tasks on the go. We introduce GlassMail, a Large Language Models (LLMs)-based wearable assistant on OHMDs for mobile email creation. Our formative study identified two challenges of the LLM-based wearable email assistant: (i) achieving efficient and accurate understanding of user intentions, and (ii) ensuring effective information presentation for email processes. Through two empirical studies, we developed a “Single Turn with Optional Clarification ” approach for accurate user intention recognition and a “Fade Context with Optional Audio ” mode for effective email processing. An observation study then evaluated GlassMail ’s feasibility in composing formal and semi-formal emails, supporting the usefulness and effectiveness of GlassMail in simple scenarios and yielding insights into potential future improvements for complex scenarios. We further discuss the design implications for the future development of wearable AI-enabled assistants.
Exploring Augmented Reality Interface Designs for Virtual Meetings in Real-world Walking Contexts
Research has shown that walking during meetings improves creativity, memory, attention, health, and happiness. While mobile technologies have freed users from having to be stationary during virtual meetings, mobile phones pose several usability challenges, such as reduced productivity due to small screens and safety concerns. In this paper, we present the first exploration of augmented reality (AR) interface design for virtual meetings in real-world walking conditions. We conducted design sessions in-situ with 16 user interface and AR designers using a 2×2 experimental design: meeting while walking in two levels of traffic conditions and with two types of meeting formats. Results show that the designed AR windows averaged 14.5 times in viewing size compared to the most popular smartphone screens. Also, Traffic Level significantly affected the size, opacity, and placement of windows, as well as the preference of anchoring modes, while Meeting Format significantly affected size and opacity. Furthermore, clustering analysis identified two groups of designs that can serve as initial reference designs for further customization and research.
Understanding Gesture and Microgesture Inputs for Augmented Reality Maps
We explore the potential for subtle on-hand gesture and microgesture interactions for map navigation with augmented reality (AR) devices. We describe a design exercise and follow-up elicitation study in which we identified on-hand gestures for cartographic interaction primitives. Microgestures and on-hand interactions are a promising space for AR map navigation as they offers always-available, tactile, and memorable spaces for interaction. Our findings show a clear set of microgesture interaction patterns that are well suited for supporting map navigation and manipulation. In particular, we highlight how the properties of various microgestures align with particular cartographic interaction tasks. We also describe our experience creating an exploratory proof-of-concept AR map prototype which helped us identify new opportunities and practical challenges for microgesture control. Finally, we discuss how future AR map systems could benefit from on-hand and microgesture input schemes.
SESSION: Sensibilities of the Theatrical, Affect & Tangibility
Session details: Sensibilities of the Theatrical, Affect & Tangibility
Tangible Affect: A Literature Review of Tangible Interactive Systems Addressing Human Core Affect, Emotions and Moods
Tangible user interfaces (TUIs) have been applied to assess, communicate, or regulate human core affect, emotions, or moods. Previous studies identified TUIs as an innovative way to serve people’s affective needs. This review examines the design and evaluation of tangible interactive systems that focus on human core affect, emotions, and moods. We provide an overview of current studies. We summarize how tangibility can be leveraged to support affective interaction, and we propose the dimensions of tangible affective interaction, deriving guidelines for design. We highlight three main challenges: understanding tangible affective interaction within real-life scenarios, utilizing embodied interaction to express or influence affective states, and establishing benchmarks for evaluating tangible affective interfaces.
KiPneu: Designing a Constructive Pneumatic Platform for Biomimicry Learning in STEAM Education
Biomimicry, a methodology adapted from nature, always inspires optimum solutions and innovative technologies in human history. To get children interested in, excited about, and inspired by biomimicry, we introduce KiPneu, a robotic platform that facilitates biomimicry education through hands-on, solution-oriented learning and a digital learning environment. KiPneu allows children to mimic flexible animal locomotion, like fish swimming or worm squirming, using low-cost building blocks and non-electrical pneumatic actuators. We provide five types of non-electrical tangible valves to adjust robot motion characteristics, such as direction and speed, through engaging tangible programming. Additionally, to facilitate the whole learning process, KiPneu comes with interactive instructional interface that visualize and simulate the pneumatic system. To validate KiPneu’s educational efficacy, we conducted a three-day workshop with 21 children aged 5-12. Pre-and-post surveys revealed KiPneu not only enhanced their understanding of animal locomotion mechanisms but also spurred interest in creative construction using acquired knowledge.
Tangible Scenography as a Holistic Design Method for Human-Robot Interaction
Traditional approaches to human-robot interaction design typically examine robot behaviors in controlled environments and narrow tasks. These methods are impractical for designing robots that interact with diverse user groups in complex human environments. Drawing from the field of theater, we present the construct of scenes—individual environments consisting of specific people, objects, spatial arrangements, and social norms—and tangible scenography, as a holistic design approach for human-robot interactions. We created a design tool, Tangible Scenography Kit (TaSK), with physical props to aid in design brainstorming. We conducted design sessions with eight professional designers to generate exploratory designs. Designers used tangible scenography and TaSK components to create multiple scenes with specific interaction goals, characterize each scene’s social environment, and design scene-specific robot behaviors. From these sessions, we found that this method can encourage designers to think beyond a robot’s narrow capabilities and consider how they can facilitate complex social interactions.
ProtoBricks: A Research Toolkit for Tangible Prototyping & Data Physicalization
Building tangible interfaces or data physicalizations is a resource-intensive endeavour. There is a need for rapid means to prototype tangibles in order to facilitate research and design. To this end, we designed ProtoBricks: a research toolkit that uses capacitive bricks to facilitate rapid prototyping for tangible interfaces. Utilizing toy bricks that do not contain electronics, ProtoBricks can record brick position and color. Specialized knowledge is not required to build our system as it uses widely available components and 3D printing. We contribute the full software and hardware specification of the toolkit. We evaluate the utility of the toolkit by reporting on past use cases and prototyping workshops. We show that the toolkit facilitates creativity and effectively supports prototyping. ProtoBricks lowers the entry threshold for experimenting with tangible interfaces and enables researchers and designers to focus on the interaction with their prototype, delegating implementation to the toolkit.
SoftBioMorph: Fabricating Sustainable Shape-changing Interfaces using Soft Biopolymers
Bio-based and bio-degradable materials have shown promising results for sustainable Human-Computer Interaction (HCI) applications, including shape-changing interfaces. However, the diversity of shape-changing behaviors achievable with these materials remains unclear as the fabrication knowledge is scattered across multiple research fields. This paper introduces SoftBioMorph, a fabrication framework that aims to integrate the fabrication know-how of sustainable soft shape-changing interfaces with biopolymers. Based on the example of Sodium Alginate, the framework contributes (1) a set of material synthesis processes that modify the biopolymer’s properties to fulfill different functions; (2) a set of DIY crafting-based assembling techniques that functionalize the material and assembling properties to achieve three primitive types of change in shape; and (3) a series of application cases that demonstrate the versatility of the framework. We further discuss limitations, research questions, and fabrication challenges, presenting a comprehensive approach to sustainable prototyping in HCI.
Red [Redacted] Theatre: Queering Puzzle-Based Tangible Interaction Design
“Red [Redacted] Theatre” is a tangible puzzle-based experience exploring queer history. We used queer methods to design puzzles with archival materials, guiding participants from a more normative “surface” understanding of history into exploring queer histories, breaking familiar language and constructs through play and tangible interaction. Our reflections contribute design considerations for queering tangible interaction through puzzles: (1) designing and deconstructing layers of queer understanding, (2) attending to situatedness in designing tangible puzzle artifacts, and (3) designing puzzles that solve to complexity. We also offer insights on how facilitation and debrief of the experience can prompt reflection on queer history and ways of understanding. Overall, this pictorial contributes to Queer HCI by deepening queer tangible interaction, particularly for exploring queer history.
SESSION: Un/health and Data at Stake
Session details: Un/health and Data at Stake
How Can I Signal You To Trust Me: Investigating AI Trust Signalling in Clinical Self-Assessments
Individuals are increasingly interested in and responsible for assessing their own health. This study evaluates a fictional AI dermatologist for assistance in the self-assessment of moles. Building on the Signalling Theory, we tested the effect of textual descriptions provided by a virtual dermatologist, as manipulated across ‘Ability’, ‘Integrity, ’ and ‘Benevolence’, along with the clinical assessment, ‘benign’ or ‘malignant’, affect users’ trust in the aforementioned trust pillars. Our study (N = 40) follows a 2 (Ability low/high) × 2 (Integrity low/high) × 2 (Benevolence low/high) × 2 (mole assessment benign/malignant) within-subject factorial design. Our results demonstrate that we can successfully influence perceptions of ability and benevolence by manipulating the corresponding aspects of trust but not perceived integrity. Further, in the case of a malignant assessment, participants’ perception of trust increased across all aspects. Our results provide insights into the design of AI support systems for sensitive use cases, such as clinical self-assessments.
Designing for Participatory Data Governance: Insights from People with Parkinson’s
In today’s age of data-driven healthcare, the growing utilization of health data to inform critical aspects of patient care and medical research places an ever increasing significance on its governance. This study aims to explore the perspectives of individuals living with Parkinson’s Disease regarding their needs and preferences in relation to data governance. We first conducted a survey (n=52) to explore the types of data people with Parkinson’s generate through their self-care practices, and then conducted 3 workshops with 9 participants to understand their perspectives on the governance of this type of data. Through this work, we highlight the factors that motivate them to collect self-care data, and present their requirements for its governance, which could inform the design of future infrastructure to support these needs. We also showcase how speculative approaches can be used to engage communities in discussions around data collection and governance.
The Unanticipated Use of Fitness Tracking Technologies During Post-COVID Syndrome
Long COVID is a new illness, and we still lack medical knowledge about possible treatments. This has led to the independent adoption of fitness tracking technologies for the management of long COVID. Through semi-structured interviews with 21 people with long COVID who used fitness tracking technologies, we found that fitness tracking devices were used to pace energy through finding correlations between activities and data, provide proof of illness to themselves, healthcare providers and friends and family, and to help gain a sense of agency and control during a new and enigmatic illness. Our findings support and extend design openings for the development of “pacing technologies”. These include the importance of customization and supporting the user in their recovery if appropriate, applying strategies for mitigating anxiety when tracking, facilitating the sharing of pacing data with friends and family and healthcare providers, and supporting users in identifying the optimal components of rest.
Proposing a Context-informed Layer-based Framework: Incorporating Context into Designing mHealth Technology for Fatigue Management
Owing to the multi-factorial nature of fatigue, leveraging context to effectively monitor and intervene with fatigue symptoms presents a significant challenge. This paper aimed to understand how to incorporate context into designing mHealth systems for fatigue management. We conducted a two-week field study with 20 fatigue-vulnerable individuals using an activity-tracking sensor and self-reporting. We conducted data-prompted interviews to explore phenomena about participants’ fatigue experiences. Findings show a heterogeneous relationship between context and fatigue, which can be attributed to the phenomena that: (1) participants were influenced by multiple fatigue-inducing factors for different durations; (2) broad contexts moderated participants’ perceptions and coping strategies in response to local contexts; (3) the predictability and repetition of activities influenced participants’ fatigue perception and coping strategies. We propose a context-informed layer-based framework integrating these phenomena and discuss implications for designing fatigue management tools informed by our framework.
Designing for Personalization in Personal Informatics: Barriers and Pragmatic Approaches from the Perspectives of Designers, Developers, and Product Managers
As the trend towards personalization in health science and technology fields has proliferated, researchers focusing on personal informatics (PI) tools for health management have grappled with when and how to design for a personalized user experience. While previous literature has addressed users’ perspectives on personalization, an important and underexplored research area is how personalization is approached within organizational contexts. This paper reports findings from qualitative interviews with designers, developers, and product managers at a multinational company that designs, builds, and commercializes PI tools for health management. The results point to four types of barriers participants consistently encountered, often leading personalization efforts to be scaled back, and three pragmatic approaches to personalization often adopted in the face of these barriers. The paper concludes with design implications for incorporating personalization while remaining aligned with the nuances, realities, and potential strain on systems and resources inherent to the design process in company settings.
“Data Is One Thing, But I Want To Know The Story Behind”: Designing For Self-Tracking and Remote Patient Monitoring In The Context Of Multiple Sclerosis Care
We report on design-focused inquiry into future multiple sclerosis (MS) healthcare; including a multi-stage design process with experienced MS clinicians, and formative evaluations with people living with MS. MS is a chronic, progressive, and unpredictable inflammatory neurological disease of the central nervous system that affects at least 2.8 million people worldwide. Walking impairments affect up to 85% of people diagnosed with MS. Responding to this, our focus is on design for longitudinally monitoring mobility, and in particular using wearable sensors that generate data on gait metrics to support clinical and self-care decision-making. We contribute to HCI research in three ways: (1) a detailed case study design process, including artifacts; (2) metaphorical framing concepts, with associated use cases illustrated through design scenarios; and (3) understanding of virtual-first practices in rehabilitation medicine that can be translated beyond MS care.
SESSION: Spiritual Imaginaries
Session details: Spiritual Imaginaries
Artificial Dreams: Surreal Visual Storytelling as Inquiry Into AI ‘Hallucination’
What does it mean for stochastic artificial intelligence (AI) to “hallucinate” when performing a literary task as open-ended as creative visual storytelling? In this paper, we investigate AI “hallucination” by stress-testing a visual storytelling algorithm with different visual and textual inputs designed to probe dream logic inspired by cinematic surrealism. Following a close reading of 100 visual stories that we deem artificial dreams, we describe how AI “hallucination” in computational visual storytelling is the opposite of groundedness: literary expression that is ungrounded in the visual or textual inputs. We find that this lack of grounding can be a source of either creativity or harm entangled with bias and illusion. In turn, we disentangle these obscurities and discuss steps toward addressing the perils while harnessing the potentials for innocuous cases of AI “hallucination” to enhance the creativity of visual storytelling.
Worlding with Tarot: Design, Divination, and the Technological Imagination
Design cards have long played an important role in reflection-directing the designer’s gaze toward unasked questions, hidden consequences, and new horizons for speculative futuring. But few questions have been asked about what motivates the design process and how deck designers see their role in inquiry and/or world building. This paper looks to the Tarot deck as one iconic example of such a process. Drawing on interviews with nine Tarot deck creators, we surface themes of Tarot as scaffolding modes of personal and collective growth, forms of carework, and pathways for different ways of knowing. We discuss expanding design inquiry methods for understanding and elevating forms of spiritual connection and care; and moving from anti-appropriation to ante-appropriation.
Untangling Cables: A Case Study of the Life & Afterlife of Digital Devices in Academic Research
As researchers and academics, we investigate our bad habitus—our everyday practices around technologies for research that reinforce dynamics of extraction, consumption, and waste—in relation to the lifecycle of technology in academic research. Through qualitative interviews, observation, and visual documentation, this case study explores the consideration of sustainability in purchasing decisions, use, maintenance, and disposal processes of digital devices used in a North American university as well the institution’s related policies and procedures and faculty members’ practices. Through this research we find tensions that complicate sustainability in the university research context. We develop a rich description of the complexities of creating sustainable practices, policies, and procedures in a university setting as a step towards becoming more sustainable in our work and in our institutions, and we offer a set of recommendations for our academic institution and systems that both advance and thwart efforts to create sustainable practices.
Manipulative Design and Older Adults: Co-Creating Magic Machines to Understand Experiences of Online Manipulation
Manipulative designs — i.e., dark patterns — have pervaded online interactions in most sectors from e-commerce to social media, banking, and healthcare. Understanding how individuals experience and cope with online manipulation is essential to support evolved design practices and regulatory measures. Yet studies on populations who may be more vulnerable to online manipulation are scarce. Through a series of “magic machines” workshops, we investigated the experiences of older adults (N=31) with online manipulation, their needs, and the strategies they imagine to resist manipulative practices. Our results show that participants tend to attribute manipulation to an “unknown” person and do not distinguish platforms from content. Through their machines, they expressed four primary needs to resist manipulation: knowledge, awareness, right to sanctuary, and control. Our study contributes insights into older adults’ experiences with online manipulation and brings design challenges for effective countermeasures to manipulation that address the needs of all users.
Materialising affective experiences: Designing for personal domestic grief practices
In this pictorial, we present the process leading to the creation of N/ERWER, an affectively engaging interactive prototype exploring personal domestic grief practices. N/ERV/ER symbolises a physical presence of the Intangible affective experiences associated with grief as an ongoing connection to the deceased in the present. N/ERV/ER was developed in a Research through Design process fueled by both personal accounts and participatory engagements using various design probes exploring experiences of loss following the death of a loved one. In this pictorial, we unfold the theoretical, methodological, and material dimensions of this process to show how designers can accommodate grief in design practice as a way of utilising and materialising the affective forces present in such sensitive contexts.
Taking the bizarre seriously: dreams as a material for interaction design
Sleep technologies promise optimised sleep by tracking sleep cycles or by affecting sleep environment. This paper expands sleep technology to address more complex notions of sleep, proposing dream technology as a path for reflective practices that focus on nightly dreams rather than hours slept. Because dreams are bizarre and hard to grasp, we turn to speculative design practices in an RtD framework to investigate the nature of dreaming. This process produced the ‘dream sticker machine’, a design proposal that visualises a user’s dreams with an AI image generator. This tangible dream trace becomes sharable in both public and personal spaces. This proposal formed the basis of a probe to how people might imagine their dreams taking form in the real world, and led to ideas for how the materials of dreams might be taken up as part of design research practice.
SESSION: Reimagining Workflows and AI
Session details: Reimagining Workflows and AI
GenFrame – Embedding Generative AI Into Interactive Artifacts
Image-generation AI models have triggered a paradigm shift in how we can express ourselves in visual art. Despite their widespread use in a short amount of time, embedding these models into interactive artifacts is still largely unexplored. In this pictorial, we unpack the design and development process of GenFrame, an image generating picture frame that utilizes generative AI capabilities to mimic traditional paintings. Our work details the necessary steps to integrate generative AI into interactive artifacts and highlights important design considerations for controlling image-generation models in order to achieve specific design intents. Our insights provide interaction designers with a more comprehensive understanding and approach towards utilizing image-generation AI models for interactive artifacts. A demo can be viewed at https://youtu.be/1rhW4fazaBY
AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
Clients often partner with AI experts to develop AI applications tailored to their needs. In these partnerships, careful planning and clear communication are critical, as inaccurate or incomplete specifications can result in misaligned model characteristics, expensive reworks, and potential friction between collaborators. Unfortunately, given the complexity of requirements ranging from functionality, data, and governance, effective guidelines for collaborative specification of requirements in client-AI expert collaborations are missing. In this work, we introduce AINeedsPlanner, a workbook that AI experts and clients can use to facilitate effective interchange of clear specifications. The workbook is based on (1) an interview of 10 completed AI application project teams, which identifies and characterizes steps in AI application planning and (2) a study with 12 AI experts, which defines a taxonomy of AI experts’ information needs and dimensions that affect the information needs. Finally, we demonstrate the workbook’s utility with two case studies in real-world settings.
PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers’ Workflows
Prototyping AI applications is notoriously difficult. While large language model (LLM) prompting has dramatically lowered the barriers to AI prototyping, designers are still prototyping AI functionality and UI separately. We investigate how coupling prompt and UI design affects designers’ workflows. Grounding this research, we developed PromptInfuser, a Figma plugin that enables users to create semi-functional mockups, by connecting UI elements to the inputs and outputs of prompts. In a study with 14 designers, we compare PromptInfuser to designers’ current AI-prototyping workflow. PromptInfuser was perceived to be significantly more useful for communicating product ideas, more capable of producing prototypes that realistically represent the envisioned artifact, more efficient for prototyping, and more helpful for anticipating UI issues and technical constraints. PromptInfuser encouraged iteration over prompt and UI together, which helped designers identify UI and prompt incompatibilities and reflect upon their total solution. Together, these findings inform future systems for prototyping AI applications.
Understanding Human-AI Workflows for Generating Personas
One barrier to deeper adoption of user-research methods is the amount of labor required to create high-quality representations of collected data. Trained user researchers need to analyze datasets and produce informative summaries pertaining to the original data. While Large Language Models (LLMs) could assist in generating summaries, they are known to hallucinate and produce biased responses. In this paper, we study human–AI workflows that differently delegate subtasks in user research between human experts and LLMs. Studying persona generation as our case, we found that LLMs are not good at capturing key characteristics of user data on their own. Better results are achieved when we leverage human skill in grouping user data by their key characteristics and exploit LLMs for summarizing pre-grouped data into personas. Personas generated via this collaborative approach can be more representative and empathy-evoking than ones generated by human experts or LLMs alone. We also found that LLMs could mimic generated personas and enable interaction with personas, thereby helping user researchers empathize with them. We conclude that LLMs, by facilitating the analysis of user data, may promote widespread application of qualitative methods in user research.
Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow
Generative AI brings novel and impressive abilities to help people in everyday tasks. There are many AI workflows that solve real and complex problems by chaining AI outputs together with human interaction. Although there is an undeniable lure of AI, it is uncertain how useful generative AI workflows are after the novelty wears off. Additionally, workflows built with generative AI have the potential to be easily customized to fit users’ individual needs, but do users take advantage of this? We conducted a three-week longitudinal study with 12 users to understand the familiarization and customization of generative AI tools for science communication. Our study revealed that there exists a familiarization phase, during which users were exploring the novel capabilities of the workflow and discovering which aspects they found useful. After this phase, users understood the workflow and were able to anticipate the outputs. Surprisingly, after familiarization the perceived utility of the system was rated higher than before, indicating that the perceived utility of AI is not just a novelty effect. The increase in benefits mainly comes from end-users’ ability to customize prompts, and thus potentially appropriate the system to their own needs. This points to a future where generative AI systems can allow us to design for appropriation.
DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI
Visually oriented designers often struggle to create effective generative AI (GenAI) prompts. A preliminary study identified specific issues in composing and fine-tuning prompts, as well as needs in accurately translating intentions into rich input. We developed DesignPrompt, a moodboard tool that lets designers combine multiple modalities — images, color, text — into a single GenAI prompt and tweak the results. We ran a comparative structured observation study with 12 professional designers to better understand their intent expression, expectation alignment and transparency perception using DesignPrompt and text input GenAI. We found that multimodal prompt input encouraged designers to explore and express themselves more effectively. Designer’s interaction preferences change according to their overall sense of control over the GenAI and whether they are seeking inspiration or a specific image. Designers developed innovative uses of DesignPrompt, including developing elaborate multimodal prompts and creating a multimodal prompt pattern to maximize novelty while ensuring consistency.
SESSION: Multisensory Healthcare Transitions
Session details: Multisensory Healthcare Transitions
Mariana’s Song: Materializing Personhood through Non-Linear Multisensory Experiences Designed for People Living with Advanced Dementia
Mariana’s Song is a multisensory experience for care home residents unable to take part in regular activities. Designed to support Personhood through a blended physical/digital installation, it is a user-led experience, centered around video and closely facilitated in concert with expert arts practitioners. The narrative and duration of the interaction is guided by each resident, unique on each viewing, which includes ‘sensory props’ that guide each resident’s experience and stimulate the senses to create a transformative space. We provide a worked example of digital, physical, and spatial design decisions taken to reinforce Personhood for residents with advanced dementia. We present learnings from 106 sessions, facilitated one to one between Artists and residents. We contribute to the literature around designing to Materialize Personhood, informed by Kitwood’s 12 key characteristics, as well as Parametricism, a novel approach to non-linear experiences, to support the unique needs of people with advanced dementia.
Voice Assistants for Mental Health Services: Designing Dialogues with Homebound Older Adults
The number of older adults who are homebound with depressive symptoms is increasing. Due to their homebound status, they have limited access to trained mental healthcare support, which leaves this support often to untrained family caregivers. To increase access, a growing interest is placed on using technology-mediated solutions, such as voice-assisted intelligent personal assistants (VIPAs), to deliver mental health services to older adults. To better understand how older adults and family caregivers intend to interact with a VIPA for mental health interventions, we conducted a participatory design study during which 6 older adults and 7 caregivers designed VIPA-human dialogues for various scenarios. Using conversation style preferences as a starting point, we present aspects of human-likeness older adults and family caregivers perceived as helpful or uncanny, specifically in the context of the delivery of mental health interventions, which helps inform potential roles VIPAs can play in mental healthcare for older adults.
Exploring Using Personalised Comics for Healthcare Communication for Patients Living With Hemodialysis
Through co-design with patients undergoing hemodialysis and their healthcare professionals, we worked towards discovering how to create a personalised, welcoming, yet quick and accurate method for medical instruction communication. Exploring possibilities of meeting the widely differing goals of patients and their healthcare professionals led to designing a personalise-able method for creating comics. Through ongoing discussions during the comic creation process, we explored variations in comic styles and personalisation factors such as choosing and modifying the appearance of the comic personalities, the settings, the central topics, and word usage to create the comics. Interest in using the approach that supports the creation of medical comics was high among patients and healthcare professionals. Rich feedback was obtained about information to be included and future direction for such medical comic creation support. We reflect on lessons learned during co-design with healthcare givers and patients.
“It depends”: Configuring AI to Improve Clinical Usefulness Across Contexts
Artificial Intelligence (AI) repeatedly match or outperform radiologists in lab experiments. However, real-world implementations of radiological AI-based systems are found to provide little to no clinical value. This paper explores how to design AI for clinical usefulness in different contexts. We conducted 19 design sessions and design interventions with 13 radiologists from 7 clinical sites in Denmark and Kenya, based on three iterations of a functional AI-based prototype. Ten sociotechnical dependencies were identified as crucial for the design of AI in radiology. We conceptualised four technical dimensions that must be configured to the intended clinical context of use: AI functionality, AI medical focus, AI decision threshold, and AI Explainability. We present four design recommendations on how to address dependencies pertaining to the medical knowledge, clinic type, user expertise level, patient context, and user situation that condition the configuration of these technical dimensions.
“Be with me and stay with me”: Insights from Co-Designing a Digital Companion to Support Patients Transitioning from Hospital to Cardiac Rehabilitation
Cardiovascular diseases (CVDs) are a major global health challenge, compounded by a significant systemic treatment gap: the underutilization of cardiac rehabilitation (CR). Central to this issue is the crucial yet underperforming referral process to CR due to various factors, including patients’ lack of information. By co-designing a CR referral assistant with cardiac patients and healthcare professionals, this work explores how technology can support patient journeys from the acute hospital to CR, overcoming existing healthcare system barriers. This work (1) contributes a map of patients’ evolving needs tailored to their pathway, (2) provides design implications for interactive digital health technologies aiming to facilitate patient transitions across established healthcare system boundaries, and (3) discusses the potential of technology as a “patient companion” during their transition to CR.
Enhancing Accuracy, Time Spent, and Ubiquity in Critical Healthcare Delineation via Cross-Device Contouring
Improving accuracy, time spent, and ubiquity of delineation has been a long-standing design aim, yet many HCI works have overlooked high-stakes and complex healthcare annotation. We explore contouring, a critical workflow aimed at identifying and segmenting tumors, usually performed on immobile desktop computers in clinics, in which limited support for mobile access leads to prolonged and subpar treatment planning. Following interviews and think-aloud studies (N = 10 physicians), we report key contouring behaviors, and later design a novel cross-device prototype that enables contouring on everyday touch devices. We compared contouring via desktop and touch in a lab study (N = 8 residents) and found that mobile phones not only yielded similar accuracy, but also took significantly less time. Our results point to three broad design guidelines for cross-device solutions deployed within standalone healthcare workflows, and highlight how incorporating different device and input modalities can improve treatment delivery in today’s distributed healthcare environments.
SESSION: Dancing, Cocreating and Explainable AI
Session details: Dancing, Cocreating and Explainable AI
DanceGen: Supporting Choreography Ideation and Prototyping with Generative AI
Choreography creation requires high proficiency in artistic and technical skills. Choreographers typically go through four stages to create a dance piece: preparation, studio, performance, and reflection. This process is often individualized, complicated, and challenging due to multiple constraints at each stage. To assist choreographers, most prior work has focused on designing digital tools to support the last three stages of the choreography process, with the preparation stage being the least explored. To address this research gap, we introduce an AI-based approach to assist the preparation stage by supporting ideation, creating choreographic prototypes, and documenting creative attempts and outcomes. We address the limitations of existing AI-based motion generation methods for ideation by allowing generated sequences to be edited and modified in an interactive web interface. This capability is motivated by insights from a formative study we conducted with seven choreographers. We evaluated our system’s functionality, benefits, and limitations with six expert choreographers. Results highlight the usability of our system, with users reporting increased efficiency, expanded creative possibilities, and an enhanced iterative process. We also identified areas for improvement, such as the relationship between user intent and AI outcome, intuitive and flexible user interaction design, and integration with existing physical choreography prototyping workflows. By reflecting on the evaluation results, we present three insights that aim to inform the development of future AI systems that can empower choreographers.
Designing Interactive Explainable AI Tools for Algorithmic Literacy and Transparency
As artificial intelligence (AI) increasingly permeates everyday life, there is a growing need for public understanding of AI’s underlying principles. Existing educational interventions and explainable AI (XAI) tools cater mainly to children or adult experts. In this paper, we present three interactive web-based tools to foster AI learning among adults without technical backgrounds. Designed according to learning sciences and user-centered design principles, these tools simplify complex AI concepts like edge detection, confidence thresholds, and sensitivity, making AI more understandable for beginners and facilitating reflection on ethical issues. We present results from a mixed-methods evaluation of the tools with 42 participants. Results show heightened familiarity and confidence in AI concepts. Our qualitative analysis additionally reveals common interaction patterns amongst participants. This paper offers both a design contribution to the AI education and XAI communities and emergent interaction patterns to support the design of transparent and learner-centered AI for adult novices.
PodReels: Human-AI Co-Creation of Video Podcast Teasers
Video podcast teasers are short videos that can be shared on social media platforms to capture interest in full episodes of a video podcast. These teasers enable long-form podcasters to reach new audiences and gain more followers. However, creating a compelling teaser from an hour-long episode can be challenging. Selecting interesting clips requires significant mental effort; editing the chosen clips into a cohesive, well-produced teaser is time-consuming. To support the creation of video podcast teasers, we first investigated what makes a good teaser. We combined insights from audience comments and creator interviews to identify key ingredients. We also identified a common workflow used by creators during this process. Based on these findings, we developed a human-AI co-creative tool called PodReels to assist video podcasters in crafting teasers. Our user study demonstrated that PodReels significantly reduces creators’ mental demand and improves their efficiency in producing video podcast teasers.
Co-Creating Question-and-Answer Style Articles with Large Language Models for Research Promotion
Research promotion enables researchers to share advanced knowledge with pertinent academic communities. The question-and-answer (QA) style articles are effective for researchers to promote their research by enabling readers to understand research on complex subjects. Recent advances in large language models (LLMs) have opened avenues for supporting researchers in creating QA-style articles for research promotion. However, without the authors’ involvement, these models may only partially capture the researcher’s intention and voice. We developed AQUA, a research probe that enables researchers to co-create QA-style articles with LLMs to promote their research papers. A user study (n=12) reveals that LLMs reduced authors’ burden and helped them understand the readers’ perspectives. Nevertheless, LLMs failed to capture the unique intent of the authors, and their automated generation discouraged authors from carefully revising their answers. Based on our findings, we discuss human-LLM interaction design to enable authors to create QA-style articles that reflect their intention.
Requirements and Attitudes towards Explainable AI in Law Enforcement
Decision-making aided by Artificial Intelligence in high-stakes domains such as law enforcement must be informed and accountable. Thus, designing explainable artificial intelligence (XAI) for such settings is a key social concern. Yet, explanations are often misunderstood by end-users due to being overly technical or abstract. To address this, our study engaged with police employees in the Netherlands, who are users of a text classifier. We found that for them, usability and usefulness are of great importance in explanation design, whereas interpretability and understandability are less valued. Further, our work reports on how design elements included in machine learning model explanations are interpreted. Drawing from these insights, we contribute recommendations that guide XAI system designers to cater to the specific needs of specialized users in high-stakes domains and suggest design considerations for machine learning model explanations aimed at domain experts.
The AI-DEC: A Card-based Design Method for User-centered AI Explanations
Increasing evidence suggests that many deployed AI systems do not sufficiently support end-user interaction and information needs. Engaging end-users in the design of these systems can reveal user needs and expectations, yet effective ways of engaging end-users in the AI explanation design remain under-explored. To address this gap, we developed a design method, called AI-DEC, that defines four dimensions of AI explanations that are critical for the integration of AI systems—communication content, modality, frequency, and direction—and offers design examples for end-users to design AI explanations that meet their needs. We evaluated this method through co-design sessions with workers in healthcare, finance, and management industries who regularly use AI systems in their daily work. Findings indicate that the AI-DEC effectively supported workers in designing explanations that accommodated diverse levels of performance and autonomy needs, which varied depending on the AI system’s workplace role and worker values. We discuss the implications of using the AI-DEC for the user-centered design of AI explanations in real-world systems.
SESSION: Care, Collective Resistance and More-than-Humans
Session details: Care, Collective Resistance and More-than-Humans
Articulating Felt Senses for More-Than-Human Design: A Viewpoint for Noticing
To rigorously approach the more-than-human world in design research, we need to become more receptive and better equipped to describe the complexities of relationality. In response, this paper advocates for the articulation of the felt sense -or tacit knowledge residing in our bodies- as a viewpoint for noticing. Assisted by micro-phenomenological interviews, we carefully described our felt senses from our experiences with a telepresence robot and smartphone photography. We illustrate how this viewpoint allowed us to access our pre-judgemental dimension, the vivid liveliness in our experiences with technologies, and the porosity of our sense of self. We contribute the felt sense as a viewpoint for noticing to design researchers interested in integrating their somatic sensibilities into their work with the more-than-human, allowing them to attune to, describe and share with other researchers the normally unattended dimension of our experiences, including aspects concerning the felt dimension of ethics.
Crip Reflections on Designing with Plants: Intersecting Disability Theory, Chronic Illness, and More-than-Human Design✱
Through an autoethnographic account of designing, exhibiting, and maintaining an interactive bioart installation with plants, we trace intersections between more-than-human design, disability theory, and lived experiences of chronic illness. Specifically, we deconstruct three “polished” exhibits of our installation through stories of breakdowns and failures, organized in three main themes: maintenance and care, buggy biodata, and collective resistance to purification and control. Our reflections show how plants, technologies, and a chronically ill body became entangled with each other conceptually and materially, surfacing new sites for more-than-human relationalities. In our discussion, we unpack how disability perspectives can expand more-than-human design practices, highlight opportunities for re-imagining exhibition spaces, and offer adaptation as a strategy for design in HCI.
Phenology Circles: A Method to Deepen Relations in More-Than-Human Design Processes
We present phenology circles, a method to deepen relations human and more-than-human in the backstage of participatory design processes. To illustrate the method, we reflect on over two years of activity in an online, global gardening community of practice (our phenology circle) initiated to coordinate and communicate with participants in the backstage of our research. Related to our research aims, the community was initiated to attune ourselves to the rhythms and interrelations of other species and between one another. Through this approach, we better understood interaction and deepened relations between humans and the more-than-human (e.g., plants, animals, spirits). The community (N=42) has shared over 1,200 images and textual posts regarding thoughts about the environment, concerns, and experiences. Reflecting on activity of eight core members, we identified shifts from scientific-natured posts to imbue sentience and storytelling, making possible a variety of cross-cultural and symbiotic encounters to appear in our online site and fostering community to encounter and embrace difference. Phenology circles illuminate more-than-human concerns intertwined with different human values and beliefs. We discuss the value of science, sentience and storytelling, reflect on facilitation, and illustrate a design pathway that links the backstage to frontstage design activities.
Embodied Traces: Multispecies Entanglement in Urban Spaces
Collisions with man-made structures such as buildings, vehicles, and energy infrastructure are a significant threat to bird populations. Throughout the U.S., groups of volunteers monitor bird-building collisions to better understand the environmental impact of collisions and to advocate for the use of bird-safe building materials. The first author participated in monitoring bird-building collisions in Atlanta during the spring and fall migration seasons of 2023. This pictorial is a reflection on the experience of producing this data based on interviews with volunteers and the experiences and photographs of the first author. We contribute a rich account of the experiential elements of data production and a discussion of more-than-human entanglements in urban spaces.
Tracing as a Strategy for Orienting to Nonhuman Perspectives
Trends in designing with living organisms have focused mainly on leveraging the organisms qualities for interactivity and designing for care through a more-than-human design lens. As DIS gains interest in interactions with living organisms and more-than-human design goals, we reflect upon these trends and call for practices that surface nonhuman perspectives and critically draw out our own. We propose tracing as a material practice that surfaces insights on care and relation in the rich space between the human-designer and organism. Through reflective writings, we present a first-person account of tracing slime mold through visualizations, embodied experiences, and sonifying electrical resistance. These tracings capture both the slime mold’s ways and our own, inviting us to observe our actions to facilitate its life. By tracing experiences of growing and making with slime mold, we re-orient toward nonhuman perspectives and discuss how such practices might advance capacities of design research.
Seeding a Repository of Methods-To-Be for Nature-Entangled Design Research
We share an emergent repository of nature-entangled methods-to-be shared, experimented with, and discussed during a conference workshop. We present them in-use, as they are in formation. We do not seek to theorise or even fully articulate these methods-to-be. Rather, to make them approachable and actionable for others by showing them not fully polished. By doing this, we advocate for increased transparency in the difficulties of creating new methods, techniques, tools, and approaches. Our contribution is threefold: we provide 1) an annotated portfolio of methods-to-be; 2) illustrative examples of how cross-pollination of these methods can enrich their situated use; and 3) a discussion of ways to further articulate the methods and deepen reflection on their roles in nature-entangled design processes.
SESSION: Fashion and Fabrication
Session details: Fashion and Fabrication
A Year of Interaction Around Town: Gathering Traces with an Interactive Knitting Machine and Community Stitch Markers
Devices for digital fabrication are becoming increasingly smaller and more portable, enabling digital fabrication research to move into new environments. In this exploratory research-through-design project, we aimed to physicalize data on-the-go using a portable digital fabrication device, and gathered community annotations and traces of the journey the machine went on with “stitch markers”. We describe the development of The Life of a Small Town, a portable knitting machine that was adapted to knit rows of stitches in response to sensor data. The machine travelled throughout a small town and “popped up” at local art events to sense and physicalize social gatherings held by an arts organization over the period of one year. Individuals participating in events could also decorate their own “stitch markers” to annotate and pin to the data. In this paper we discuss the insights from an analysis of the stitch markers and traces of the year-long journey.
SeamSleeve: Robust Arm Movement Sensing through Powered Stitching
Despite significant advances in interactive clothing over the past decade, e-textiles lack traditional fabric robustness and comfort. SeamSleeve provides a method for using garment seams as sensing channels while retaining the benefits of regular clothing design. We power conductive seams at low voltages to stitch together everyday fabric panels, resulting in a novel sensing mechanism capable of detecting arm movements. Our first empirical study (n=10) identifies optimal seam design and placement on the sleeve, by comparing traditional seam forms in the context of sensing capabilities. A second study (n=14) demonstrates that our minimal sensing approach is capable of successfully classifying 8 arm exercises with an accuracy of 84%. Our findings support the effectiveness of the approach in areas such as longitudinal physiotherapeutic rehabilitation beyond the clinic, enabling everyday motion capture.
Entering the 3D printer: negotiations of imprecision in making.
3D printing generally makes use of CAD software and slicers to mediate designerly intentions and fabrication instructions to the machine. This can obstruct the making process to such a degree that the designer risks becoming an observer of an ongoing print. To trouble this situation, we devised a manual control that enables the designer to manipulate one of the axes of the 3D printer. Through this embodied process, the incoming toolpath becomes another material to be extruded together with the 3D printing filament. The negotiations of (im)precision and intent between designer and machine are recorded and can be used as input for a new beginning. In this pictorial, we share our annotated 3D prints as a way of highlighting and paying attention to the relations between code, material, machine, and designer. We contribute with the system and reflections on embodied fabrication.
Engaging Young People in the Expressive Opportunities of Digital Fabrication Through Craft-Oriented CAM-Based Design
Digital fabrication can be a rich creative domain for young people. Most youth-oriented digital fabrication activities focus on CAD, where practitioners design solid geometry. Professional designers frequently bypass CAD and instead design at the level of the machine toolpath. This CAM-based design process allows for material exploration, unique textures, and complex shapes. We recognize young people’s role as designers and seek to adapt professional digital fabrication techniques to meaningful youth design activities. We present a youth-oriented CAM-based design curriculum consisting of 1) hands-on activities that introduce machine toolpath concepts, 2) a learner-oriented CAM software interface that scaffolds digital toolpath creation, and 3) project-oriented design activities that integrate CAM-based design and manual craft for ceramics and textile production. We evaluated our approach through a workshop with twelve high school students. Our research shows that young people can skillfully apply CAM-based design and manual craft to create functional and personally meaningful artifacts.
Animated Linen: Using High-twist Hygromorphic Yarn to Produce Interactive Woven Textiles
HCI research has demonstrated that textiles have interactive potential, with the ability to transform or self-shape, whether through material, structure, or the addition of non textile elements. Many materials for developing interactive or animated textiles – textiles which change during use – are fossil fuel-based, require electricity for activation, or are only available in small quantities. In this pictorial we present our exploration of high-twist linen yarn as an actuator material in woven textiles. Through experimental design research, we have defined key parameters affecting use of the material, and identified combinations of material and structure producing contrasting textile movement. The resulting woven textiles may be activated by spraying with water, or in high humidity, and the actuation is repeatable after drying, offering multiple modes of interaction. We offer proposals for HCI applications for the animated linen yarn, alongside a guide to facilitate producing and designing animated woven linen textiles.
Practice-driven Software Development: A Collaborative Method for Digital Fabrication Systems Research in a Residency Program
Building new software tools for professional digital fabrication requires that HCI researchers understand domain-specific materials and fabrication workflows to ensure software operations align with professional manufacturing requirements. To bridge the research-practice divide, we adopt a practice-driven software development methodology for digital fabrication in an artist-in-residence program. In our method, HCI researchers and craft professionals collaboratively develop software tools over three months. We piloted our methodology through two consecutive computational ceramics residencies with five professional craftspeople. The teams produced five novel software tools for clay 3D printing and hundreds of ceramic artifacts. We provide a detailed description of our methodology through artist and HCI researcher accounts and an analysis of the integration of software ideation, implementation, and debugging with professional art and craft production. Our work demonstrates a systematic mechanism for achieving meaningful digital fabrication software contributions with mutual benefit for artists and researchers.
SESSION: Data Explorations, Personas & UI
Session details: Data Explorations, Personas & UI
Enabling Tabular Data Exploration for Blind and Low-Vision Users
In a data-driven society, being able to examine data on one’s own terms is crucial for various aspects of well-being. However, current data exploration paradigms, such as Exploratory Data Analysis (EDA), heavily rely on visualizations to unveil patterns and insights. This visual-centric approach poses significant challenges for blind and low-vision (BLV) individuals. To address this gap, we built a prototype that supports non-visual data exploration and conducted an observational user study involving 18 BLV participants. Participants were asked to conduct various analytical tasks, as well as free exploration of provided datasets. The study findings provide insights into the factors influencing inefficient data exploration and characterizations of BLV participants’ analytical behaviors. We conclude by highlighting future avenues of research for the design of data exploration tools for BLV users.
“There’s Something About Noura”: Exploring Think-Aloud Reasonings for Users’ Persona Choice in a Design Task
Stakeholders like designers use personas to learn about users. After persona development, stakeholders are usually presented with a persona set. However, there is little research on how stakeholders select a persona from a persona set. A think-aloud analysis with 37 stakeholders who were asked to select a persona for a content design task reveals that persona selection is influenced by comparative, non-comparative, and subjective elements. Persona choice is often made with task compatibility in mind: interests, professions, and education were important contextual factors in our focal task. Storifying is commonly applied by stakeholders, reflecting personas’ narrative nature. The persona’s picture is often evoked, in addition to nationality and name, though demographics do not play a decisive role. Stakeholders refer to a host of persona attributes when explicating their persona choice. Overall, reasonings for persona choice are multifaceted and individualistic, as we might expect given the information-richness of personas.
RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual Effects
This paper introduces RealityEffects, a desktop authoring interface designed for editing and augmenting 3D volumetric videos with object-centric annotations and visual effects. RealityEffects enhances volumetric capture by introducing a novel method for augmenting captured physical motion with embedded, responsive visual effects, referred to as object-centric augmentation. In RealityEffects, users can interactively attach various visual effects to physical objects within the captured 3D scene, enabling these effects to dynamically move and animate in sync with the corresponding physical motion and body movements. The primary contribution of this paper is the development of a taxonomy for such object-centric augmentations, which includes annotated labels, highlighted objects, ghost effects, and trajectory visualization. This taxonomy is informed by an analysis of 120 edited videos featuring object-centric visual effects. The findings from our user study confirm that our direct manipulation techniques lower the barriers to editing and annotating volumetric captures, thereby enhancing interactive and engaging viewing experiences of 3D volumetric videos.
“Malicious” Pictorials: How Alt Text Matters to Screen Reader Users’ Experience of Image-Dense Media”
People who are blind face accessibility and usability barriers every time a digital image author omits or creates uninformative alternative or “alt” text. Often, content creators are simply unaware of screen reader users’ experience and accessibility best practices; there is also a lack of research knowledge about how to design accessible image-dense media, like pictorials. We present (1) a content analysis of 149 pictorials, (2) a thematic analysis of three think-aloud sessions with an expert screen reader user, and (3) practical lessons learned from a co-design workshop with three blind scholars. Our research finds that pictorials are increasingly accessible, but they are still far from providing a usable experience. We identify nine “malicious” alt text patterns that riddle pictorials and three novel accessible alt text patterns to guide authors of image-dense media.
GraspUI: Seamlessly Integrating Object-Centric Gestures within the Seven Phases of Grasping
Objects are indispensable tools in our daily lives. Recent research has demonstrated their potential to act as conduits for digital interactions with microgestures, however, the primary focus was on situations where the hand firmly grasps an object. We introduce GraspUI, an exploratory design space of object-centric gestures within the seven distinct phases of the grasping process, spanning pre-, during, and post-grasp movements. We conducted ideation sessions with mixed-reality designers from industry and academia to explore gesture integration throughout the entire grasping process. The outcome was 38 storyboards envisioning practical applications. To evaluate the design space’s utility, we performed a video-based assessment with end-users. We then implemented an interactive prototype and quantified the overhead cost of performing proposed gestures through a secondary study. Participants reacted positively to gestures and could integrate them into existing usage of objects. To conclude, we highlight technical and usability guidelines for implementing and extending GraspUI systems.
Fidgets: Building Blocks for a Predictive UI Toolkit
The rapid growth of AR platforms, combined with the rising predictive power of intelligent systems, will fundamentally change interactive computing. Interaction will increasingly happen on the go, causing I/O to become constrained, ultimately leading to reliance on user intent prediction for aid. In this pictorial, we argue that to support the development of such systems, new predictive UI toolkits are required. We place the reader in the shoes of an App designer and outline the challenges that will be faced. We then describe a new predictive toolkit, leveraging Fuzzy Widgets, or “Fidgets” as the main UI building block. Fidgets extend Responsive Design into the realm of intelligent systems, to adapt not only to spatial constraints, but to system predictions as well. We then describe a working implementation of a predictive music application, built using our described framework, showcasing its benefits and range of adaptive abilities.
SESSION: Designing Healthcare Technologies
Session details: Designing Healthcare Technologies
Children’s perspectives on pain-logging: Insights from a Co-Design Approach
Pain is an essential indicator of health and guides clinical treatments. Logging pain is important in supporting this. However, there is little research into pre-adolescent children’s pain logging tools. Utilising the Bluebells method to engage children as co-designers, we gathered children’s perspectives on pain-logging tools; in the first workshop by using tangible design approaches to support creative thinking, and in the second workshop by discussing developed prototypes based on the children’s designs. Our findings highlight design concepts that the research team – despite many years of pain-related research – had not considered in the context of paediatric logging, namely a) prioritizing children’s privacy in social settings while using pain-logging tools; b) emphasizing personalization to boost engagement; and c) logging general well-being of children alongside pain intensity to collect more insightful data. These findings thus demonstrate the value of co-designing pain-logging technologies with children.
Understanding How Parents Deal With the Health Advice They Receive: A Qualitative Study and Implications for the Design of Message-based Health Dissemination Systems for Child Health
Message-based health information dissemination systems can potentially improve maternal and child health (MCH). By conveying health information to parents, SMS- and chatbot-based systems can support parents’ learning and empower them to make better health decisions for their children. However, there is limited design advice for creating message-based dissemination systems for MCH. To help address this gap, we conducted 14 participatory workshops with 42 parents from Portugal and South Africa, exploring how parents learned to care for their children’s health. Our findings showed how parents reflected on the health advice they received, by assessing the fit of the advice to their child’s characteristics, their values and beliefs, the advice’s feasibility, or the intention and competence of the advice giver. Based on these insights, we propose four design implications for creating message-based health information dissemination systems tailored to parents and their children.
Porous by Design: How Childcare Platforms Impact Worker Personhood, Safety, and Connection
Care work is always already unequal. It involves looking after others’ physical, psychological, emotional, and developmental needs. Paid care work tends to be conducted in private spaces, lack regulation, and reproduce unequal dynamics between clients and workers. These conditions lead to porous boundaries, a permeability experienced by workers between care and work, professional and personal, and private and public (sectors and spheres). Drawing on interviews with 16 workers who find work using Care.com, we argue that the porous boundaries of care work are reified in new ways through the design and use of emerging digitally mediated matching platforms. This has particular impacts for ranking personhood, reducing worker safety, and increasing atomization. In contrast, we find benefits in the forum-like structure and visible, interactive conversations of other platforms used to access childcare work. We end with a discussion of porousness by design and the trouble of locating design within worker platforms.
Functionality and User Review Analysis of Mobile Apps for Mindfulness Eating and Eating Disorders
A growing number of mobile apps have focused on healthy or problematic eating, albeit limited research has focused on evaluating such apps from users’ perspectives. To address this, we evaluated the functionalities of 27 apps on mindfulness eating, and eating disorders from the Apple App, and Google Play Stores, and conducted a content analysis of 1248 user reviews, totaling over 60,000 words. Findings indicate the main functionalities of tracking data on eating behaviors, emotions, thoughts, bodily sensations, symptoms, as well as triggers of eating disorders, and of providing interventions such as mindfulness, goal setting, psychoeducation, CBT, and holistic ones. Findings also highlight key usability and ethical challenges, which we used to inform five design implications namely tracking and reflecting on multiple aspects of mindfulness and healthy eating, supporting personalized interventions and AI-based ones, as well as the sensitive design for diagnosis, and for tracking and monitoring problematic data.
Hicclip: Sonification of Augmented Eating Sounds to Intervene Snacking Behaviors
In this paper, we present a field study on using sonification of augmented eating sounds to intervene snacking behaviors in daily routines. The sonic feedback achieved through a snack storing device named Hicclip for verifying snacking behaviors and producing augmented eating sounds. The study was conducted with nine participants who were commonly addicted to snacking. The effectiveness of the sonification was examined by comparing snack-related data and questionnaire over the three study weeks: a baseline week, a Hicclip intervention week, and a post-intervention week. We also analyzed interview results to understand user experiences and opportunities for future research. Quantitative results showed that the snacking pattern has been improved due to reduced eating duration and snack consumptions. Qualitative results suggested that Hicclip may benefit self-regulation, afford easy adoption, and support data acquisition. We discuss design implications for embodiment of augmented eating sounds for healthy snacking.
SESSION: Expanding Human-Nature Relations
Session details: Expanding Human-Nature Relations
Designing for Interdependence of Bees, Garden, Designer, and the Changing Season
The speculation of alternative relationships between humans and nonhumans is a crucial element of examining our dominating stance towards nature amidst the ongoing environmental crisis. The notion of designing for interdependence challenges this status quo, focusing on a local ecosystem and embedding carefully crafted artefacts to foster more-than-human relationships. The presented work engages with the intricate interdependence of bees, artefacts, time and location through an ongoing case study of designing with and living with red mason bees (Osmia bicornis). Unpacking the first author’s design journey, we elucidate how attentive observations of the garden and engagement in activities such as the making of a beehouse have mobilized caring for the local ecosystem. With our focus on relationships emerging from situating oneself in a location for multispecies cohabitation, our research describes the dynamics of sharing resources between humans and nonhumans and its potential as a repertoire of more-than-human design.
Encountering Human-Plant Relations: a Discussion of How Interaction Design Can encourage Human Sensibility to More-than-human Plants
In this paper we discuss how interaction design experiments and interventions can play a role in sensitising humans to the more-than-human—specifically to plants and their significance to our lifeworld. Our work is rooted in two ideas (1) that deepening the understanding of human-plant relations is important for navigating the multiple planetary crises that we are currently facing, and (2) that interaction design allows for exploring the posthuman ontological model that rejects human exceptionalism and regards all entities as entangled and mutually constitutive. In discussing how design on several levels makes possible more-than-human reflections, the paper presents qualitative data gathered from interviews, diary entries, and observations from deploying two of our plant-electronics designs. Through identifying and organising surprises and recurring patterns in how people responded into three types of sensibilities (hermeneutic, existential, and socio-cultural), we discuss what happens when interaction design mediates entanglements between humans, plants, and other more-than-human actors.
Algae Alight: Exploring the Potential of Bioluminescence through Bio-kinetic Pixels
Incorporating living microorganisms in artifacts offers opportunities for novel modes of expression and interaction. Bioluminescent algae are unicellular microorganisms that produce light in response to kinetic stimuli and have been a focus of design and HCI research when exploring expressivity of living media. This study advances prior work using bioluminescent algae through designing and engineering a Living Light Interface comprising of bio-kinetic pixels. The resulting interactive system translates digital input into the biological domain by modulating the bioluminescent mechanism and creating different pixel states. The kinetic design of the vibration module uses adjustable weights to induce a wide range of lighting patterns. The hardware design is coupled with organism-centric algorithms, which allow for the generation of dynamic light patterns across the interface. The paper provides a comprehensive visual narrative of a design process that brings these living organisms to the forefront of our technological imagination, blurring the boundaries between biology, algorithmic control, and tangible interfaces.
Dear Nature: Using data drawings to promote sensemaking in human-nature relations
Humans are experiencing both anxiety and hopelessness regarding how they, as individuals, can affect positive change to mitigate climate change in complex human-made systems. This exploratory design research investigates how data drawings, created from personal, place-based observations of nature, can enhance human-nature connection and sensemaking in the Anthropocene. Can cooperative inquiry across different geographies, and the consequent confrontation of experiences, lead to a deeper awareness of larger, more-than-human systems? We introduce (1) a post-humanist design method called Dear Nature for supporting mindful reflection and sensemaking in human-nature relations, (2) our methodology and process based on data humanism, (3) analyses and outcomes from a five-week-long data collection and (4) considerations regarding how we can explore human-nature data reflections collectively to decenter the human in design. This first humble step towards non-anthropocentric design could lead to insights into post-humanist design thinking and the role of the human (and more-than-human) in designed systems.
Nature Networks: Designing for nature data collection and sharing from local to global
Human-nature relations are formed by the social and economic forces underlying local, physical places. Yet, many citizen science platforms are designed for national or global nature data collection. How local knowledge of wildlife can be effectively assembled to support this is an open question. This study explored what a localised nature network might look like and how to design for what people care about, notice, record, and share. We undertook research with 14 participants in an inner-city location, using interviews and a variety of technology probes. We found that participants’ social and physical boundaries, values regarding family and community, and diverse questions impacted human-nature relations and nature data collection. Speculated physical-digital networks indicated novel combinations of platforms and technologies to drive ongoing engagement and learning. This study contributes insight into how nature data is formed by the realities of locality, and how a fractured nature/data landscape might be mended.
CoRoot: A Collaborative Planting System to Support Connection between Grandparents and Young Adults
Grandparent-grandchild relationships are essential and evolving in life. As grandchildren enter adulthood and grandparents age, it becomes harder to maintain close relationships between grandparents and young adults. To facilitate intergenerational relationships, we developed CoRoot, a collaborative planting system for connecting grandparents-adult grandchildren. In this system, grandparents and grandchildren grow vegetables together (Planting Toolkit) and share the growing process or stories (Vegetable Planting Diary). We conducted a preliminary study with two families to identify design issues and refine the prototype. Subsequently, we conducted a field study with four families for six weeks to better understand real-world use. We found that CoRoot promotes the connection between grandparents and young adults. In collaborative planting, grandparents shared their knowledge and life experiences. Grandchildren learned skills from older adults and gained a deeper understanding of them. We further discussed how CoRoot fosters engagement, knowledge exchange, and lasting connections in intergenerational relationships.
SESSION: Dark Manipulations
Session details: Dark Manipulations
“My Mother Told Me These Things are Always Fake” – Understanding Teenagers’ Experiences with Manipulative Designs
Manipulative and deceptive design practices are ubiquitous, impacting technology users in various ways across several domains. Certain groups are likely more susceptible to these impacts but have not received sufficient attention yet. In this paper, we seek to characterize one such understudied group, describing teenagers’ experience of manipulative design. We conducted semi-structured interviews with six teenagers between 15 and 17 years old, to understand their daily interactions with manipulative designs in three contexts: social networks, video games, and e-commerce. Using reflexive thematic analysis, our findings describe how risk is a shared experience for teenagers, and interrogate how teenagers’ personal and social context shape their experience of risk. We relate our findings to existing knowledge about how the general population is impacted by manipulative design practices and consider opportunities to further understand and support the experiences of teenagers and other vulnerable groups.
Opportunities, tensions, and challenges in computational approaches to addressing online harassment
Given the scale at which online harassment occurs, researchers and practitioners alike have turned to computationally driven approaches to address it. However, because harassment is highly contextual and personal, designing effective solutions to this problem can be extremely challenging. This paper examines how harassment-mitigation systems studied in human-computer interaction (HCI) consider victim-centered principles in their design. Through a scoping literature review and close reading of 17 papers, we contribute—(1) a characterization of how novel and existing systems consider victims’ identity characteristics, definitions of harassment, and preferred strategies for dealing with harassment; (2) challenges faced by the systems along these dimensions to surface limitations, gaps, and tensions; (3) practical recommendations for researchers, designers, and practitioners to overcome these challenges. In doing so, we offer potential new directions to positively design computational approaches to addressing online harassment with victim-centered principles in mind.
“Who Knows? Maybe it Really Works”: Analysing Users’ Perceptions of Health Misinformation on Social Media
Health misinformation, defined as health-oriented information that contradicts empirically supported scientific findings, has become a significant concern on social media platforms. In response, platforms have implemented diverse design solutions to block such misinformation or alert users about its potential inaccuracies. However, there is limited knowledge about users’ perceptions of this specific type of misinformation and the actions that are necessary from both the platforms and the users themselves to mitigate its proliferation. This paper explores social media users’ (n = 22) perceptions of health misinformation. On the basis of our data, we identify specific types of health misinformation and align them with user-suggested countermeasures. We point to the critical demands for anti-misinformation solutions for health topics, emphasizing the transparency of information sources, immediate presentation of information, and clarity. Building on these findings, we propose a series of design recommendations to aid the future development of solutions aimed at counteracting misinformation.
Labeling in the Dark: Exploring Content Creators’ and Consumers’ Experiences with Content Classification for Child Safety on YouTube
Protecting children’s online privacy is paramount. Online platforms seek to enhance child privacy protection by implementing new classification systems into their content moderation practices. One prominent example is YouTube’s “made for kids” (MFK) classification. However, traditional content moderation focuses on managing content rather than users’ privacy; little is known about how users experience these classification systems. Thematically analyzing online discussions about YouTube’s MFK classification system, we present a case study on content creators’ and consumers’ experiences. We found that creators and consumers perceived MFK classification as misaligned with their actual practices, creators encountered unexpected consequences of practicing labeling, and creators and consumers identified MFK classification’s intersections with other platform designs. Our findings shed light on an interwoven network of multiple classification systems that extends the original focus on child privacy to encompass broader child safety issues; these insights contribute to the design principles of child-centered safety within this intricate network.
Care Layering: Complicating Design Patterns
Over the past two decades, discussions of design patterns have turned from encouragement (what to do) toward discouragement (what to avoid). Termed dark, deceptive, or otherwise harmful, user experience (UX) patterns that serve to monetize engagement while reproducing and sedimenting structural inequities are prevalent, which calls for a shifting conversation around UX development and learning. This pictorial uses a visual case study of a childcare worker platform to help critically contextualize largely abstracted or universalizing UX patterns. Developing a form of critical documentation we call Care Layering, we show how approaching UX patterns as embodied and culturally-situated resources sheds light on both limitations and opportunities around gig work platform engagement. We end with a discussion of how Care Layering helps designers work towards greater accountability in UX design.
Participatory Design to Address Disclosure-Based Cyberbullying
Disclosure-based cyberbullying is defined as sharing personal information without consent, often with malicious intentions. This affects young adults and leads to long-term personal damage (i.e., career, reputation, and family). Current strategies on social media platforms fall short of addressing this problem. We engaged 20 young adults (18-34) through a participatory design approach in co-designing solutions that reflect their concerns, needs, and desires. Participants were divided into three groups based on their cyberbullying experience as victim, discloser, or attacker. Our participants designed 15 solutions for prevention, intervention, support, and awareness. These solutions reveal the need for privacy control, anonymity, and autonomy for victims, a restorative justice system, social media accountability, and the involvement of influencers and content creators in awareness and education. We provide implications for progressive and adaptive social media awareness campaigns, a compassionate justice system, and soft control of personal information ownership.
SESSION: GenAI for Designers
Session details: GenAI for Designers
”Clay to Play With”: Generative AI Tools in UX and Industrial Design Practice
Generative artificial intelligence (GAI) is transforming numerous professions, not least various fields intimately relying on creativity, such as design. To explore GAI’s adoption and appropriation in design, an interview-based study probed 10 specialists in user experience and industrial design, with varying tenure and GAI experience, for their adoption/application of GAI tools, reasons for not using them, problems with ownership and agency, speculations about the future of creative work, and GAI tools’ roles in design sensemaking. Insight from reflexive thematic analysis revealed wide variation in attitudes toward GAI tools – from threat-oriented negative appraisals to identification of empowerment opportunities – which depended on the sense of agency and perceived control. The paper examines this finding in light of the Coping Model of User Adaptation and discusses designers’ metacognitive skills as possible underpinnings for their attitudes. Avenues for further research are identified accordingly.
Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?
User Experience (UX) practitioners, like UX designers and researchers, have begun to adopt Generative Artificial Intelligence (GenAI) tools into their work practices. However, we lack an understanding of how UX practitioners, UX teams, and companies actually utilize GenAI and what challenges they face. We conducted interviews with 24 UX practitioners from multiple companies and countries, with varying roles and seniority. Our findings include: 1) There is a significant lack of GenAI company policies, with companies informally advising caution or leaving the responsibility to individual employees; 2) UX teams lack team-wide GenAI practices. UX practitioners typically use GenAI individually, favoring writing-based tasks, but note limitations for design-focused activities, like wireframing and prototyping; 3) UX practitioners call for better training on GenAI to enhance their abilities to generate effective prompts and evaluate output quality. Based on our findings, we provide recommendations for GenAI integration in the UX sector.
Exploring the Impact of Artificial Intelligence-Generated Content (AIGC) Tools on Social Dynamics in UX Collaboration
Artificial Intelligence-Generated Content (AIGC) tools have gradually been integrated into the daily workflow of UX practitioners. While existing research has explored the integration of AIGC tools in daily workflow, little is known about their impact on social dynamics within UX collaboration. We conducted four focus groups and eight semi-structured interviews with 26 UX practitioners to investigate how AIGC tools influence social dynamics in UX collaboration. Our findings indicated that AIGC tools not only mitigated conflicts but also introduced potential new conflicts. AIGC tools expanded the roles of UX practitioners and fostered a team culture characterized by exploring and discussing. Participants have higher expectations for AI-assisted design in user understanding and prototype evaluation, and team-motivated AI tools learning. Based on these findings, we discussed the benefits and concerns of conflict resolution through AIGC and the importance of teams in AI learning. Finally, we proposed several suggestions for future AI design research.
Development and translation of human-AI interaction models into working prototypes for clinical decision-making
In the standard interaction model of clinical decision support systems, the system makes a recommendation, and the clinician decides whether to act on it. However, this model can compromise the patient-centeredness of care and the level of clinician involvement. There is scope to develop alternative interaction models, but we need methods for exploring and comparing these to assess how they may impact clinical decision-making. Through collaborating with clinical, AI safety, and HCI experts, and patient representatives, we co-designed a number of alternative human-AI interaction models for clinical decision-making. We then translated these models into ‘Wizard of Oz’ prototypes, where we created clinical scenarios and designed user interfaces with different types of AI output. In this paper, we present alternative models of human-AI interaction and illustrate how we used a co-design approach to translate them into functional prototypes that can be tested with users to explore potential impacts on clinical decision-making.
Studying Self-Care with Generative AI Tools: Lessons for Design
The rise of generative AI presents new opportunities for the understanding and practice of self-care through its capability to generate varied content, including self-care suggestions via text and images, and engage in dialogue with users over time. However, there are also concerns about accuracy and trustworthiness of self-care advice provided via AI. This paper reports our findings from workshops, diaries, and interviews with five researchers and 24 participants to explore their experiences and use of generative AI for self-care. We analyze our findings to present a framework for the use of generative AI to support five types of self-care, – advice seeking, mentorship, resource creation, social simulation, and therapeutic self-expression – mapped across two dimensions – expertise and modality. We discuss how these practices shift the role of technologies for self-care from merely offering information to offering personalized advice and supporting creativity for reflection, and we offer suggestions for using the framework to investigate new self-care designs.
The CoExplorer Technology Probe: A Generative AI-Powered Adaptive Interface to Support Intentionality in Planning and Running Video Meetings
Effective meetings are effortful, but traditional videoconferencing systems offer little support for reducing this effort across the meeting lifecycle. Generative AI (GenAI) has the potential to radically redefine meetings by augmenting intentional meeting behaviors. CoExplorer, our novel adaptive meeting prototype, preemptively generates likely phases that meetings would undergo, tools that allow capturing attendees’ thoughts before the meeting, and for each phase, window layouts, and appropriate applications and files. Using CoExplorer as a technology probe in a guided walkthrough, we studied its potential in a sample of participants from a global technology company. Our findings suggest that GenAI has the potential to help meetings stay on track and reduce workload, although concerns were raised about users’ agency, trust, and possible disruption to traditional meeting norms. We discuss these concerns and their design implications for the development of GenAI meeting technology.
SESSION: Undisciplined, Queer and Inclusive Online Experiences
Session details: Undisciplined, Queer and Inclusive Online Experiences
Light Bridge: Improving social connectedness through ambient spatial interaction
While social relationships are important for human well-being, maintaining these relationships can be difficult, especially for individuals living apart from friends and loved ones. We present “Light Bridge”, an ambient spatial interaction concept designed to convey a sense of closeness and co-presence through changes in ambient lighting that reflect the partner’s location and activity, thus providing a subtle form of communication that blends seamlessly into daily routines. We tested our concept in a Wizard-of-Oz study with pairs of subjects in a separated locations (N=20) with a controlled protocol of activities to mimic a daily routine. Our study addressed four questions: social connectedness, privacy, distraction and joint action motivation. The results showed that ambient light can effectively convey a sense of physical closeness between separated individuals without causing distraction or irritation in terms of privacy. The study provides valuable insights regarding the potential of ambient spatial interaction in smart homes to improve the sense of social connectedness.
Let Information Flow Into Awareness: A Design Space for Human-Like Experiences to Promote Informal Communication for Hybrid Work
This paper addresses the challenges of promoting awareness among coworkers and teams for hybrid work, thereby promoting informal communication. Derived from the merits of onsite experiences, we intend to design human-like experiences to let information naturally flow into awareness. Through annotated portfolios and documented design space schemas, we filtered a design space comprising three aspects: social cues as information carriers, processing information with ambiguity, and ambient information delivery. All aspects are equipped with actionable options. To assess the effectiveness of our design space, we conducted structured workshops with design researchers. Through thematic analysis of workshop data, we informed our design space with one more aspect: built-in lightweight interactions. This design-space thinking process helped us gain an in-depth understanding of human-like experiences and insights into using our design space for ideation.
“This is the kind of experience I want to have”: Supporting the experiences of queer young men on social platforms through design
Queer young men (similar to others in the LGBTQ+ community) depend heavily on social platforms but their use can often be problematic. Their needs are often not adequately considered in the design of general platforms and they can be exposed to intra-community harms on LGBTQ+ specific platforms such as dating apps. To explore how social platform design could be improved to better support the needs of queer young men, we conducted a co-design study. We recruited 13 queer men working in technology design to generate new concepts for social platform features. We then refined these concepts and evaluated them in group sessions with end users, a different cohort of 15 queer young men. Here we present mockups of the concepts and findings from evaluations. Our findings show specific ways that providing more agency to social platform users could improve their experiences and we discuss implications for design.
Exploring Intervention Techniques to Alleviate Negative Emotions during Video Content Moderation Tasks as a Worker-centered Task Design
Videos are dynamic and multi-modal compared to other types of content, making automatic filtering difficult, which is why content moderators play a crucial role. However, video content moderators are exposed to more profound emotional labor because videos contain rich visual information, sometimes including even harmful content, such as violent or terrifying scenes. In this work, we explore the effect of six intervention techniques on alleviating negative emotions during video content moderation tasks. We conducted one online crowdsourcing experiment and two controlled user studies to find out that (i) interleaving with positive videos or (ii) cartoonization could significantly reduce negative emotions in the moderators. Participants reported that the advantages of these approaches are in helping reduce negative emotions at the time of moderation while existing approaches focus on post-task activities (e.g., relaxation, talking with others, or getting a hobby). We discuss the applicability of our findings to broader tasks, including improvement in intervention techniques.
Towards integrated learning experiences on social media: An exploration of #DayInTheLife videos for career exploration
Though social media platforms contain rich information and insights on professional life, encounters with this content are often fleeting and disconnected, raising questions about the extent social media content is valuable for career identity formation. This paper reports on a research through design study that explores the potential of social media for supporting integrated learning experiences, through investigating and prototyping experiences around the use of TikTok #DayInTheLife videos for career exploration. We conducted semi-structured interviews of 10 college students to understand the value of social media content for career exploration and the feasibility of integrating such content towards reflective learning experiences. A qualitative analysis revealed that #DayInTheLife videos offer firsthand insights into professions that facilitates aspects of career identity formation, and have the potential to prompt and motivate further exploration. However, they are also limited due their short-form, disconnected, entertainment-oriented nature, the distracting context in which they exist, and the potential lack of representation in recommended content. We also had the students participate in an experience prototype in which we used native social media interactions such as comments, mentions, and direct messages to integrate encounters of disparate posts towards holistic and reflective learning experiences. We found that integrating encounters can facilitate more intentional reflection, add interactivity, and provide a sense of agency. We also surfaced contextual risk factors and design factors for designing integrated learning experiences on social media. We build on our findings to introduce and discuss a concept we call SIMPLE apps (Social media Interactions Merged for Purposeful Learning Experiences) and to discuss broader design implications for better harnessing social media content towards purposeful integrated learning.
CounterSludge in Alcohol Purchasing on Online Grocery Shopping Platforms
Abstract: We investigate how deceptive patterns (sludge) within online grocery shopping can influence the purchase of alcohol through design intervention, and how to counter them. Previous research investigated online shoppers’ purchasing behaviors in sustainability and healthy eating. However, current research in alcohol is limited to modifying simulated platforms to aid in the increase of purchasing lower alcoholic beverages by altering product offerings. We conducted a heuristic evaluation on online shopping platforms highlighting the use of sludge, before developing five design intervention prototypes. We then iterated on these interventions through an online alcohol purchasing questionnaire (N=20) and two follow-up activities (N=11) (interview with design probes; product swap questionnaire), evaluating how the interventions could counter sludge. Our goal is to develop interventions that engage light to moderate drinkers in alcohol reduction. We found participants want a greater presence of alcohol units and product grading imagery in conjunction with neutral-toned health warnings.
SESSION: The Troubles of AI & Design
Session details: The Troubles of AI & Design
Do LLMs Meet the Needs of Software Tutorial Writers? Opportunities and Design Implications
Creating software tutorials involves developing accurate code examples and explanatory text that engages and informs the reader. Large Language Models (LLMs) demonstrate a strong capacity to generate both text and code, but their potential to assist tutorial writing is unknown. By interviewing and observing seven experienced writers using OpenAI playground as an exploration environment, we uncover design opportunities for leveraging LLMs in software tutorial writing. Our findings reveal background research, resource creation, and maintaining quality standards as critical areas where LLMs could significantly assist writers. We observe how tutorial writers generated tutorial content while exploring LLMs’ capabilities, formulating prompts, verifying LLM outputs, and reflecting on interaction goals and strategies. Our observation highlights that the unpredictability of LLM outputs and unintuitive interface design contributed to skepticism about LLM’s utility. Informed by these results, we contribute recommendations for designing LLM-based tutorial writing tools to mitigate usability challenges and harness LLMs’ full potential.
Dynamic Agent Affiliation: Who Should the AI Agent Work for in the Older Adult’s Care Network?
The population of older adults experiencing cognitive decline is growing faster than the number of workers who can care for them. Artificially intelligent (AI) agents could assist these older adults, keeping them in their homes longer. For this to happen, older adults must be willing to adopt and rely on agents. Would they trust an agent that might need to report their decline to others? We conducted a speed dating study exploring the impact of agent affiliation (i.e., who the agent should work for). Our healthy and declining participants reacted positively to the idea of agents supporting them. They particularly recognized how the agent would reduce the burden placed on their family caregivers. They viewed affiliation to be dynamic, shifting from the declining older adult and orienting more to their caregivers over the course of cognitive decline. They envisioned the agent modifying its decision-making process to be like their caregivers’.
Towards Feature Engineering with Human and AI’s Knowledge: Understanding Data Science Practitioners’ Perceptions in Human&AI-Assisted Feature Engineering Design
As AI technology continues to advance, the importance of human-AI collaboration becomes increasingly evident, with numerous studies exploring its potential in various fields. One vital field is data science, including feature engineering (FE), where both human ingenuity and AI capabilities play pivotal roles. Despite the existence of AI-generated recommendations for FE, there remains a limited understanding of how to effectively integrate and utilize humans’ and AI’s knowledge. To address this gap, we design a readily-usable prototype, human&AI-assisted FE in Jupyter notebooks. It harnesses the strengths of humans and AI to provide feature suggestions to users, seamlessly integrating these recommendations into practical workflows. Using the prototype as a research probe, we conducted an exploratory study to gain valuable insights into data science practitioners’ perceptions, usage patterns, and their potential needs when presented with feature suggestions from both humans and AI. Through qualitative analysis, we discovered that the “Creator” of the feature (i.e., AI or human) significantly influences users’ feature selection, and the semantic clarity of the suggested feature greatly impacts its adoption rate. Furthermore, our findings indicate that users perceive both differences and complementarity between features generated by humans and those generated by AI. Lastly, based on our study results, we derived a set of design recommendations for future human&AI FE design. Our findings show the collaborative potential between humans and AI in the field of FE.
Mediating Culture: Cultivating Socio-cultural Understanding of AI in Children through Participatory Design
The surge in access to and awareness of Generative Artificial Intelligence (GenAI) such as ChatGPT has sparked discussion over the necessary technological literacies and competencies needed to effectively engage with these systems. In this context, we explore AI as a tool that mediates cultural understanding and remediates human values – that are often influenced by biases and inequities. Using participatory design for learning with a group of 13 children (ages 8-13), we engaged in five co-design sessions featuring different modalities for socio-cultural approaches to AI literacy. We found that children were more aware of the cultural mediation aspect of AI when the content of the interaction aligned with their cultural background and context. This underscored the significance of aligning the representation of culture in these GenAI systems with people’s socio-cultural ecosystems in modern technological literacies. We conclude with design principles for a more critical and holistic approach to AI literacy.
Responding to Generative AI Technologies with Research-through-Design: The Ryelands AI Lab as an Exploratory Study
Generative AI technologies demand new practical and critical competencies, which call on design to respond to and foster these. We present an exploratory study guided by Research-through-Design, in which we partnered with a primary school to develop a constructionist curriculum centered on students interacting with a generative AI technology. We provide a detailed account of the design of and outputs from the curriculum and learning materials, finding centrally that the reflexive and prolonged ‘hands-on’ approach led to a co-development of students’ practical and critical competencies. From the study, we contribute guidance for designing constructionist approaches to generative AI technology education; further arguing to do so with ‘critical responsivity.’ We then discuss how HCI researchers may leverage constructionist strategies in designing interactions with generative AI technologies; and suggest that Research-through-Design can play an important role as a ‘rapid response methodology’ capable of reacting to fast-evolving, disruptive technologies such as generative AI.
SESSION: Harmful Design, Inclusivity and Telepresence
Session details: Harmful Design, Inclusivity and Telepresence
The Ecology of Harmful Design: Risk and Safety of Game Making on a Metaverse Platform
Metaverse platforms have been on the rise in recent years, offering three-dimensional (3D), immersive virtual worlds while encouraging user-generated content (UGC) in various forms. Roblox, a popular metaverse platform, enables its users to create a holistic virtual world (i.e., develop a 3D game) for other users to interact with. However, complex UGC is also challenging to moderate. Roblox has been notorious for its users’ harmful designs, such as Nazi or terrorist role-playing mechanisms. In this study, we explore how harmful design takes place on Roblox. Through a grounded theory analysis of the ‘r/Robloxgamedev’ subreddit, we conceptualize an ecological view of harmful design, foregrounding three interconnected circumstances, namely sociotechnical risks, socioeconomic precarities, and normative (in)sensitivities, which work together to condition and give rise to harmful designs and bring about unique governance challenges to metaverse platforms. We conclude by laying out implications for design moderation.
A Context-Aware Onboarding Agent for Metaverse Powered by Large Language Models
One common asset of metaverse is that users can freely explore places and actions without linear procedures. Thus, it is hard yet important to understand the divergent challenges each user faces when onboarding metaverse. Our formative study (N = 16) shows that first-time users ask questions about metaverse that concern 1) a short-term spatiotemporal context, regarding the user’s current location, recent conversation, and actions, and 2) a long-term exploration context regarding the user’s experience history. Based on the findings, we present PICAN, a Large Language Model-based pipeline that generates context-aware answers to users when onboarding metaverse. An ablation study (N = 20) reveals that PICAN’s usage of context made responses more useful and immersive than those generated without contexts. Furthermore, a user study (N = 21) shows that the use of long-term exploration context promotes users’ learning about the locations and activities within the virtual environment.
Beyond Text and Speech in Conversational Agents: Mapping the Design Space of Avatars
Conversational agents have gained widespread popularity due to their ability to simulate and sustain contextual conversations. Prior works predominantly focused on computational challenges. However, avatars — the representation of the agent — impact user interactions and perception of conversational agents’ trustworthiness and usefulness. Despite their importance, we lack a holistic understanding of conversational agent avatar design space. In this work, we address this gap by defining a categorization of 10 dimensions that is based on the analysis and iterative coding of 266 conversational agent papers from 160 venues spanning 2003 to the present. In addition, we built an interactive browser to facilitate exploration and interaction with these dimensions and their interrelationships. Our categorization lays the groundwork for researchers, designers, and practitioners to discern task-specific and contextual aspects of conversational agent avatar design. Our work fosters innovative ideas to facilitate new interactions with avatars by surfacing current patterns and highlighting open challenges.
Superarchitectural: Challenging the Architectural Design of the Metaverse
The metaverse is an immersive virtual realm that allows millions of users to interact with each other, both described by the social intricacies of the everyday and unrestrained by the physical limitations of reality. Yet despite the creative potential to disrupt how the built environment is represented, the metaverse tends to simulate the architectural conventions of physical reality. By visiting the virtual reality sites of 30 interiors, buildings, and plazas within popular metaverse platforms, we identified outliers in convention to curate a ‘Metaverse Design Catalogue’ of architectural features that were uncommon or unrealistic in physical reality. We thus challenged 21 architectural experts to use the catalogue as a catalyst to design an architecture that goes beyond convention to become ‘superarchitectural’. Based on their redesigned and reimagined floor plans and perspective drawings, we inform the design language of a superarchitectural metaverse through the dimensions of architectural diegesis, mutability, and asymmetry.
Disembodied, Asocial, and Unreal: How Users Reinterpret Designed Affordances of Social VR
Although Social Virtual Reality (SVR) affordances are designed to enable embodied social activities and interactions within virtual environments, the ways that users perceive and interpret these affordances can shape how SVR platforms are used and experienced. In this study, we examined the design and use of SVR affordances based on qualitative survey data from 100 SVR users. We observed that user practices diverge in important ways from intended designs, adding complexity to conventional interpretations of SVR platforms as embodied social environments. This research highlights dynamic user behaviour in which users interpret and reconfigure the affordances of SVR platforms, ranging from asocial use cases to actions that reflect the current limits of embodied communication. We contribute findings that may improve SVR design by revealing opportunities to foreground user needs and expectations, leveraging both the designed possibilities of SVR and the interpretations of those possibilities.
Re-envisioning Remote Meetings: Co-designing Inclusive and Empowering Videoconferencing with People Who Stutter
Videoconferencing (VC) has become a prominent and normalized mode of professional and personal communication, introducing universally experienced challenges such as reduced non-verbal cues and “Zoom Fatigue.” But People who stutter (PWS) encounter these obstacles with extra hurdles as existing VC technologies often rely on assumptions about speech patterns that don’t accommodate stuttering. Leveraging and driven by the unique insights and experiences of PWS on VC, we conducted a two-phase co-design study with PWS to explore and reflect on the design space for inclusive and empowering VC technologies from their perspectives. Our findings present a broad design space for tools that support PWS before, during, and after VC, focusing on aspects such as supporting self-disclosure, educating non-stuttering audiences, and promoting personal reflection for long-term self-growth. While many design ideas by our participants embody universal value to all VC users, some carry an activism approach that proactively disrupts existing communication flows and norms to redistribute the power between stuttering and non-stuttering speakers in VC meetings. This work contributes to a thorough analysis of the design space and empowering PWS to be drivers and designers of inclusive VC experiences.
SESSION: Education at Play
Session details: Education at Play
Tutorial mismatches: understanding the frictions due to interface differences when following software video tutorials
Video tutorials are the main medium to learn novel software skills. However, the User Interface (UI) presented in a video tutorial may differ from the learner’s UI because of customizations or differences in software versions. We investigate the frictions resulting from such differences on a learners’ ability to reproduce a task demonstrated in a video tutorial. Through a morphological analysis, we first identify 13 types of “interface differences” that differ in terms of availability, reachability and spatial location of features in the interface. To better assess the frictions resulting from each of these differences, we then conduct a laboratory study with 26 participants instructed to reproduce a vector graphics editing task. Our results highlight interesting UI comparison behaviors, and illustrate various approaches employed to visually locate features.
Visualizing the Unseen Design Work of Educators
Educators are often described as practitioners and can be subject to deficit-oriented characterizations that position their work as focused on the passive dissemination of knowledge. This pictorial argues that educators are designers, and their curation of learning environments and experiences constitutes an underappreciated and complex design practice. Further, the design work that educators engage in is significant and consequential as it can define or reimagine who participates and what is valued in educational spaces – playing an important role in creating more equitable educational outcomes. In this pictorial, we leverage photos captured and curated by educators of their learning environments in library makerspaces and youth-serving technology centers to make their unseen design work and impact visible. Beyond making educators’ expertise more visible, this pictorial also offers design considerations for designers of technologies, materials, and experiences that may be situated in educational environments.
Co-designing a knowledge management tool for educator communities of practice
Knowledge management involves finding, expanding, and using knowledge in an organisation to achieve goals. Its role is crucial in higher education to improve problem-solving, research, and teaching by acquiring, sharing, and applying knowledge. Higher education institutions can promote knowledge management through Communities of Practice, but doing so remains challenging due to cultural, organisational, and technological reasons. We present findings of the first step of co-design workshops with authentic higher education teaching teams that sought to understand (a) their practices as a community and any motivators and impediments to their community development; (b) how they perceived the tools they use for knowledge management; and (c) the kinds of tools they believed could help them better conduct knowledge management and develop as Communities of Practice. Our findings suggested four essential design requirements and informed our development of a new tool to support the knowledge management needs of higher education teaching teams.
Anther: Cross-Pollinating Communities of Practice via Video Tutorials
Communities of practice (CoPs) play a crucial role in cross-pollination and learning within various skill-based and craft domains. These communities often share common materials, concepts, and techniques across related practices. However, due to their insular nature, exchanging knowledge between CoPs has been challenging, leading to fragmented knowledge marked by differing vocabularies and contexts. To address this issue, we introduce Anther, a system designed to highlight shared concepts and semantic overlap between distinct CoPs. Anther projects concepts onto a 2-dimensional space, providing users with comprehensive, contextual, and conceptual views. We conducted a user study, demonstrating Anther’s effectiveness in aggregating and disseminating community-based knowledge, bridging gaps between CoPs, and supporting the cross-pollination of knowledge between CoPs. Further, we present interaction vignettes that illustrate how Anther can ease entry into new domains and aide in discovering new creative techniques. This work can benefit maker communities by fostering collaborative knowledge-building across diverse domains.
Understanding User Needs for Task Guidance Systems Through the Lens of Cooking
To design intuitive and effective context-aware task guidance systems, we must understand users’ thought processes and the obstacles they experience when they perform tasks. Though task guidance systems have proven beneficial in many domains for improving task performance and reducing user frustration, there is a lack of general guidelines and design principles for their development. Prior work has shown that recipe-based cooking is a strong medium for studying task planning and execution. In response, we conducted a contextual inquiry study in home kitchens, observing eight different participants’ cooking sessions. We used affinity diagramming of our notes and transcripts to identify common obstacles faced by participants and establish user needs in the areas of object interaction, safety, knowledge base, and task coordination. We discuss how these findings can inform the design of technology-driven solutions for task guidance systems beyond cooking.
RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices
Reading and repeatedly retelling a short story is a common and effective approach to learning the meanings and usages of target words. However, learners often struggle with comprehending, recalling, and retelling the story contexts of these target words. Inspired by the Cognitive Theory of Multimedia Learning, we propose a computational workflow to generate relevant images paired with stories. Based on the workflow, we work with learners and teachers to iteratively design an interactive vocabulary learning system named RetAssist. It can generate sentence-level images of a story to facilitate the understanding and recall of the target words in the story retelling practices. Our within-subjects study (N=24) shows that compared to a baseline system without generative images, RetAssist significantly improves learners’ fluency in expressing with target words. Participants also feel that RetAssist eases their learning workload and is more useful. We discuss insights into leveraging text-to-image generative models to support learning tasks.
SESSION: Playful Encounters and Game Experiences
Session details: Playful Encounters and Game Experiences
When Should I Lead or Follow: Understanding Initiative Levels in Human-AI Collaborative Gameplay
Dynamics in Human-AI interaction should lead to more satisfying and engaging collaboration. Key open questions are how to design such interactions and the role personal goals and expectations play. We developed three AI partners of varying initiative (leader, follower, shifting) in a collaborative game called Geometry Friends. We conducted a within-subjects experiment with 60 participants to assess personal AI partner preference and performance satisfaction as well as perceived warmth and competence of AI partners. Results show that AI partners following human initiative are perceived as warmer and more collaborative. However, some participants preferred AI leaders for their independence and speed, despite being seen as less friendly. This suggests that assigning a leadership role to the AI partner may be suitable for time-sensitive scenarios. We identify design factors for developing collaborative AI agents with varying levels of initiative to create more effective human-AI teams that consider context and individual preference.
Co-Designing Location-based Games for Broadband Data Collection
Crowdsourced data collection is a scalable approach to collecting mobile broadband performance data across space. However, existing platforms for crowdsourced mobile broadband measurements are not designed to engage workers over time or space, which can lead to spatial misrepresentation and stale data. With the insight that games and play offer naturally engaging frameworks for users, we held five iterative, participatory design sessions with 11 participants to co-design a catalog of 11 game concepts that could be used to create more spatially representative mobile broadband data sets. Importantly, we found that while games varied substantially with respect to theme, all used a few common game mechanics to incorporate mobile broadband data collection into play. This indicates that a designed prototype might focus on offering a customizable gaming structure that would allow communities and individuals to create thematic content that could overlay onto a set of common mechanics that could support more representative geospatial data collection.
Longitudinal Evaluation of Casual Puzzle Tablet Games by Older Adults
Despite growing interest in mobile games for older adults, there is limited exploration of older adults’ gaming behaviors, perceptions, and experiences as they engage with casual puzzle games over a period of time. To address this, we conducted a 9-month study with 20 older adults, examining training needs, in-situ experiences, and preferences. Participants were trained on tablet PCs and ten selected games. During the study, participants documented their experiences and attended technology workshops. Gaming behaviors were logged and analyzed using descriptive and inferential statistics, revealing patterns and statistically significant differences in play frequency and duration over the course of the study. Thematic analysis identified facilitators and barriers to engagement such as customization, co-play experiences, and health issues. Based on these findings, we recommend incorporating educational elements, enhancing user control, leveraging identity and nostalgia, supporting social interactions, designing for tangible interaction, and emphasizing the importance of learning aids. Future research should test the effectiveness of these recommendations in increasing older adults’ engagement with casual games.
Shared Bodily Fusion: Leveraging Inter-Body Electrical Muscle Stimulation for Social Play
Traditional games like “Tag” rely on shared control via inter-body interactions (IBIs) – touching, pushing, and pulling – that foster emotional and social connection. Digital games largely limit IBIs, with players using their bodies as input to control virtual avatars instead. Our “Shared Bodily Fusion” approach addresses this by fusing players’ bodies through a mediating computer, creating a shared input and output system. We demonstrate this approach with “Hidden Touch”, a game where a novel social electrical muscle stimulation system transforms touch (input) into muscle actuations (output), facilitating IBIs. Through a study (n=27), we identified three player experience themes. Informed by these findings and our design process, we mapped their trajectories across our three experiential spaces – threshold, tolerance, and precision – which collectively form our design framework. This framework facilitates the creation of future digital games where IBIs are intrinsic, ultimately promoting the many benefits of social play.
ICY Interfaces: Exploration of Ice’s Ephemeral Features for Digital Game User Experience
As we observe, fleeting interactions unfold fascinatingly as the ice melts into transparency, scattering light into beautiful patterns. In interaction design, such ephemeral phenomena inspire the integration of transient features into interfaces to create unique user experiences. These interfaces generally foster ambient interactions, yet their potential in digital games remains relatively underexplored. We introduce ICY Interfaces—MeltPress, FrostPad, and IceSquish—gaming interfaces that utilize ice’s playful ephemeral qualities to explore new dimensions in the digital game experience. Building on previous research, we conducted material exploration and a workshop with ice to investigate how its ephemerality benefits digital game interface design. Our findings from the playtest reveal how ice’s physical disappearance influences gameplay interactions, its melting process enhances sensory experiences, and its unpredictability introduces engaging challenges. These results highlight the playful potential of ice in a digital game environment and underscore its role within the context of ephemeral materials and interfaces.
Not All Those Who (Mind-)Wander Are Lost: Exploring Game-Unrelated Thoughts
Task-unrelated thoughts (TUTs), colloquially referred to as mind-wandering or daydreaming, are phenomena that can interfere with attention and focus, but are also associated with mental health, creativity, and learning. In digital games, it is unclear how players experience game-unrelated thoughts (GUTs), whether GUTs should be encouraged by game designers, or how it may impact player experience. We ran an initial study to confirm whether GUTs are common (50 of 100 participants reported experiencing them). We then collected 840 minutes of gameplay data from 12 participants playing games they: (1) found relaxing, (2) lost track of time in, and (3) spent most hours playing. Eye-tracking data and experience sampling were used to contextualize a phenomenological analysis of gameplay data. We identified four themes encompassing gameplay, GUTs, and gaze behaviour: these provide a foundation for future research and game design incorporating GUTs.
SESSION: Participation Otherwise
Session details: Participation Otherwise
User Participation through WhatsApp – Mobile Instant Messaging as a Medium for Distributed Co-design
User participation is often time and resource intensive for both the designers and participants. This is especially problematic in cases where designers and users are not co-located and have to travel to participatory meetings. Distributed co-design through remote participation technologies has been hoped to alleviate these issues. However, distributed co-design has its own issues, such as requirement of an access to suitable technology independently by the participants. In this paper we present three cases where distributed co-design has been mediated through WhatsAppTM, the currently most wide-spread mobile instant messaging app. Our lessons learned from these projects indicate that mobile instant messaging has potential in distributed co-design, and the continuous access to communication channels can enable detailed discussions between the designers and the participants. To summarise our findings, we present a framework to help future academics and practitioners to plan and implement (part of) their co-design process in mobile instant messaging services.
Black to the Future – The Power of Designing Afrofuturist Technology with Black Women, Femmes, and Non-Binary People
Black women, femmes, and non-binary people (BWFNBP) are historically excluded in tech spaces, resulting in an exclusion of their perspectives in technology innovation. To explore how this group imagines the future of technology, and how they see themselves within it, we conducted a series of speculative design workshops. We asked participants to design a future communication technology and address its benefits, harms, and strategies for mitigating those harms, all through an Afrofuturist lens. Our findings show that when BWFNBP utilize Afrofuturism as a design and speculation framework, they bring their whole selves to the futures they envision. Their past and present experiences not only inform the features of their technologies but also influence how they imagine the technology impacting their communities. Beyond a conversation about what technology could be, the participants in this study take it one step further and examine what it should be.
“\”But will they listen?\” Learnings from a design after design experiment in civics”
A growing research area within DIS and HCI is digital civics, where technologies are explored to create new interactions and relationships between public officials and citizens, but what happens to the relationship after a public digital service is implemented? In this paper we present one view on how we can create opportunities for regular citizens to take agency and contribute to the rigid process of public digital design. We argue that designers and researchers should focus on civic participation in design after design as an opportunity to increase democracy. We base our results on learnings from an engagement with a volunteer-based IT helpdesk at a public library in Denmark and three employees from the Danish Agency for Digital Government. In this paper we focus on a specific period of the engagement which consists of three workshops and backstage design activities.
Ode to Barelas: Supporting Youth Agency Cultural Expression, and Community Engagement Through an Interactive Mural
Community mural projects provide youth with the opportunity to learn new skills, have a voice, and deepen their connection to their communities. This pictorial reflects on a participatory community-based STEAM project in which a group of teenagers collaborated with a muralist (Author 2) and two interaction design researchers (Authors 1 and 5) to create an Interactive Mural—a traditional mural with embedded electronics. We analyzed students’ experience of agency, expression, and connection to community throughout the project. We discuss how large-scale community-centered projects that provide opportunities for student agency and expression are rich learning environments that can promote technological fluency.
Between Two Worlds: Analysing the Effects of Immersive and Non-Immersive Prototyping for Participatory Design
Participatory Design (PD) aims to create technological solutions that improve users’ lives and to foster users’ empowerment. While PD has received extensive attention regarding its processes and methods, less research exists on the effects of PD. With the increasing inclusion of Extended Reality (XR) into PD, this gap necessitates a critical analysis of how and to which end XR can be best used in PD activities to achieve the desired effects. Through the analysis of 16 XR prototyping workshops, we explore the impact of immersive and non-immersive XR prototyping on PD effects. Furthermore, we delineate differences between the immersive and non-immersive approaches. Our work contributes to a comprehensive understanding of the effects of XR prototyping for PD, rooted in theoretical and empirical insights. It further illustrates what each approach offers and presents design considerations to assist practitioners in successfully integrating XR prototyping into their design practice.
Baking An Institutional Doughnut: A Systemic Design Journey for Diverse Stakeholder Engagement
Doughnut Economics offers a contemporary compass for navigating the complexities of creating a safe and just space where humanity can flourish while respecting ecological boundaries. This pictorial reports on how the Doughnut Economics model can be applied as a tool for facilitating complex stakeholder engagement. We present a novel visual framework and facilitation method to enable systemic and values-led thinking in the context of establishing a new interdisciplinary academic institution. Using a participatory design process, 115 stakeholders from academic, research, and administrative backgrounds explored this model over two studio sessions, co-creating an institutional compass to navigate the socio-ethical challenges of their professional practices. We leverage the pictorial format to (i) demonstrate the application of the Doughnut Economics model as an effective visual framework for fostering structured dialogue to surface shared and contesting values and boundaries and (ii) facilitate complex stakeholder engagement through a systemic design journey that encourages emergent dialogue.
SESSION: Across Realities #1
Session details: Across Realities #1
The Jamais Vu Effect: Understanding the Fragile Illusion of Co-Presence in Mixed Reality
Collaboration in distributed mixed reality (MR) creates co-presence illusions by bringing a digital representation of the remote collaborator into the local user’s physical space. However, constraints in each physical environment, such as different spatial layouts and furniture, can hinder collaboration and momentarily break the illusion of co-location. Since the remote user’s physical space is invisible to the local user, the source of the interaction problem is often imperceptible, leading to familiar real-world interactions suddenly feeling unfamiliar—a concept we term the Jamais Vu Effect. In this paper, we conceptualise and demonstrate this effect through a five-part user study that elicits the user experience of the Jamais Vu Effect in MR. We contribute a shared vocabulary and understanding of spatial interface design challenges to help designers and researchers discuss and improve distributed collaboration in MR.
Navigating the Virtuality-Reality Clash: Reflection and Design Patterns for Industrial Mixed Reality Applications
Creating Mixed Reality applications poses distinct challenges for development and design. One of the challenges is designing Mixed Reality application-specific experiences in the wild. In this paper, we present a structured reflection approach to revisit projects from the past. In applying this structured reflection to the data collected during a nine-month industrial project, we unveiled the Virtuality-Reality Clash. To generate a sufficient data corpus, we structurally analysed git commits, tickets, emails, handwritten notes, and weekly snapshots of the 3D designs. The clash could be narrowed down in our data corpus to the situations in which we were fusing the real environment with the virtual content. Finally, we could find five design patterns for MR experience. With these patterns, we aim to help developers and designers of MR applications identify situations where Virtuality and Reality clash and propose approaches to address them.
Exploring the Design Space of Input Modalities for Working in Mixed Reality on Long-haul Flights
Flexible working, including in resource-constrained settings such as long-haul flights, poses considerable challenges. While research in mixed reality (MR) offers the potential for enhancing experiences of working on airplanes (WoA), what input modalities and interactions might be suitable for such workspaces is under-explored. To address this gap, we created four design probes and demonstrated their interactions within MR for WoA using a branching decision-based interactive video. 24 participants engaged in the think-aloud study with the video where they selected which probe to watch based on personal preference and order. The follow-up interview further solidified their general preferences for intuitive and friction-free inputs, revealed strong concerns about reliability, discreetness, and portability, and showed high enthusiasm for interactive video. Based on these findings, we contribute a context-driven design space of input modalities for WoA in MR, extending the understanding of design opportunities for inputs with MR in confined spaces.
MoodShaper: A Virtual Reality Experience to Support Managing Negative Emotions
Negative emotions such as sadness or anger are often seen as something to be avoided. However, recognising, processing and regulating challenging emotional experiences can facilitate personal growth and is essential for long-term well-being. To support people in regulating and reflecting on negative emotions, we designed MoodShaper — a VR experience where participants autonomously create a virtual environment combined with emotion regulation (ER) interventions. Our system included three different interventions designed based on interviews with psychotherapists. We evaluated MoodShaper in a mixed-method between-subject study with n = 60 participants. Participants experienced one of the three ER interventions, allowing them to manipulate visual representations of negative emotions through externalisation, seclusion, or appreciation. We found that MoodShaper significantly increased positive affect while decreasing difficulties in ER and negative affect. Our work demonstrates how VR can provide technology-mediated support to reflect on, engage with and manage negative emotions. We contribute insights for future VR systems which support ER for challenging situations.
Mind Mansion: Exploring Metaphorical Interactions to Engage with Negative Thoughts in Virtual Reality
Recurrent negative thoughts can significantly disrupt daily life and contribute to negative emotional states. Facing, confronting, and noticing such thoughts without support can be challenging. To provide a playful setting and leverage the technical maturation of Virtual Reality (VR), our Virtual Reality (VR) experience, Mind Mansion, places the user in an initially cluttered virtual apartment. Here we utilize established concepts from traditional therapy and metaphors identified in prior works to let users engage metaphorically with representations of thoughts, gradually sorting the space, fostering awareness of thoughts, and supporting mental self-care. The results of our user study (n = 30) reveal that Mind Mansion encourages the exploration of alternative perspectives, fosters acceptance, and potentially offers new coping mechanisms. Our findings suggest that this Virtual Reality (VR) intervention can reduce negative affect and improve overall emotional awareness.
How People Prompt Generative AI to Create Interactive VR Scenes
Generative AI tools can provide people with the ability to create virtual environments and scenes with natural language prompts. Yet, how people will formulate such prompts is unclear—particularly when they inhabit the environment that they are designing. For instance, it is likely that a person might say, “Put a chair here,” while pointing at a location. If such linguistic and embodied features are common to people’s prompts, we need to tune models to accommodate them. In this work, we present a Wizard of Oz elicitation study with 22 participants, where we studied people’s implicit expectations when verbally prompting such programming agents to create interactive VR scenes. Our findings show when people prompted the agent, they had several implicit expectations of these agents: (1) they should have an embodied knowledge of the environment; (2) they should understand embodied prompts by users; (3) they should recall previous states of the scene and the conversation, and that (4) they should have a commonsense understanding of objects in the scene. Further, we found that participants prompted differently when they were prompting in situ (i.e. within the VR environment) versus ex situ (i.e. viewing the VR environment from the outside). To explore how these lessons could be applied, we designed and built Ostaad, a conversational programming agent that allows non-programmers to design interactive VR experiences that they inhabit. Based on these explorations, we outline new opportunities and challenges for conversational programming agents that create VR environments.
SESSION: Collaborative, Material and Sensitive Data
Session details: Collaborative, Material and Sensitive Data
VISMOCK: A Programmable Smocking Technique for Creating Interactive Data Physicalization
Data physicalization is a research area that explores representing data attributes through manipulating the geometric and physical properties of tangible objects. We introduce VISMOCK, a data physicalization approach that leverages a fabric manipulation technique called “smocking”. VISMOCK supports the creation of interactive and dynamic data physi-calizations by extending the smocking technique with programmable components such as thermochromic pigments and shape memory alloys. Using a research-through-design methodology, we develop an initial design space for VISMOCK that shows how data can be represented using visual and tactile variables, as well as the affordances of VISMOCK. We demonstrate the generative power of our design space through four exemplars, created using VISMOCK. We use these exemplars to discuss the advantages and limitations of VISMOCK as a tool for data physicalization.
A Retrospective Autoethnography Documenting Dance Learning Through Data Physicalisations
We present a retrospective autoethnography grounded in data-driven design. The first author collected her movement data and subjective experience of learning the dance repertoire of modern dance pioneer Isadora Duncan, which together were encoded into the design of a set of plaster artefacts physicalising her embodied dance learning progression. The artefacts reflect the first author’s bodily transformation, mirroring her transition from discomfort to ease, and changes in her expressive capabilities. Our method offers an alternative to documentation of embodied learning through design. Throughout our design process we leverage on the movement data, the field notes and the first author’s memory of her journey, all of which constitute entangled and complementary input into her experience of dance learning. We show that the data physicalisations provided a gateway into the intangible experience and allowed for a deep and reflexive understanding of our dataset.
Art Digital Jewellery as Atmospheres: An autobiographical RtD exploration into IoT for poetic contexts
In a process of Research-through-Design two art digital jewellery pieces were created on the same bespoke IoT device that connects a person with the sea: a hand-held piece (SeaVessel) and a necklace (Thalassa). Drawing on the notion of atmosphere, the design development (2020-21) and the first author’s lived experience with the pieces (2021-2022) are illustrated and narrated. We share insights into how the pieces augmented intimacy with a significant place and offer a space for self-reflection for the first author to explore their connection with the sea and ultimately the self. We offer insights gained from working with IoT, live data and online servers in such a poetic context. We contribute to experiential approaches to designing reflective and curious ways of interacting with digital technology with the notion of atmosphere as a concept for aesthetics for interaction design and an understanding of data as poetic data.
Participation in Data Donation: Co-Creative, Collaborative, and Contributory Engagements with Athletes and their Intimate Data
Data donation is an emerging practice enabling personal data collection for research. While it offers opportunities to access new insights into people’s behavior and experiences through their digital-trace data, the role of individuals – as research participants – is limited in most data donation projects. They primarily contribute with data, limiting the perspectives included and accounted for around critical research-design decisions. In this paper, we explore the opportunity to embed data donation in research processes that are not only contributory but collaborative and co-created. To do so, we present a participatory data donation case study focused on athletes’ perceptions of the impact of their menstrual cycle on their sports performance through their physical activity data. Based on the data donation experiences of 20 athletes, our paper provides insights into people’s preferences and expectations in participatory data donation processes and discusses considerations for supporting various degrees of participation in future data donation research.
Changing Perspective on Data in Designing for Active Environments
Smart solutions provide increasing quality and availability of data. This brings new challenges for designers as it offers novel design opportunities and interlaces disciplines. At the same time, physical inactivity is a big societal challenge and dedicated urban planning and design can contribute to more active lifestyles. In this paper, we investigate how user-generated big data can support designers in shaping more activity-friendly and adaptive environments, addressing both timely challenges. Bridging the fields of HCI and urbanism, we introduce two data lenses. The individual lens primarily builds on empathic design skills and calls for a highly personal approach. The collective lens emphasizes systematic and holistic design skills, focusing on creating overview and surfacing collective interests. Through exploratory data visualizations, using a large dataset from a run-tracking smartphone application combined with public data sources, and a workshop, we investigate how these lenses can yield meaningful insights. We discuss the value of these lenses to the urban design and HCI communities and address the challenges and opportunities that arise at the cross-section of these perspectives.
Sensitive Data Donation: A Feminist Reframing of Data Practices for Intimate Research Contexts
Data donation is an emerging practice for collecting personal data. However, recent data donation approaches are insufficient in intimate research contexts as they perceive data as neutral and objective and do not consider the contexts where data is generated and shaped nor offer choices beyond whether to disclose data. In this paper, we investigate how Data Feminism can inform an alternative form of data donation and propose the Sensitive Data Donation (sDD) method. It recognizes the sensitive nature of data and assumes the importance of situating and contextualizing it through balanced participation from donors, either as contributors, collaborators, or co-creators. To develop the method, we conduct a scoping literature review where we conceptualize data donation theories and practices. These serve as a base to critique recent approaches and propose an alternative: sDD. It comprises five principles integrated into a five-phase approach. We conclude by discussing its limitations and future challenges.
SESSION: Plurality Coffee: My Robot and Me
Session details: Plurality Coffee: My Robot and Me
Olfactory Puppetry: Merging Olfaction and Puppetry to Explore the Future of Olfactory Social Robots
While olfaction is yet an uncommon sense employed in interactions between humans and social robots, this pictorial questions, “What if robots emit smell?” As olfaction is closely linked to human emotion, it has a high potential to enrich the affective qualities of human-robot interaction (HRI). To explore possible scenarios and usages of olfactory social robots, we employ a novel method, Olfactory Puppetry—puppetry that integrates olfaction, with two approaches. Through our experimental journey, we understand how people envision and what they expect of olfactory robots, as well as fostering discussions on what new interactions could be made possible. We believe such discussions would trigger creative ideas and suggest design directions, serving as a meaningful starting point for designing olfactory HRIs.
“This really lets us see the entire world:” Designing a conversational telepresence robot for homebound older adults
In this paper, we explore the design and use of conversational telepresence robots to help homebound older adults interact with the external world. An initial needfinding study (N=8) using video vignettes revealed older adults’ experiential needs for robot-mediated remote experiences such as exploration, reminiscence and social participation. We then designed a prototype system to support these goals and conducted a technology probe study (N=11) to garner a deeper understanding of user preferences for remote experiences. The study revealed user interactive patterns in each desired experience, highlighting the need of robot guidance, social engagements with the robot and the remote bystanders. Our work identifies a novel design space where conversational telepresence robots can be used to foster meaningful interactions in the remote physical environment. We offer design insights into the robot’s proactive role in providing guidance and using dialogue to create personalized, contextualized and meaningful experiences.
Understanding On-the-Fly End-User Robot Programming
Novel end-user programming (EUP) tools enable on-the-fly (i.e., spontaneous, easy, and rapid) creation of interactions with robotic systems. These tools are expected to empower users in determining system behavior, although very little is understood about how end users perceive, experience, and use these systems. In this paper, we seek to address this gap by investigating end-user experience with on-the-fly robot EUP. We trained 21 end users to use an existing on-the-fly EUP tool, asked them to create robot interactions for four scenarios, and assessed their overall experience. Our findings provide insight into how these systems should be designed to better support end-user experience with on-the-fly EUP, focusing on user interaction with an automatic program synthesizer that resolves imprecise user input, the use of multimodal inputs to express user intent, and the general process of programming a robot.
Designing Plant-Driven Actuators for Robots to Grow, Age, and Decay
Designing plant-driven actuators presents an opportunity to create new types of devices that grow, age, and decay, such as robots that embody these qualities in their physical structure. Plant-robot hybrids that grow and decay incorporate unpredictable and gradual transformations inherent across living organisms and suggest an alternative to the design principles of immediacy, responsiveness, control, accuracy, and durability commonly found in robotic design. To explore this, we present a design space of primitives for plant-driven robotic actuators. Proof-of-concept prototypes illustrate how concepts like slow change, slow movement, decay, and destruction can be incorporated into robotic forms. We describe the design considerations required for building plant-driven actuators for robots, including experimental findings regarding the mechanical properties of plant forces. Finally, we speculate on the potential benefits of plant-robot hybrids to interactive domains such as robotics.
“Push-That-There”: Tabletop Multi-robot Object Manipulation via Multimodal ‘Object-level Instruction’
Encouraging Bystander Assistance for Urban Robots: Introducing Playful Robot Help-Seeking as a Strategy
Robots in urban environments will inevitably encounter situations beyond their capabilities (e.g., delivery robots unable to press traffic light buttons), necessitating bystander assistance. These spontaneous collaborations possess challenges distinct from traditional human-robot collaboration, requiring design investigation and tailored interaction strategies. This study investigates playful help-seeking as a strategy to encourage such bystander assistance. We compared our designed playful help-seeking concepts against two existing robot help-seeking strategies: verbal speech and emotional expression. To assess these strategies and their impact on bystanders’ experience and attitudes towards urban robots, we conducted a virtual reality evaluation study with 24 participants. Playful help-seeking enhanced people’s willingness to help robots, a tendency more pronounced in scenarios requiring greater physical effort. Verbal help-seeking was perceived less polite, raising stronger discomfort assessments. Emotional expression help-seeking elicited empathy while leading to lower cognitive trust. The triangulation of quantitative and qualitative results highlights considerations for robot help-seeking from bystanders.
SESSION: Dis/Embodied and Playful Explorations
Session details: Dis/Embodied and Playful Explorations
Smell Above All: Envisioning Smell-Centred Future Worlds
Take a deep breath; what do you smell? While sight and hearing dominate our perceptions, the sense of smell is often overlooked, even undervalued. However, the importance of our sense of smell goes beyond detecting odours. It shapes our emotions, memories, behaviour, and quality of life. Recent advances in olfactory interfaces have sparked discussions about the future of smell in human-computer interaction. While efforts mainly focus on the realization of new olfactory interfaces, here we collectively explore alternative worlds centred around the sense of smell. First, we conducted a design futuring workshop involving individuals with varying smell capabilities and expertise to envision and discuss smell-centred futures. Then, through iterative reflection, we arrived at three smell worlds presented as narratives and visuals. From this conceptual work, we offer new perspectives and generative possibilities for design research that prioritizes our noses.
Body Sensations as Design Material: An Approach to Design Sensory Technology for Altering Body Perception
Sensory technologies alter how we perceive our body, which can have profound implications for multiple domains. Prior work has contributed a myriad of artefacts and evaluative studies, but we still lack design knowledge to design meaningful body perception alterations facilitated by sensory technologies. To address this gap, we draw from soma design to propose a methodological approach centered on body sensations as design material. We articulate our approach based on a project on co-designing wearables to alter body perception together with professional dancers. Our approach involves engaging participants in articulating and sharing body sensations to others, and exploring somatically sensory stimuli to co-design concepts for future technologies. We contribute experiential facets of body sensations, movement and sensory potentials to alter sensations, methods and design strategies, and a collection of ideas. Our work can be relevant to design communities interested in sensory technologies, perceptual alterations and body sensations as material.
Embodied fitting for ultra-personalized workwear – opportunities and perceived risks of embodied vs. digital tools
There is a rise of digitally fabricated products based on users’ data, known as ultra-personalization. In earlier works, tools that allow future wearers to express their needs and wishes are scarce. We took a Research through Design approach to create two tools for future wearers to capture their needs and wishes for fit in the context of workwear: a digital and an embodied tool. Five wearers of workwear have tried the two tools and selected their preferred one. Ultra-personalized lab coats were then produced for them. Participants were interviewed about their perceptions of the tools. We present opportunities and perceived risks associated with both tools, such as issues concerning the accuracy of measurement, profession-specific movements, and effort. The embodied tool was met with a high appreciation while the digital tool has more potential for scalability. This work suggests that qualities of both tools can be reconciled.
Grand challenges in CyclingHCI
Cycling Human-Computer Interaction (CyclingHCI) refers to the study and design of user interfaces and interactions between bicycles and riders in the context of cycling-related experiences. To date, however, there has yet to be a structured agenda for CyclingHCI to clarify the immediate challenges researchers should address next and facilitate the advancement of the field. We, three CyclingHCI researchers who collectively designed, developed, evaluated 18 CyclingHCI projects, reflected on our experiences to derive 10 grand challenges that we articulate with design opportunities and considerations grouped into: (1) Pushing the technological boundaries for cycling, (2) Understanding and protecting cyclists, and (3) Spatially situated cycling interaction. Our findings provide practical implications for research and practice in CyclingHCI, with which we aim to enrich the cycling experience through the safe integration of technology.
The Undertable: A Design Remake of the Mediated Body
Tables are a ubiquitous piece of furniture, a familiar sight in most environments from intimate to public. The dimensions of social interplay surrounding every single table are profoundly complex. In our project, we lift the importance of the neglected space under the table through the playful development of a tangible prototype. We approached this by a design remake of the Mediated Body: a wearable prototype encouraging touch between strangers using the conductivity of the skin. Instead, we leverage the familiarity of tables as a means to encourage playful explorations of bare-skin touch. We report in visual and textual form on the emerging design knowledge throughout our design process, including first-person narratives by the designers. We contribute with (1) a series of counterfactual table artifacts inspired by the Mediated Body; (2) a sequence of participant studies analysed through reflexive thematic analysis and summarised into the notion of “an odd invitation” as a new lens for homo explorens; and (3) an appeal to the importance of design remakes for research-through-design.
Embodied Mediation in Group Ideation – A Gestural Robot Can Facilitate Consensus-Building
This paper explores how a gesture-based robot influences human-human interaction in group ideation. The robot was mounted on a whiteboard and responded with six different gestures (e.g., nodding, following speakers with gaze) to specific situations. We coded the participants’ interactions from videos and gathered their experience through post-session interviews. The most frequently invoked robot behavior was following the speaker with gaze. As a result, participants felt socio-emotionally supported and responded by moving the ideation ahead (individual level) and consensus-building (group level). In fact, the groups with the robot showed more consensus-building than the two reference groups without the robot. Participants had different views on the role of the robot in the group, such as active outsider, supportive group member, or assistant. The latter tried to use the robot as decision-support. All in all, to include a robot to mediate human groups seems a promising future application domain.
SESSION: Designs for Listening
Session details: Designs for Listening
“Tuning in and listening to the current”: Understanding Remote Ritual Practice in Sufi Communities
Design research and HCI increasingly explore techno-spirituality. We investigated Sufi practices of group zikr, a ritual practice of remembrance of God. We focus on zikr groups that offer online or hybrid participation. We conducted a qualitative study using interviews with practitioners and collaborative autoethnography as researchers/practitioners. Our findings surface themes of (i) shared spiritual energy, (ii) sensory experiences’ role in spiritual energy, (iii) impact of technological mediation on sensory and spiritual experiences, and the (iv) importance of community. Our discussion contributes design considerations for techno-spirituality around (1) attunement, (2) practical audiovisual suggestions, and (3) ‘sensational forms’. Overall, we offer detailed experiential accounts of entanglements of sensory perception, spirituality, and attunement, and present generative design reflections suggesting avenues of further design research in supporting religious, faith-based, and/or spiritual practices in HCI.
Tuning In to Intangibility : Reflections from My First 3 Years of Theremin Learning
This paper presents an autoethnography of 3 years of learning to play the theremin, an instrument lacking tangible feedback. While the theremin is typically invoked in HCI to emphasize the importance of the tactile modality, we interrogate how I, the player, attained musical proficiency without touch.
Through a thematic analysis of 235 journal entries, our study distills my strategies for navigating the instrument as well as my personal transformations along the way. We discover that without touch, accurate and musical playing on the theremin relies on continuous auditory feedback, proprioception, and imaginative processes. We discuss challenges and opportunities for embodied and tangible interaction in light of these findings.
Open Sonifications: A Manifesto for Many Ecologies of Data and Sound
Open Sonifications is a manifesto for a more inclusive approach to data sonification. The practice of turning data into sound is dominated by approaches that prioritise analysis and algorithmic sound synthesis. This pictorial expands the boundaries of what is considered sonification by challenging existing conventions around what we can do with sonification, who gets to use sonification, why we sonify, what counts as valid data or valid sound, and the tools used to make sonifications. Alongside conventional sonification values such as insight and accuracy, we also value raw energy, expressiveness, polyphony, participation, decentralisation, performance, accessibility, approachability, community, and plurality of perspectives. The approaches we describe do not seek to replace established methods, but to enrich them. We contribute a set of values, examples and instructions for joining this movement, and reflections on how people working beyond sonification might benefit from these ideas.
Remote Rhythms: Audience-informed insights for designing remote music performances
This paper examines the design of technology for remote music performances, from the perspective of their audiences. In this process, we involved a total of 104 participants across the different stages of our project. Initially, we collected qualitative data from remote audiences using several methods, including surveys, interviews, and observations. Through the thematic analysis of this data, we identified four design dimensions consisting of 17 key elements that illustrate what audiences value in remote music spaces. We applied these insights in a participatory design workshop with diverse stakeholders, contributing to the development of speculative design ideas in this field. The paper concludes by presenting key design insights for future technology advancements in remote music performances. The research contributes to the evolving design space of remote music performances, offering valuable perspectives for researchers, designers, and industry stakeholders.
Sonic Entanglements with Electromyography: Between Bodies, Signals, and Representations
This paper investigates sound and music interactions arising from the use of electromyography (EMG) to instrumentalise signals from muscle exertion of the human body. We situate EMG within a family of embodied interaction modalities, where it occupies a middle ground, considered as a “signal from the inside” compared with external observations of the body (e.g., motion capture), but also seen as more volitional than neurological states recorded by brain electroencephalogram (EEG). To understand the messiness of gestural interaction afforded by EMG, we revisit the phenomenological turn in HCI, reading Paul Dourish’s work on the transparency of “ready-to-hand” technologies against the grain of recent posthumanist theories, which offer a performative interpretation of musical entanglements between bodies, signals, and representations. We take music performance as a use case, reporting on the opportunities and constraints posed by EMG in workshop-based studies of vocal, instrumental, and electronic practices. We observe that across our diverse range of musical subjects, they consistently challenged notions of EMG as a transparent tool that directly registered the state of the body, reporting instead that it took on “present-at-hand” qualities, defamiliarising the performer’s own sense of themselves and reconfiguring their embodied practice.
All Is Heard: Reimagining The Sounds of Home With Care
As interactive sounds become embedded in our homes, we turn to the lens of care for exploring our audible relationships with home’s nonhuman things, thereby revealing the potential of reimagining with care the design of domestic sound interactions. This paper presents a participatory study on Japanese dwellers’ experiences with domestic sounds and their convergence with care for other species, objects, climatic elements, and machines. Our findings reveal three thematic aural engagements with care: touching sounds, marginal maintenance, and beyond-human attunements. These themes inspired the speculation of three resounding objects as a creative device for critically contemplating the opportunities and troubles presented by care ethics for designing sounds in more-than-human homes. We conclude by challenging with care conceptions of sound, knowledge, and agency, toward audible futures where a multiplicity of voices is heard.
SESSION: Creative & Aesthetic Design Expression
Session details: Creative & Aesthetic Design Expression
Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools
This essay proposes and explores the concept of Collage for the design of AI writing tools, transferred from avant-garde literature with four facets: 1) fragmenting text in writing interfaces, 2) juxtaposing voices (content vs command), 3) integrating material from multiple sources (e.g. text suggestions), and 4) shifting from manual writing to editorial and compositional decision-making, such as selecting and arranging snippets. The essay then employs Collage as an analytical lens to analyse the user interface design of recent AI writing tools, and as a constructive lens to inspire new design directions. Finally, a critical perspective relates the concerns that writers historically expressed through literary collage to AI writing tools. In a broad view, this essay explores how literary concepts can help advance design theory around AI writing tools. It encourages creators of future writing tools to engage not only with new technological possibilities, but also with past writing innovations.
Exploring Aesthetic Qualities of Deep Generative Models through Technological (Art) Mediation
Deep Generative Models (DGM) have had a great impact both on visual art and broader visual culture. In this research-through-design project we investigate the use of a DGM for helping museum visitors explore the aesthetics of Edvard Munch’s art. We designed and built an interactive drawing table that allows a user to explore a StyleGAN model trained on sketches by Edvard Munch. The paper makes two novel contributions: 1. It presents a system that allows users to interact with a DGM by drawing on paper (rather than the typical text prompts used by most current systems). 2. We demonstrate how this mode and quality of interaction establish a unique perspective on Munch’s drawings as a practice. Through qualitative evaluation, we discuss how this setup led users towards a specific hermeneutic drawing strategy that enables building competency with the model and by proxy the data it is trained on. We suggest that the resulting interaction may contribute to an “education of attention” helping museum visitors to become attentive to certain visual qualities in Munch’s drawing practice. Finally, we discuss how the concepts of technological mediation and relationality are useful for designing how the output of a DGM is understood by its users.
Text-to-Image AI as a Catalyst for Semantic Convergence in Creative Collaborations
Design ideation requires both creative and divergent thinking as well as collaboration and exchange to bring forth new insights and design possibilities. In this paper, we investigate such a collaboration with the use of emerging generative AI tools, and we explore how they may help in achieving agreement and semantic convergence between designers. We conducted workshops using Text-to-Image AI with design students and found that in addition to known advantages of brainstorming, the text prompts required by the AI system engaged students in verbal articulations that aided their design process. In collaborative contexts, the generated images shifted the centre of attention for team members to reach convergence through sharing their interpretations. We conclude that text-to-image AI can be beneficial for students’ design processes as a catalyst for brainstorming rather than a tool for generating design imagery and presentations.
Expanding the Design Space of Vision-based Interactive Systems for Group Dance Practice
Group dance, a sub-genre characterized by intricate motions made by a cohort of performers in tight synchronization, has a longstanding and culturally significant history and, in modern forms such as cheerleading, a broad base of current adherents. However, despite its popularity, learning group dance routines remains challenging. Based on the prior success of interactive systems to support individual dance learning, this paper argues that group dance settings are fertile ground for augmentation by interactive aids. To better understand these design opportunities, this paper presents a sequence of user-centered studies of and with amateur cheerleading troupes, spanning from the formative (interviews, observations) through the generative (an ideation workshop) to concept validation (technology probes and speed dating). The outcomes are a nuanced understanding of the lived practice of group dance learning, a set of interactive concepts to support those practices, and design directions derived from validating the proposed concepts. Through this empirical work, we expand the design space of interactive dance practice systems from the established context of single-user practice (primarily focused on gesture recognition) to a multi-user, group-based scenario focused on feedback and communication.
Exploring Optimal Combinations: The Impact of Sequential Multimodal Inspirational Stimuli in Design Concepts on Creativity
The continuous advancements in Generative Artificial Intelligence (GenAI) enable designers to efficiently obtain vast amounts of inspirational materials during the design conception phase. Various modalities of stimuli can impact designers’ creativity. However, the relationship between the sequence of different modalities (Text, Images, and Virtual Models) and creativity remains to be explored. In this study, we investigate the effects of 6 sequential combinations of 3 modalities on designers’ creativity. Designers (N=36) used the creative stimulation platform InspireMe, accessing inspirational stimuli in various sequences to execute their design tasks. We analyzed the impact on creativity through assessments of work innovation and user interviews. The findings indicate that the sequence of material combinations significantly influences designers’ creativity, with varying impacts on the different sub-attributes of creativity. Drawing on experimental findings and qualitative research, this paper introduces a creativity demand-matching inspirational stimulation interaction strategy—the CDI strategy—that assists designers in developing more targeted design solutions. Furthermore, we propose the 3C Framework, a design strategy for creativity stimulation platforms aimed at designers in the intelligent era. This framework aims to guide the development of creative stimulation platforms for designers in the intelligent era.
Smiles Summon the Warmth of Spring: A Design Framework for Thermal-Affective Interaction based in Chinese Cí Poetry
Thermal-affective experience is a growing topic in Human-Computer Interaction. However, research linking thermal and affective experience in technology use has not moved beyond attempts to establish broad, sweeping associations, such as between warmth and positive affect. One of the obstacles to progression is the need for frameworks and vocabularies that describe and conceptualise the richness of thermal perception and affective experience. To help conceptualise associations between thermal perception and affective experience we turn to Cí poetry, a form of classical Chinese literature rich with evocative descriptions of embodied, environmentally situated, first-person experience. We conducted a lexical analysis to identify thermal-affective associations, and propose a design framework addressing thermal design. We demonstrate the value of this framework via analysis of existing thermal design exemplars.
SESSION: Art, Dissensus and other Sensibilities
Session details: Art, Dissensus and other Sensibilities
Seeing Art Differently: Design Considerations to Improve Visual Art Engagement for People with Low Vision
Most people with a visual impairment retain some residual vision, yet accessibility strategies and research for visual art focus on non-visual aids and strategies. Little is known about how image enhancements may benefit people with low vision in art settings. Our study explored the challenges of people with visual impairments in museums and art galleries and the potential of visual enhancements to improve accessibility. Through online focus groups with 18 participants, we uncovered multifaceted visual challenges. Participants emphasised the role of non-visual senses to reduce visual ambiguities. Subsequently, we presented participants with the concept of image enhancements on Head-Mounted Displays (HMDs) as a potential approach to enhance visual accessibility. Participants expressed nuanced perspectives, advocating for this technology to improve visual art. The findings provide valuable insights for enhancing the visual experience of people with low vision in museums and galleries through design.
“Being Eroded, Piece by Piece”: Enhancing Engagement and Storytelling in Cultural Heritage Dissemination by Exhibiting GenAI Co-Creation Artifacts
Cultural Heritage is not just about tangible artifacts; it also includes intangible elements such as personal memories, community ties, and envisioned futures. Traditional museums and archives often emphasize physical items like architectural pieces and photos, while overlooking people’s personal and emotional connections to cultural heritage. To illustrate the personal connections people have with cultural heritage sites, we designed an exhibition that displayed images created by participants, which represent their perspectives and future visions of cultural heritage sites. The exhibition’s images, generated through GenAI, helped participants narratively describe cultural heritage locations, allowing them to express their visions of future threats like over-tourism and climate change on these sites. Contrary to constraints, co-creating with Generative AI associates participants with personal memories of cultural heritage, stimulating personal narratives and promoting deep reflection on cultural heritage preservation. The dissemination strategies we designed illustrate the use of GenAI to empower the expression of matters of cultural value beyond the physical.
Designing for Dissensus: Socially Engaged Art to access experience and support participation.
For social justice-oriented design, pluralistic participatory practices are recognised as important. This paper explores Socially Engaged Art (SEA) as one such practice, which uses dialogical inquiry to address social justice issues. We describe a case study of SEA-informed digital storytelling workshops in a rurban community in Ireland. The workshops aimed to support diverse adolescent participants in identifying and critiquing place-based social issues with a view to imagining alternative and inclusive futures. SEA was instrumental in uncovering contradictory and hidden experiences of community life. We propose incorporating SEA into design processes to: i) challenge assumptions around identity and belonging; ii) recognise diverse experiences; and iii) breakdown social-engagement barriers. As we ask questions on what design is for, who should design, and why design matters, SEA can centre marginalised experiences, support polyvocal participation and help develop participant agency to work towards more equitable futures in rapidly changing environments.
Negotiating Conceptual and Practical Frictions in Making the Capra Short Film: Extending a Research through Design Artifact with Video
As the practice of hiking becomes captured through personal data, it is timely to consider how technology might support noticing and connecting to nature as well as one’s self over time. Capra is a system we designed that brings together the capture, storage, and exploration of personal hiking data with an emphasis on longer-term use. In this pictorial, we unpack our process of making a short film that aims to communicate the workings and experience of Capra to a broader audience. We encountered frictions in mobilizing key theoretical concepts framing Capra as a Research through Design (RtD) artifact in the making of our film. We reflect on tactics for working through such frictions, how they can support future work, and how the filmmaking process can offer a valuable approach for distributing RtD artifacts to broader audiences.
Enhancing Reentry Support Programs Through Digital Literacy Integration
Challenges faced by formerly incarcerated individuals in the United States raise questions about our society’s ability to truly provide second chances. This paper presents the outcomes of our ongoing collaboration with a non-profit organization dedicated to reentry support. We highlight the multifaceted challenges individuals face during their reentry journey, including support programs that prioritize supervision over service, unresponsive support systems, limited access to resources, financial struggles exacerbated by restricted employment opportunities, and technological barriers. In the face of such complex social challenges, our work aims to facilitate our partner organization’s ongoing efforts to promote digital literacy through a web application that is integrated into their existing processes. We share initial feedback from the stakeholders, draw out four implications: supporting continuity of care, promoting reflection through slow technology, building in flexibility, and reconfiguring toward existing infrastructure, and conclude with a reflection on our role as partners on the side.
AI Art for Self-Interest or Common Good? Uncovering Value Tensions in Artists’ Imaginaries of AI Technologies
The design justice of AI technologies is influenced by value tensions, by which values get embedded and excluded in design processes. In this study, we investigate Euro-Western artists’ expressed values for the design of Creative AI technologies. We conducted four workshops, in which artists engaged in value sketching and critical discussion around Creative AI, and analyzed emerging values. In the sketches, the artists predominantly expressed values of self-interest, while when encouraged for a critical discussion artists expressed values aligned with interest of others. We discuss the challenge of aligning Creative AI with design justice, when values of common good are not inherently part of the technological imaginaries, but rather need to be explicitly evoked. We open up these value tensions, discussing values’ actionability, and the implications for design futures of Creative AI. With this paper, we extend the empirical studies in value sketching and socio-technical landscape of Creative AI.
SESSION: Drones and Robots
Session details: Drones and Robots
REX: Designing User-centered Repair and Explanations to Address Robot Failures
Robots in real-world environments continuously engage with multiple users and encounter changes that lead to unexpected conflicts in fulfilling user requests. Recent technical advancements (e.g., large-language models (LLMs), program synthesis) offer various methods for automatically generating repair plans that address such conflicts. In this work, we understand how automated repair and explanations can be designed to improve user experience with robot failures through two user studies. In our first, online study (n = 162), users expressed increased trust, satisfaction, and utility with the robot performing automated repair and explanations. However, we also identified risk factors—safety, privacy, and complexity—that require adaptive repair strategies. The second, in-person study (n = 24) elucidated distinct repair and explanation strategies depending on the level of risk severity and type. Using a design-based approach, we explore automated repair with explanations as a solution for robots to handle conflicts and failures, complemented by adaptive strategies for risk factors. Finally, we discuss the implications of incorporating such strategies into robot designs to achieve seamless operation among changing user needs and environments.
Shaping and Being Shaped by Drones: Programming in Perception-Action Loops
In a long-term commitment to designing for the aesthetics of human–drone interactions, we have been troubled by the lack of tools for shaping and interactively feeling drone behaviours. By observing participants in a three-day drone challenge, we isolated components of drones that, if made transparent, could have helped participants better explore their aesthetic potential. Through a bricolage approach to analysing interviews, field notes, video recordings, and inspection of each team’s code, we describe how teams 1) shifted their efforts from aiming for seamless human–drone interaction, to seeing drones as fragile, wilful, and prone to crashes; 2) engaged with intimate, bodily interactions to more precisely probe, understand and define their drone’s capabilities; 3) adopted different workaround strategies, emphasising either training the drone or the pilot. We contribute an empirical account of constraints in shaping the potential aesthetics of drone behaviour, and discuss how programming environments could better support somaesthetic perception–action loops for design and programming purposes.
Drones as Accessibility Probes in Able-Bodied Norms: Insights from People with Lived Experiences of Disabilities
We present an exploratory, in-the-wild study in which a small hobby drone and a game controller were freely used by five people in their domestic environments, indoors and outdoors. All participants had motor disabilities affecting their arms and hands, and two also used wheelchairs. One participant contributed as a community researcher, assisting in data analysis, reflecting on findings, drawing conclusions, and co-authoring this paper. The findings reveal several usability and accessibility issues, along with potential risks and opportunities for the use of hobby drones in everyday situations. Beyond these insights, we discuss the importance of including people with lived experience of disability in research to shape a holistic and inclusive understanding of the use of mainstream artifacts such as hobby drones. This also helps prevent able-bodied design norms from limiting who can use drone technology and how it is used.
Wheel of Plush: A Co-Design Toolkit for Exploring the Design Space of Smart Soft Toy Materiality
Soft toys foster strong and enduring early childhood attachments, with positive effects extending into adulthood. Smart toys are vulnerable to exploits and can harm users. Bridging this contrast, pairs of smart objects, equipped with only simple sensors and actuators, may support peripheral and emotional awareness. At the same time, how exactly such pairs should negotiate the soft/smart spectrum to yield positive long-term impacts for people connecting through them is a design challenge. We engage this with the Wheel of Plush co-design toolkit. It enables 8×8 different sensor-actuator combinations in plush for co-designers to explore multimodal interactions for smart soft toy pairs connected over distance. We detail our design process and offer insights on designing a toolkit that combines artisanal plush material with simple sensors and actuators. With the Wheel of Plush, we contribute a toolkit for exploring smart soft toy materiality, data-frugal multimodal interaction, and opportunities for smart soft toy pair co-design.
SnuggleBot the Companion: Exploring In-Home Robot Interaction Strategies to Support Coping With Loneliness
We explored the use of three robot interaction strategies to support people living with loneliness (physical comfort, social engagement, requiring care), by building these into a robot prototype and deploying the robots into homes for long-term evaluation. We placed our original prototype, SnuggleBot, unsupervised into the homes of seven people for at least 7 weeks (optionally up to 6 months), with bi-weekly interviews, to investigate how people engage with our three robot interaction strategies. Our qualitative analysis illuminated how people engaged the robot based on all three interaction strategies. Further, some participants showed signs of bonding with the robot as well as self-reported wellbeing benefits, while some participants failed to achieve sustained use over time. Our results provide strong support for future research into robots developed with our interaction strategies, and general potential for supporting wellbeing.
How to Train Your Drone: Exploring the umwelt as a design metaphor for human-drone interaction
How To Train Your Drone is a novel human-drone interaction that demonstrates the generative potential of a design metaphor: the umwelt. We describe the concept of the umwelt and detail how we applied it to inform our soma design process, creating an interactive space where somatic understandings between human and drone could emerge. The system was deployed for a month into a shared household. We describe how three people explored and shaped the umwelts of their drones, leading to unique and intimate human-drone couplings. We discuss the compatibility of the umwelt to soma design practice and identify future avenues for research inspired by artificial life and evolutionary robotics. As our contribution, we illustrate how the umwelt as a design metaphor, can open up a generative new design space for human-drone interaction.
SESSION: Hyperobjects, Climate Change Data and Activism
Session details: Hyperobjects, Climate Change Data and Activism
Designing with What Remains
Goods are designed, produced, and eventually purchased by consumers who will typically use them as their marketing and branding suggests. What happens when, instead, existing products are used to provide the foundational components for novel devices? We argue that post-consumer product reuse should be a central and ongoing concern in the design of interactive devices. This paper examines four case studies of interactive devices produced by reusing and repurposing existing consumer products. By critically analyzing these case studies, we demonstrate how reuse is leveraged in practice to exploit macro and micro economic niches. These case studies illustrate factors beyond noble ecological intent that make reuse sustainable: mechanical robustness, affordability, and technological bootstrapping. We conclude with a discussion of the challenges and opportunities inherent to this mode of production.
Autonomy, Affect, and Reframing: Unpacking the Data Practices of Grassroots Climate Justice Activists
Though not often considered primary users or creators of climate change data, grassroots climate activism is increasingly data driven. This study looks at the ways in which grassroots climate justice groups engage with data to further their goals. The authors employ a qualitative research design rooted in reflexive thematic analysis for this project, with methods including a series of semi-structured interviews and an analysis of digital content produced by grassroots climate justice groups. We identify five distinct functions of data which support the work of local climate activists. These functions highlight how engagement with data in this context is intertwined with autonomy, affect and the reframing of what counts as climate data and expertise. This study contributes to further understanding of the relationship between grassroots climate justice groups and data practices, highlights barriers groups face with data engagement, and offers recommendations for HCI to further support local climate action.
Towards Relatable Climate Change Data: Untangling Tensions in Engaging with a Hyperobject
This research investigates the potential of emerging communication strategies to enhance engagement with climate change data through HCI, by recognizing the critical challenge of effectively communicating complex hyperobejcts. We designed “Finding Arcadia”, an interactive artefact centred on ocean climate data, to explore how data humanism, storytelling, decentering the human in the narrative, and positive framing influence user engagement and perception of the information. Findings from a study in-the-wild (N=42) and a post-experience survey conducted six months later (N=19) foregrounds strategies to foster deeper engagement and connection with the information but also tensions in engaging with such a complex topic. We contribute to climate change communication and HCI research with the design decisions, study outcomes, and reflections on ways in which communication strategies can promote understanding and connection with a hyperobject.
Systemic Sustainable HCI: Integrating Collaborative Modeling into a Design Process to Address Rebound Effects
The introduction of a new product or service into society can have detrimental effects on the environment due to changes in usage and practices. Human-Computer Interaction (HCI) designers need practical methods to address it. In this paper, we explore whether systemic modeling could help designers understand and mitigate these rebound effects. Drawing on systemic design and system dynamics, we prototyped a collaborative modeling methodology that ten professional designers applied to a case study in two workshops. We share insights on their adoption of the methodology and their feedback on its usefulness, usability, and feasibility. Our results indicate that designers find the systemic modeling approach relevant and useful for addressing rebound effects. Influence diagrams can help structure design ideas and identify unaddressed points, while dynamic modeling can help compare design strategies. Based on these findings, we propose a framework for integrating this approach into a design process.
Threads of Traceability: Textile IDs in the Fabric of Sustainable Fashion
Textile fabrication, an ancient human technology, has evolved over millennia, transitioning from a focus on affordability and speed to a current emphasis on sustainability. With Textile ID, we envision a digital garment passport that seamlessly incorporates directly into textile surfaces as a design element to bridge the gap between sustainability and consumer engagement, transforming garments into interactive storytellers of their ecological journey. The visual surface of the garment can be scanned with a smartphone to access a unique identifier embedded within the fabric, which provides essential information about the product’s lifecycle. This work discusses the design space of various visual and textile parameters, proposes design possibilities and insights for implementation. Finally, we showcase a set of sample garment designs and provide design recommendations for designers to use in their future work.
Understanding the Initial Journey of UX Designers Toward Sustainable Interaction Design: A Focus on Digital Infrastructure Energy Reduction
Environmental sustainability is increasingly important, and actions on “digital sustainability” are expanding to reduce energy consumption from digital infrastructures. As many digital services today have extensive user bases, exploring sustainable design features holds significant potential for reducing environmental impact. However, further exploration of foundational research is still necessary to enable broader and more effective adoption of digital sustainability in design practice. This study focuses on understanding important considerations when encouraging more designers, especially those with limited expertise in sustainability-oriented design, to integrate sustainable practices into digital services—acknowledging that embracing unfamiliar approaches presents natural challenges. We conducted design workshops and debriefing interviews with user experience (UX) designers unfamiliar with design for sustainability to explore their early encounters with sustainable interaction design (SID) in the context of digital infrastructure energy reduction. Our study provides insight into designers’ initial perceptions and challenges with sustainable design and discusses opportunities for their broader engagement.
SESSION: Response-able Design
Session details: Response-able Design
Tuning into the World: Designing Community Safety Technologies to Reduce Dysfunctional Fear of Crime
Platforms like Nextdoor and Citizen can increase users’ fear of crime by broadcasting frequent, local, and personalized information about potential safety risks. These platforms can contribute to a dysfunctional fear of crime, which undermines a person’s quality of life and mental health without actually making them feel safer. In this work, we conducted a mixed-methods study to understand the potential for design to foster a functional fear of crime, which motivates precaution without negatively impacting quality of life. We first interview individuals with a dysfunctional fear of crime and then validate interview results with a survey. Through this process, we identified five strategies for designers to support users in developing a more functional fear of crime. These strategies surface overarching theoretical and design implications for designers and researchers of safety platform with the ultimate goal of supporting safety, quality of life, and mental health for users of these platforms.
Engaging Women with Gestational Diabetes Mellitus in the Design of Self-Management Apps
Gestational diabetes mellitus (GDM) is an increasingly prominent health issue in pregnant women. While various technology solutions have been developed to support self-management of women with GDM, usability and functionality limitations have precluded their adoption. More active engagement with women with GDM in the design process could mitigate these limitations, thus we developed a design method to support the involvement of women with GDM in the design phase of a GDM self-management system. Thirteen online workshops were conducted involving five women with GDM and two postpartum women with previous GDM, who participated in idea generation, paper-based sketching, and group discussions, followed by interviews to gather their experiences of participation. We found that women valued their inclusion in these design workshops and felt confident sharing their ideas, from which we introduce recommendations for design procedures that enhance the contributions of women with GDM in the design of self-management apps.
Close a door to open a window: Unpacking strategies and know-how in making comfortable.
Together with 11 lower-income households, we explored how assumptions embedded in domestic energy interfaces match their needs, expectations, and everyday practices around domestic energy use and comfort. Catalyzed by cultural probes, residents (1) shared the diverse strategies and know-how involved in making themselves comfortable, (2) offered arguments for their everyday futures, and (3) explained what we can learn from them. A first tentative definition of energy interfaces opens the door to what we consider interfaces, how they support developing know-how and how they co-construct everyday practices. Residents’ resourceful solutions elicit reflections on what is considered valid participation in the energy transition. From the findings, we synthesize alternative starting points for the design of energy interfaces: We outline design opportunities to expand the (un)comfortable by revisiting comfort as fluid and multi-sensory. Furthermore, we sensitize how design can build on residents’ existing strategies for making comfortable, rather than replace them.
Toward Building Design Empathy for People with Disabilities Using Social Media Data: A New Approach for Novice Designers
Design empathy is a core HCI concept for understanding user perspectives in design processes. Although researchers advocate for leveraging design empathy in the design of assistive technology, educating novice designers about this is challenging; this is especially true in HCI classrooms when the target population includes people with disabilities, and students who do not have a disability are less aware of the diversity of disability. To help students better understand disability experiences, HCI education often adopts “be-like” (mimicking disabled-experience) approaches. However, accessibility researchers advocate adopting the “be-with” approach—learning about other’s experiences through companionship. To mitigate the logistical challenges of being-with in a classroom setting, we developed a “be-connected” approach, which facilitates learning about the disability experience through the narratives of real individuals. Using social media posts from a spinal cord injury subreddit, we developed and deployed an activity aiming to develop design empathy. Our qualitative evaluation showed a notable transformation in students’ design thinking process, suggesting an opportunity to leverage social media data to learn about disabled perspectives and develop design empathy.
Creating Resources for Designing with and for Care Ecologies in HCI
Amidst a growing body of work in HCI focusing on designing systems that engage with care relationships, there is increasing interest in expanding notions of care beyond transactional practices, towards broader notions of “care ecologies”. However, how can we support care systems designers to apply these concepts in practice? This paper presents the Care Spectrums, a set of sensitising concepts for designers to explore and apply to their practice. Developed as a response to a design probe exercise (the CareTree) which was carried out with 14 participants over one month, the Care Spectrums respond to the multiplicity of expressions of care in participants’ everyday lives. Translated into a design resource (the ‘Co-Designing with Care’ card deck), and trialled with 10 designers, the Care Spectrums revealed hidden caring and uncaring practices in designers’ projects, and stimulated opportunities for designing with and for people’s complex and entangled care ecologies.
Examining Algorithmic Metrics and their Effects through the Lens of Reactivity
Algorithmic systems that provide quantitative assessments of labor practices have proliferated in response to growing calls for accountability and transparency. Workers, according to the existing literature, are able to make sense of algorithmic metrics and even find ways to manipulate them. Human reactions like these show how metrics for tracking and measuring labor productivity can have unintended consequences, such as gaming the system, beyond the original goal of collecting accurate data on worker output. However, these metrics’ effects have not been much discussed from the perspective of reactivity. Drawing from Espeland and Sauder’s work theorizing reactivity to measures, I offer a deeper understanding of responses to algorithmic systems. Distilling three intertwined facets from their insights—quantification towards accountability, agency, and reflexivity—I frame my fieldwork findings on warehouse workers’ experiences with labor-tracking technologies. I describe the patterns of algorithmic system effects. Lastly, I explore potential design directions, viewed through the lens of reactivity.
SESSION: Resistance, Methods and Alternatives
Session details: Resistance, Methods and Alternatives
“Shotitwo First!”: Unraveling Global South Women’s Challenges in Public Transport to Inform Autonomous Vehicle Design
We call attention to the challenges associated with Global South women’s safety in public transportation and investigate the potential of autonomous vehicles (AVs) in providing them with greater mobility and broader opportunities. In a mixed-methods study with Bangladeshi women (n=23), we explored their safety issues, including sexual harassment and assault, to inform AV design, especially for shared rides. Our focus group findings revealed women’s distressing experiences of abuse and undertaken safety measures in public transport of Global South. We conducted co-design sessions utilizing virtual reality (VR) scenarios and investigated participants’ perceptions of potential AV designs addressing unique safety concerns and transportation challenges. Participants suggested prioritizing their own safety, achieved through design justice of equitable AV, over the current, often ineffective, retributive justice. Our work contributes to AV design, ICTD, and feminist HCI by suggesting implications for designing community-based and culturally contextual transportation infrastructure for Bangladeshi women and similar other communities.
“For Us By Us”: Intentionally Designing Technology for Lived Black Experiences
HCI research to date has only scratched the surface of the unique approaches racially minoritized communities take to building, designing, and using technology systems. While there has been an increase in understanding how people across racial groups create community across different platforms, there is still a lack of studies that explicitly center on how Black technologists design with and for their own communities. In this paper, we present findings from a series of semi-structured interviews with Black technologists who have used, created, or curated resources to support lived Black experiences. From their experiences, we find a multifaceted approach to design as a means of survival, to stay connected, for cultural significance, and to bask in celebratory joy. Further, we provide considerations that emphasize the need for centering lived Black experiences in design and share approaches that can empower the broader research community to conduct further inquiries into design focused on those in the margins.
RePresent: Enabling Access to Justice for Pro Se Litigants via Co-Authored Serious Games
Increasing numbers of people represent themselves in legal disputes—known as pro se litigants. Many lack the skills, experience, or knowledge to navigate legal proceedings without a lawyer, resulting in limited access to justice. Serious games may provide an effective, interactive, and engaging way of educating pro se litigants about the law and enabling their access to justice. Through participatory design with legal experts and an authoring tool, we co-designed RePresent, a serious game that helps individuals with limited access to legal support prepare for pro se litigation. A total of 965 people played RePresent, and 149 provided feedback on their player experience. Results show that RePresent was engaging and valuable for learning about the law and pro se litigation. Our work highlights avenues for co-design methodologies with co-creative authoring tools that facilitate serious game design, contributing a potentially scalable solution to enable access to justice via co-authored serious games.
Querying the Quantification of the Queer: Data-Driven Visualisations of the Gender Spectrum
Critical data studies denounce that heteronormative formalisations of sex/gender identity obscure queer lives from datasets and computational processes. Since sex/gender categories cannot convey nonbinarity and multiplicity, this project addresses the inherent tension between queerness and systematisation with an anticategorical quantitative visual approach, combining psychometric scales with visualisations of the gender spectrum. Indeed, using queer and feminist methodologies, we propose a new data structure resulting from the codesign of an interactive visualisation of the gender spectrum, made in an iterative dialogue with gender-diverse people. Our contribution is twofold: an inclusive data structure for the self-assessment of sex/gender identity, and an interactive visualisation, acting as a critical design object, educational and counselling tool, and basis for data collection, annotation, and analysis. Although this project could both empower and harm sexual minorities, its originality is seeing nonbinarity and multiplicity not as appendices to a normative system, but as queering design principles.
Counting Up: Designing Agonistic Data Collection in the Court Room
This paper details an activist court-watching campaign to collect data during eviction hearings, working with a tenant’s rights organization and a legal aid group. Our work provides a thick description of compiling an agonistic data practice that studies up, fleshing out the conception brought forth by previous scholars. We describe this project’s planning, implementation, results, and challenges to demonstrate how designing data production can affect fairness and accountability in the courtroom. From our account, we offer insights for scholars wishing to contribute to data activism projects: to focus on designing data production not data products, to look for affective potential within messy data, and to shift epistemic burdens onto those who may benefit more from the data.
SESSION: Shadows, Disengagements Somatic Design
Session details: Shadows, Disengagements Somatic Design
In Praise of Shadows: Sensibility and Somaesthetic Appreciation for Shadows in Interaction Design
Wherever there is light, there is shadow — an inevitable immaterial material, with an undeniable presence in interaction gestalt. Inspired by Jun’ichirō Tanizaki’s book In Praise of Shadows, we report on our journey towards building a sensibility and somaesthetic appreciation for shadows physically, metaphorically, and poetically. To investigate the idea of cultivating attentiveness to shadows, we embarked on a “dérive”-inspired adventure, gathering first-person perceptions and photographs of tables. We analysed these examples of tables and their shadows in order to inform the development of a project on interactive tables. Along with the artifact collection, we present a set of concepts, an initial prototype, and reflections on our own experience in developing this sensibility. We discuss shadows as a design material and an example of a design sensibility: a skill that can be sparked, fostered, and ultimately embedded.
Autosomatographical Narratives: Towards the Articulation of Felt Accounts of Pain for Somaesthetic Design
Chronic pain can be debilitating, taking over our senses and potentially destroying our means of describing our experiences. In this pictorial, we contribute three concepts based on the analysis of autosomatographies, a first-person narrative approach from illness and disability studies to communicate experiences of pain. These concepts suggest designers consider pain as (a) self-agentic, (b) a tool for empowerment, and (c) a means of creative expression. We illustrate the generation of these concepts through the first author’s autosomatographical journals, highlighting how visual research greatly contributes to articulating expressive felt qualities of pain for design use. To exemplify the translation of these concepts into tangible prototypes, we retrospectively analysed two soma design case studies created to appreciate our bodies and learn from pain. We discuss the importance of acknowledging the persistence of chronic pain and the ethics of engaging in this type of design work.
Movits: a Minimalist Toolkit for Embodied Sketching
We present the design and evaluation of the Movits, a minimalist toolkit for embodied sketching design explorations. The toolkit includes technology probes featuring minimalist wearable digital units that support the hands-on exploration and design of movement-driven interactions using multisensory feedback. The Movits are self-contained and generate audiovisual or vibrotactile patterns in response to movement-based inputs. We present the theoretical and empirical grounding driving our design process. We discuss the findings of using the Movits during four co-design workshops with design students, technologists, dancers and physiotherapists, where they resulted in being generative and adaptable to a range of embodied design approaches. We contend that the Movits can be favourable for those interested in a holistic design approach to wearables in general and specifically for those targeting movement-based application domains.
Designing for Embodied Sense-making of Mathematics: Perspectives on Directed and Spontaneous Bodily Actions
While mathematics is conventionally viewed as an abstract discipline, contemporary perspectives on embodied cognition underscore the significance of integrating students’ bodily experiences into the learning process. However, the efficacy of embodied learning activities, as compared to traditional methods, remains under scrutiny. We argue that both directed and spontaneous bodily actions should be considered when designing embodied learning activities, and explore such bodily actions through two studies. A quantitative user study involving directed bodily actions in Virtual Reality and on tablet reveals vr’s support for math-anxious and body-aware learners, and distinct movement patterns related to varying mathematical abilities. A subsequent qualitative analysis identifies key characteristics of spontaneous bodily actions, namely coarseness, muscle tension, repetitions, anchors, perspective, and metaphors. Derived from both studies, we propose design recommendations, advocating for expanded embodied interaction design, consideration of embodied metaphors, coarse gesturing for deep features identification, supporting of sense-making anchors, and in-vr learning assessments.
Articulating Mechanical Sympathy for Somaesthetic Human-Machine Relations
We present mechanical sympathy as a generative design concept for cultivating somaesthetic relationships with machines and machine-like systems. We identify the qualities of mechanical sympathy using the design case of How to Train your Drone (HTTYD), a unique human-drone research product designed to explore the process by which people discover and co-create the somaesthetic potential of drones. We articulate the qualities – (i) machine-agency, (ii) oscillations, and (iii) aesthetic pursuits – by using descriptive and reflective accounts of our design strategies and of our co-creators engaging with the system. We also discuss how each quality can extend soma design research; conceptualizing of appreciative, temporal, and idiosyncratic relationships with machines that can complement technical learning and enrich human-machine interaction. Finally, we ground our concept in a similar selection of works from across the HCI community.
Exploring the Somatic Possibilities of Shape Changing Car Seats
Through a soma design process, we explored how to design a shape-changing car seat as a point of interaction between the car and the driver. We developed a low-fidelity prototyping tool to support this design work and describe our experiences of using this tool in a workshop with a car manufacturer. We share the co-designed patterns that we developed: re-engaging in driving; dis-engaging from driving; saying farewell; and being held while turning. Our analysis contributes design knowledge on how we should design for a car seat to ‘touch’ larger, potentially heavier parts of the body including the back, shoulders, hips, and bottom. The non-habitual experience of shape-changing elements in the driver seat helped pinpoint the link between somatic experience and intelligent rational behaviour in driving tasks. Relevant meaning-making processes arose when the two were aligned, improving on the holistic coming together of driver, car, and the road travelled.
SESSION: Privacy and Smart Homes
Session details: Privacy and Smart Homes
Designing Interactive Privacy Labels for Advanced Smart Home Device Configuration Options
Labels inform smart home users about the privacy of devices before purchase and during use. Yet, current privacy labels fail to fully reflect the impact of advanced device configuration options like sensor state control. Based on the successful implementation of related privacy and security labels, we designed extended static and interactive labels that reflect sensor states and device connectivity. We first did expert interviews (N = 10) that informed the final label design. Second, we ran an online survey (N = 160) to assess the interpretation and usability of the novel interactive privacy label. Lastly, we conducted a second survey (N = 120) to investigate how well our interactive labels educate users about sensor configuration. We found that most participants successfully used the interactive label and retrieved sensor information more efficiently and correctly. We discuss our findings in the context of a potential shift in label use toward control and use-case-based interaction.
Designing Smart Home Technology For Passive Co-Presence Over Distance
When families live in the same home, they feel a sense of connection through the subtle, passive aspects of family life. Over distance, these passive aspects are hard to experience as most communication technologies support sharing conversations or activities. Through a research-through-design methodology, we explored the design of smart home technologies for passive co-presence over distance. Based on our design explorations, we arrived at an interaction space which includes the dimensions of Activity, Solitude, Synchronicity, and Spontaneity. Our research-through-design process additionally resulted in the design of two smart home systems. The There Chair employs the senses of touch and sight to passively display when a remote family member is sitting at the dining room table. The Fragrance Frame is a paired picture frame that detects when a remote family member is passing by their frame, and emits a scent reminiscent of togetherness. We reflect on our design decisions and propose considerations for future design.
How We Use Together: Coordinating Individual Preferences for Using Shared Devices at Home
Smart home devices increasingly aim to offer personalized experiences based on individual usage patterns and preferences. However, individuals in multi-person households often find themselves navigating daily life by coordinating with other members. This article aims to understand how users align their preferences for shared technology and their expectations for future smart homes in such contexts. To this end, we conducted a 7-day diary study and in-depth interviews to observe how members of multi-person households coordinate their preferences in everyday life. We discovered that members’ device usage is not solely influenced by their own preferences, but is significantly affected by the preferences and social contexts of other members. We identified three strategies for coordinating preferences: coordination according to shared priorities, coordination according to shared rules, and coordination for relationships. Participants expected future smart homes to support their coordination needs. We propose design implications for multi-user smart homes that support coordination.
Airbnb Digital Concierges: Designing Interactions for Authentic Guest Experience
Airbnb is well known for providing guests with local experiences by regular people worldwide. Many pioneer hosts experiment with emerging smart products to improve guests’ experiences. However, these initial efforts often merely provide functionality, neglecting to offer guests authentic experiences motivated by hosts’ authenticity. We addressed this gap by using bespoke design methods to understand hosts’ stories and co-develop interaction concepts that can express their authenticity and fit their Airbnb. We implemented a set of Digital Concierges (DCs) and deployed them in a real Airbnb to investigate guests’ experiences. Our results show that our design enhanced guests’ sense of welcome, fostered their discovery with serendipitous cues, and built richer connections with the host and locals. We concluded with the design implications of co-designing artifacts to embody hosts’ authenticity and use multiple modalities to enhance perceived authentic experiences. Most importantly, we shared our insights about working with hosts as co-creators.
Digital, Analog, or Hybrid: Comparing Strategies to Support Self-Reflection
Recently, various digital solutions have emerged to enhance the process of self-reflection, which can be crucial for personal growth and resilience. However, whether technology can meaningfully match or augment a traditional approach like pen or paper remains to be ascertained. Our objective was to build an better understanding of design paradigms’ role in introspection. Through formative iterations, informed by Self-Determination Theory (SDT), we designed and developed different tool formulations (Analogue, Digital, and Hybrid) for comparison. Participants (N = 48) received one variant, completing a pre- and post-six-week assessment with the Self Reflection and Insight Scale (SRIS) and intermediary self-reports for qualitative feedback. We found scores for Hybrid and Digital conditions change significantly, suggesting format decisions have meaningful impacts on the efficacy of designs to alter intrinsic motivation toward introspective behaviour. We also identify determinants and design considerations to help others conceive solutions to support or stimulate a component of broader well-being.
Manual, Hybrid, and Automatic Privacy Covers for Smart Home Cameras
Smart home cameras (SHCs) offer convenience and security to users, but also cause greater privacy concerns than other sensors due to constant collection and processing of sensitive data. Moreover, privacy perceptions may differ between primary users and other users at home. To address these issues, we developed three physical cover prototypes for SHCs: Manual, Hybrid, and Automatic, based on design criteria of observability, understandability, and tangibility. With 90 SHC users, we ran an online survey using video vignettes of the prototypes. We evaluated how the physical covers alleviated privacy concerns by measuring perceived creepiness and trustworthiness. Our results show that the physical covers were well received, even though primary SHC users valued always-on surveillance. We advocate for the integration of physical covers into future SHC designs, emphasizing their potential to establish a shared understanding of surveillance status. Additionally, we provide design recommendations to support this proposition.
SESSION: Design Politics and Ethics
Session details: Design Politics and Ethics
Creative Precarity in Motion: Revealing the Hidden Labor Behind Animating Virtual Characters
This paper illuminates the obscured labor and challenges of actors—those vitalizing virtual characters through their embodied performances through motion capture (mocap) technology. Mocap in the entertainment industry has amplified the presence of virtual entities across various media platforms. Despite the expanding influence of such technologies, the mocap actors behind them often remain unseen and, at times, undercompensated. Central is the paradox where the realism and vitality of virtual characters contrast sharply with the obscured, and sometimes undervalued, human efforts behind them. We present an empirical study based on semi-structured interviews with fourteen actors, offering a nuanced understanding of their lived experiences. We highlight the various precarities they face, which include navigating the market, preserving artistic integrity, and facing vulnerabilities as employees. By juxtaposing these findings with established discourses on invisible labor, we discuss three major invisibility factors. Furthermore, we propose design interventions to empower and rightfully recognize human labor.
The Pictorial is Neurodivergent: A Case for More Fidgets and Fewer Fixes
This pictorial presents a case for designers of interactive systems to focus on supporting self-determined behaviors instead of aiming to “fix” behaviors deemed abnormal or problematic by society. It proposes a decision map to help determine whether it is ethical or appropriate to design a product or system for behavior intervention. Through the lens of neurodiversity, this work considers how designs for behavior intervention often mirror techniques utilized in behavioral therapies aimed at preserving neurotypical social expectations at the expense of individual autonomy. This pictorial also examines devices used for behavior intervention through photographs of existing products and technologies that support self-stimulatory behaviors (stimming). It utilizes a fictional narrative with digital illustrations and AI-generated imagery to explore the impact of an imagined technology designed to modify the behavior of neurodivergent individuals. “Fidgets” are presented as a product exemplar that supports and celebrates self-determined, intuitive behaviors rather than suppressing or modifying behaviors to conform to social norms.
“My Sense of Morality Leads to My Suffering, Battling, and Arguing”: The Role of Platform Designers in (Un)Deciding Gig Worker Issues
HCI and design studies have increasingly identified challenges for gig workers and advocated for designs centered around worker justice. However, there’s an existing research gap in understanding how platform designers approach gig worker issues in their practice. Our study engaged ten platform designers from food delivery and ride-hailing platforms to investigate this gap. Through semi-structured interviews, we uncovered their strategies, the extent of authority and responsibilities, and the range of obstacles they encounter in influencing decision-making that could affect gig workers’ experiences with the platforms. While platform designers were aware of gig worker issues, they confronted challenges from business goals, decision-making power, policies, and job security in promoting worker well-being. We discuss the jurisdiction of platform designers and propose that HCI research should further support them, who are deeply engaged in the gig economy and have the potential to participate in addressing social justice issues.
Ethics Pathways: A Design Activity for Reflecting on Ethics Engagement in HCI Research
This paper introduces Ethics Pathways, a design activity aimed at understanding HCI and design researchers’ ethics engagements and flows during their research process. Despite a strong ethical commitment in these fields, challenges persist in grasping the complexity of researchers’ engagement with ethics —practices conducted to operationalize ethics—in situated institutional contexts. Ethics Pathways, developed through six playtesting sessions, offers a design approach to understanding the complexities of researchers’ past ethics engagements in their work. This activity involves four main tasks: recalling ethical incidents; describing stakeholders involved in the situation; recounting their actions or speculative alternatives; and reflection and emotion walk-through. The paper reflects on the role of design decisions and facilitation strategies in achieving these goals. The design activity contributes to the discourse on ethical HCI research by conceptualizing ethics engagement as a part of ongoing research processing, highlighting connections between individual affective experiences, social interactions across power differences, and institutional goals.
One Size (Doesn’t) Fit All: Exploring design considerations for digital body dissatisfaction interventions with underrepresented populations
Traditionally, body dissatisfaction interventions have been designed with a focus on females from Western cultures. However, with growing research indicating that body dissatisfaction is experienced across society, regardless of gender and cultural background, it is increasingly important that future interventions incorporate a broader range of socio-cultural experiences. We conducted a two-phase co-design study with thirteen participants (seven females, six males), aged 18-24, from diverse cultural backgrounds. Phase 1 aimed to understand the influencing factors that frame the development of body image perceptions. Drawing on insights from Phase 1, Phase 2, then explicitly focused on gathering design insights for digital tools for body dissatisfaction interventions. Four narrative design concepts were used to provoke discussions and ideate around potential digital interventions. Through this paper, we contribute unique insights into the experiences and digital intervention preferences of underrepresented people in body image research and highlight future directions to create more inclusive digital interventions.
Art Directing Blended Experiences
Interactions between the physical and the digital have become increasingly ubiquitous. They are particularly challenging to visualize and illustrate, as they usually unfold over time and space and involve multiple devices, locations, services, and actors. Traditional approaches to sketching user experiences and storyboarding fall short of clearly illustrating how meaningful relationships between people, things, technology, and places are created when they are consistently transitioning from physical spaces into digital spaces and back again. To address this gap, the pictorial showcases how a design framework based on blending theory informs our illustrative best practices and techniques for Art Directing Blended Experiences. The pictorial further discusses the opportunities and challenges of this narrative sketching technique to help designers further explore and capture the essence of how things, relationships, people, and change weave together new personal social spaces and deeper meanings of physical place.