More than Human Centred Design


SemanticCollage: Semantic Search for Visual Inspiration

  • Janin Koch, Aalto University, Helsinki, Finland
  • Nicolas Taffin, ExSitu, Inria, Paris, France
  • Andrés Lucero, Aalto University, Helsinki, Uusimaa, Finland
  • Wendy E. Mackay, LRI, Université Paris-Saclay, Paris, France
  • Corresponding email(s): mail@janinkoch.de
  • Research group webpage
  • ACM DL Link: Associated Paper or Pictorial

Have you ever struggled to find images online, like an illustration for a project outline, or to show a new style you like? Well, that might be hard because web-search engines are optimized to find the perfect match to a clear query — but they are not made to let you discover or combine ideas, especially visually. And yet, rough ideas and informal concepts often bring the most innovative results. Hence, designers often use visual collages also called mood boards to explore new project ideas. SemanticCollage allows you to search for images with words and images for more complex searches, by analyzing the semantic contents of each image. It also provides semantic overviews of the whole collage, or of selected parts in the form of tag-clouds. You can manipulate all labels for better overviews, or make your visual search more relevant by changing their individual semantic weight for the new search after you dragged an image to the search, which is shown underneath it, or when using the image editing tools. This can support you to find new ideas, make sense and reflect on your visual ideas, and also communicate them better to others by revealing relevant keywords.

Who is the target audience and why design for them? The project focuses on the creation of visual collages for the purpose of professional designers. However, this tool is not limited to this and can be of further assistance to anyone interested in visual collages. As we can see the proliferation of online platforms such as Pinterest, where users often collect images for personal ideation projects, we believe that such a tool can be of interest to the general audience of visual-thinking people. Beyond the application, we also think that this work could be of interest to machine learning enthusiasts working on creative applications. We would happily share our experiences, as this is a rather new field of exploration. Although there are major advances in technology, the accessibility of technology is often limited. The more abstract and creative the task, the more difficult it is often to facilitate such technical benefits. We believe that by focusing on such creative processes, we can not only build ML technology that makes current advancements accessible to such contexts, but we can, beyond that, support rather silent aspects such as sense making and reflection that were not previously feasible.

What were the challenges or limitations encountered in this project? In the context of creativity and technology, our work addresses a variety of issues. We first identified what challenges designers face in the mood board process. We conducted a participatory design workshop and surveys to respond to this. While visual ideation is a very intuitive process, professional mood board design requires refined expertise in abstract design thinking. This significantly limited the selection of potential participants for the workshops, surveys and the final study. Subsequently, we investigated potential technical solutions to address previously defined limitations. Semantic labeling was quickly identified as useful, but we carefully analyzed the benefits and limitations of building our own labeling algorithm rather than using existing ones. This type of algorithm is highly sensitive to the initial categories of input data, which then limit the images used by the designer. Due to the lack of large and diverse datasets to overcome this problem, we have decided to use existing algorithms in our work. Furthermore, the interaction design had to provide all the required functionalities for digital moodboard design, while integrating system suggestions in a non-distracting way to provide a fluid visual ideation process. Several prototypes and designer tests were carried out prior to the final design.

What are the opportunities and next steps for this project? SemanticCollage provides an example of how semantic analysis can be used to improve the ideation and reflection of visual material in the context of mood board design. However, many creative practices, including sketching, crafting and sculpting, could also benefit from the exploration of open-ended image and shape associations. Enriching system knowledge with semantic labels can improve the suitability of agent contributions and would also open up new semantic spaces to be explored by creative agents. An example of this would be the shared creative interaction between designers and artificial agents in ideation. A first step could be a design tool where designers and machine learning algorithms have shared agencies to explore, exchange and communicate ideas.

To the Demo Visitors: In this work we focused on a certain aspect of visual ideation: search, sense making and reflection of and with visual material. We’re interested in your perspective on where you would see semantic analysis of visual material useful otherwise.