In 2014, an international group of scholars from various fields analysed the "societal dimensions" of synthetic biology in an interdisciplinary summer school. Here, we report and discuss the biologists' observations on the general perception of synthetic biology by non-biologists who took part in this event. Most attendees mainly associated synthetic biology with contributions from the best-known public figures of the field, rarely mentioning other scientists. Media extrapolations of those contributions appeared to have created unrealistic expectations and irrelevant fears that were widely disconnected from the current research in synthetic biology. Another observation was that when debating developments in synthetic biology, semantics strongly mattered: depending on the terms used to present an application of synthetic biology, attendees reacted in radically different ways. For example, using the term "GMOs" (genetically modified organisms) rather than the term "genetic engineering" led to very different reactions. Stimulating debates also happened with participants having unanticipated points of view, for instance biocentrist ethicists who argued that engineered microbes should not be used for human purposes. Another communication challenge emerged from the connotations and inaccuracies surrounding the word "life", which impaired constructive debates, thus leading to misconceptions about the abilities of scientists to engineer or even create living organisms. Finally, it appeared that synthetic biologists tend to overestimate the knowledge of non-biologists, further affecting communication. The motivation and ability of synthetic biologists to communicate their work outside their research field needs to be fostered, notably towards policymakers who need a more accurate and technical understanding of the field to make informed decisions. Interdisciplinary events gathering scholars working in and around synthetic biology are an effective tool in addressing those issues.