Article Contents
UTEST, the online community of user experience (UX) practitioners and researchers, celebrates its 20th anniversary this year. If you are a member, you know it as a place where you can ask questions, get help and opinions from your UX colleagues, contribute to discussions, find out about potential jobs and interesting non-profit events, and even try out unpolished, inchoate ideas knowing that discussions will be professional and respectful. If you aren’t a member, consider joining us by writing to tharon@clemson.edu and asking for an application.
(Note that we usually spell the community’s name, UTEST, all capitals, no hyphen. Since the birth of the UTEST community, a number of commercial companies have given themselves similar-sounding names. This may be a tribute to the catchy name of our community and the reputation we have earned over the years. However, these companies are serving their own commercial interests and have no connection with our online community. The online community UTEST is not affiliated with any of these commercial companies.)
Over the past 20 years, UTEST members have come to love their community. With permission from these UX colleagues, we offer just a few of many testimonials:
UTEST is better than Google! When I have a question about methodologies, I get answers that are backed by years of experience, often times from those who are pioneers in our field. —Rebecca Destello
As a one-person Usability team, it’s invaluable to be able to confer with colleagues on stuff that no one else around here understands. —Mary Mascari
My years involved in UTEST have been a virtual Master Class in all areas touching User Experience. I owe much of my career progress to this fantastic community. —Tomas Sola
UTEST…was my connection [to usability], my link to mentors and education, and even occasionally my cheering section as I broke into the field. I owe the UTEST community a world of thanks for helping me start and grow my career. —Mitch Berg
The safety of UTEST for its members seems to me to arise from the fact that it is closed…and private…so we may feel relatively free to speak frankly and to float ideas that are less than fully developed and not (yet) rigorously supported by evidence. —Douglas W. Anderson
In this essay, we (Tharon Howard, the community manager, and the members of the UTEST advisory council) first briefly describe UTEST and its history:
- What is UTEST?
- How did UTEST happen?
We then offer our thoughts in answering two other questions you may have:
- Why has UTEST been so successful as an online community?
- What did a survey of UTEST members tell us about the community?
What Is UTEST?
The following is from the UTEST’s website:
UTEST exists to promote new approaches to user experience (UX) practices, increase general knowledge of UX-related disciplines, and facilitate healthy and productive discussions among community members while protecting their privacy and their rights to their own intellectual property.…
UTEST is not a “list,” “alias,” “archive,” “database,” or other public resource. It is a private COMMUNITY of professionals working together to create new knowledge and better practices.
How Did UTEST Happen?
Tharon tells it this way:
UTEST started out of a chance hallway conversation between sessions of the 1992 Conference on College Composition and Communication.
Mark Simpson from Microsoft, Bill Karis from Clarkson University, and Tharon began chatting about the work they were doing and the problems they were encountering running research studies and trying to build usability labs.
Back then, we all had very similar problems: How do you recruit test subjects? How much should you pay them? How do you report your findings to your clients in a way that actually gets them to implement your recommendations? How do you keep clients from turning your studies into fishing expeditions by generating so many different research questions that there’s no way to design a study that can collect all the necessary data? How do you convince administrators and managers to support you with resources? How do you train your staff or teach students to do this sort of work? Etc., etc., …
Someone observed that it would be really great if there were some way usability people could talk to each other about these kinds of nitty-gritty problems so that we didn’t always have to reinvent the wheel in isolation. Wouldn’t it be terrific, we all agreed, if someone could pull together a small group of professionals who also had enough dirt under their fingernails to be interested in the day-to-day business of conducting this kind of research? Wouldn’t it be great to share best practice stories with each other?
We couldn’t talk about our proprietary findings, of course, but surely we could talk about ways to improve our common practices and methodologies. Best of all, wouldn’t it be great to be able to gripe about a problem involved in running a usability lab and actually get a sympathetic response from someone who shared a visceral understanding of what you were suffering?
Over the next several months, UTEST grew out of private email exchanges among the original three colleagues and their friends. At first, we followed the model that the Royal Society used in the 1600s, passing messages from one person to the next with each recipient critiquing and commenting on the message. We began to build what Diana Crane called an “invisible college” (1972).
Unfortunately, however, this receive-critique-and-forward approach to email distribution was just too clunky. It might have worked in the 1600s, but it was an inefficient use of the technologies available to us.
As a result, Tharon reluctantly created the first UTEST email list—reluctantly for two reasons: First, getting the hardware, software, and, most particularly, the personnel resources donated to create and maintain UTEST has been and continues to be a constant issue. The resources are extremely costly. Second, and more importantly, Tharon’s experience as a list owner for several other groups had demonstrated pretty clearly that, while it was relatively easy to create a discussion group, it was extremely difficult (and potentially impossible) to create a community—particularly with the email distribution software packages then available.
Indeed, it was because of the software limitations and the desire to create a safe community that we began to set some pretty rigid rules for membership and participation. We wanted to maintain the same kind of trust we had in each other when we used the receive-critique-and-forward approach, and we wanted to maintain the same dynamics and quality of discussions.
And so, from the beginning, we established many of the policies UTEST still uses today:
- UTEST is open only to practitioners and researchers who can benefit from and contribute to the community. Potential members are invited to join through personal referrals. They must fill out a short application explaining their relevant experience and what they expect to get from and, just as importantly, what they will give back to the community.
- UTEST’s messages are not archived so that members feel free to try out new ideas and offer opinions that might change as the discussion progresses.
- UTEST members agree to not forward or redistribute messages, to quote messages only if they have the authors’ permission, to not advertise on UTEST, and to ask before announcing any event or survey.
- UTEST is a community of professionals. The netiquette policies make clear that members should assume that peers are reading their messages: peers who respect them and who in turn deserve their respect.
UTEST started with about 30 people from the invisible college. Over the years, that number slowly grew until it reached almost 1,600 in the late 1990s. However, the economic crash of 2001 took quite a few members into other fields and careers. Now, we have started to see many of them coming back, and many newly minted professionals have joined us. In 2013, UTEST membership hovers between 1,200 and 1,300 members.
Part of the expansion of UTEST from its small beginnings happened because of UPA (now UXPA). Shortly after Janice James started UPA (first as a birds-of-a-feather session at a SIGCHI conference in 1991, then as a separate meeting in 1992), some of the early UPA members discussed the need for an online community. However, by the time they were ready to start one, UTEST already existed. With both UPA and UTEST still being small, it seemed redundant to start a second online community. So, while UTEST is an independent online community with no official ties to UXPA, in a sense it is also the online community for UXPA members.
Although UTEST started with a focus on usability testing, as usability researchers and practitioners expanded their work to user-centered design and its larger tool kit of usability techniques and then even further into helping with the entire user experience, the community embraced these new topics and welcomed a broader spectrum of UX professionals.
Why Has UTEST Been so Successful as an Online Community?
For 20 years and counting, UTEST has been a strong, active, productive online community. We, the community manager and the advisory council, take some credit for that success, but most of the credit goes to the members of the community.
As keys to UTEST’s success and as advice for others considering starting an online community, we offer the following 13 suggestions. We present these suggestions within Tharon’s four-part model for successful online communities: RIBS (Howard, 2010).
RIBS stands for
- Remuneration
- Influence
- Belonging
- Significance
Remuneration
A community needs members who invest time and energy through their participation. People remain in an online community only if they believe they are getting value for their participation—both from getting and for giving to the community.
Remuneration is not necessarily about money. It can be derived from getting anything one values and from giving what others value. It can be a sense of influence, a feeling of belonging, and a belief that the community is significant. Thus, all four elements of RIBS work together.
UTEST supports remuneration in at least four ways:
Provide a safe environment for questions and discussions
As we mentioned above, UTEST does not archive discussions. Our policies tell members they cannot share UTEST discussions outside of the community (unless they have permission from all the community members whose words they want to share). This results in often fascinating and useful discussions.
Maintain a really good level of signal to noise
One of the main reasons for keeping UTEST closed and private is to keep spammers from finding it. In 20 years, glitches due to spamming have been extremely rare.
UTEST policies also keep the noise level low. For example, we allow announcements of jobs but not requests from job seekers. We allow announcements of not-for-profit events but not announcements of commercial events or products. Community members must check with the community manager before announcing even a not-for-profit event, as well as for permission to invite other community members to participate in a survey or other assessment.
The community manager and advisory council also try very hard not to bother community members with administrivia.
Encourage contributions and help members contribute appropriately
Contributions are critical to an online community. The first three guidelines for new members on UTEST’s help page welcome everyone’s contributions this way:
Read the posts before posting. This may seem obvious, but it helps to get a feel for the breadth and depth of discussion as well as the variety of topics and styles of posters.
Don’t be afraid to ask. This is a community of people with broad interests and experiences who, by definition, are interested in discussing topics related to usability and user experience.
Don’t be afraid to chime in. Even those who are new to the community or new to the field often have an experience or perspective that makes a significant contribution to the discussion.
However, to keep a community’s discussions civil and to ensure that contributions facilitate good discussions, UTEST has (and enforces) guidelines on what makes a good contribution. We encourage debates and disagreements as long as they remain professional.
Have consistent and clear governance
Perhaps the most important key learning from UTEST is to develop clear policies and always to consider the impact that policies and the community architecture will have on community members’ experience.
UTEST’s policies are always available for members to review. Applicants for membership in the community must acknowledge that they have read the policies.
When an issue arises that is not clearly dealt with in the policies, we not only deliberate on the specific issue but also discuss whether to add to or change the policies—always considering how our decisions will support the key qualities of UTEST as a safe, low-noise community that remunerates its members.
Influence
To feel welcome in, connected to, and valued as part of the community, members must believe they have a voice in the community and how it is run.
UTEST supports influence in at least these four ways:
Let people know the community manager welcomes comments
UTEST’s website encourages interactions between members and the community manager and advisory council. For example, you will find this encouragement on UTEST’s help page:
When in doubt, check it out. If you ever have a question that isn’t clarified to your satisfaction by the community’s website, send your question in an email to Tharon Howard at tharon@clemson.edu.
Create a strong, representative advisory council
UTEST’s advisory council helps the community manager by
- reviewing requests for announcements where agreement with policy is not immediately obvious,
- setting policy and deliberating carefully about possible changes in policy,
- responding to requesters once the council has decided—always with a respectful and clear rationale for the decision,
- supporting the community manager to keep the discussions within the policies and flame-free,
- reviewing posts and initiating discussions within the council of any problematic posts,
- sometimes sending a gentlegram (our term for a message that is gentle in tone) to a member who may not realize that others might have been hurt by or might have misinterpreted the tone or content of a message,
- reminding the community from time to time of how to stay within the policies,
- watching carefully the boundaries of what the community covers—resisting mission creep but allowing the community to broaden naturally as it has from usability to user-centered design to UX and from technologies that were relevant 20 years ago to technologies that are current today, and
- following up with members who leave the community, asking those members to assess their experiences with UTEST and to suggest ways to improve the community.
The advisory council collaborates on the same principles as the larger community: listening to each other, sharing sometimes quite divergent views, and always responding with courtesy and respect. Unlike the community at large, where consensus is never necessary, the advisory council strives for consensus.
Respond quickly to comments and questions from members
Tharon has been the only community manager UTEST has had. For more than 20 years, he has been there for every member of the community while managing several other online communities and keeping up a full load of academic teaching and research. Yet he manages to respond quickly to every crisis and query.
Sometimes the response is a request for patience as the advisory council deliberates on the issue across their several time zones, but it’s still important to let members know that their questions are important and are being heard.
Let members help other members
One of the foundational principles of UTEST is that as a community it is all about members helping each other. As the testimonials we quoted at the beginning of the essay show, this mutual support is critical to the success of UTEST. Members derive a sense of influence by helping each other.
Belonging
To make an online group into a fully functioning, interactive community, members must feel it is a privilege to be part of the community. They have to feel bonded to other community members.
UTEST supports belonging in at least these three ways:
Protect exclusivity
As Tharon has explained in his book, Design to Thrive, human beings are motivated by a desire to accumulate social capital just as much as they desire financial capital. “They don’t want to be just another face in the crowd. They want to be part of the in crowd” (Howard, 2010, 168).
Part of UTEST’s success is its exclusivity. UTEST’s by-invitation-and-application-only policy and its private status have made joining the community desirable for many UX professionals.
Of course, exclusivity as a success factor isn’t limited to UTEST. Many newer communities have adopted exclusivity as a model for success. Facebook became popular initially because it was available only at certain, select universities. Google’s Gmail gained popularity initially because it gave its early members a limited number of private invitations that they could use with their friends. More recently, Pinterest has users register and then wait a day or two (building the anticipation) for the privilege of creating an account.
Create an application process that socializes new members
Part of being an exclusive community is asking people to apply for membership. UTEST’s short application form also helps new members understand the nature of the community—that it is a community, not just a listserve; that they must have relevant experience to be part of the community; and that they are expected to both benefit from and contribute to the community.
It also gives new members a clear sense of who is in the community and whom new members are addressing when they participate. In UTEST’s case, community members must be practicing UX professionals. We sometimes accept graduate students who are well into their research. In general, however, the application form makes clear that UTEST is not a community for novices just starting to learn about UX who do not have a professional stake in UX. Even without “I’m just curious” beginners, UTEST includes community members over a wide range of ages, interests, specialties within UX, specialties within domains, and venues of work.
The application helps new members develop a clear sense of why and how they belong.
Have rituals
Successful communities use linguistic capital (special words) or secret gestures (for example, special handshakes) as a way for people to show they are members of that community. These ritualized forms of spoken, written, or gestural communication create a sense of belonging among members. They can be critical to the longevity of the community.
In popular culture, quoting the lyrics from a popular song, for example, can show an audience to whom you are speaking that you belong to the same group as they do. It can help build relationships and establish a mutual esprit de corps.
Online communities also need rituals.
For UTEST, one of these rituals is a long tradition of making and allowing jokes and other whimsical posts on Fridays. As a rule, UTEST members are extremely serious in their tone. Wanting to avoid even the potential for misunderstanding or unintentionally giving offense, they avoid sarcasm, irony, and irreverent comments during the community’s normal conversations. On Fridays, however, members allow each other a little leeway.
Community members will frequently introduce a banal or humorous topic by saying, “Normally, I wouldn’t bring this up, but since it’s Friday…” Sometimes, when they feel the need to introduce a little irony but want to be sure not to give offense, they may write, “I know it isn’t Friday yet, but…”, signaling that the message is to be taken with a “casual Friday” interpretation.
Another UTEST ritual that comes from its sense of exclusivity and the fact that it is a closed, private community of UX professionals is that we often refer to it with tongue in cheek as “the online community that shall not be named.”
A more serious ritual is a tradition of announcing new books by community members, usually by someone other than the author of the book.
Significance
Members of a community want to feel that they have made a good choice in joining the community. They must see the community as the best way to achieve their goals within the scope they are seeking. For UTEST, these goals are to give and get information, friendship, and support with other UX professionals virtually throughout the year and across geography.
In addition to the points we have already made under remuneration, influence, and belonging, UTEST supports significance in at least these two ways:
Encourage useful, productive discussions
UTEST members engage deeply in substantive discussions, responding to questions and observations with personal analysis, fresh insights, deeper questions, and tangential topics. In stating their understanding of a request or problem, members provide a range of perspectives and a means to broaden and deepen all members’ understanding of the topic under discussion. In suggesting solutions or approaches, they are able to recount experiences and share observations that might not be available in any other forum or venue. UTEST discussions enable practitioners and researchers with diverse interests and unique backgrounds to benefit from each other’s expertise and knowledge.
Allow a wide range of topics and questions within the overall domain
Community members ask and answer a wide range of questions on many topics, including how people deal with methods, new technologies, ethical and legal issues, preferred social interactions, etc. Members share links and references to published sources, relevant personal experiences, jobs they know about, books they have found interesting, and more.
For example, as mobile devices and tablets have become more pervasive and as many community members have moved into agile environments, we have seen threads on how to shift methods for user research, design, writing, and evaluation for these new situations. As the business landscape changes, UTEST members use the community as a sounding board for questions, issues, and ideas about their roles in their organizations or their consulting practices or their universities.
What Did a Survey of UTEST Members Tell Us About the Community?
For 10 days in January 2008, the UTEST advisory council conducted a survey to find what UTEST community members felt about three issues. We have not previously reported the results of this survey outside of the UTEST community, but they are relevant in this essay.
Although the survey results are from five years ago, continued feedback from the community and email interviews we hold when someone leaves the community lead us to believe strongly that the survey results are still valid. People leave the community because changes in their personal or professional lives take them away from UX. They invariably say that they would stay in the community if it were not for those life changes, and they praise the role the community had in their professional development.
What We Asked
In the survey, which Tharon posted in an email to the community, we asked these three sets of questions:
- UTEST is intended for practicing professionals. Should we tighten the criteria we use for accepting new members and make it more difficult to be a member of UTEST? Should we exclude people new to usability, people changing careers, graduate students, and others? Or should we continue with our present policy of accepting such people?
- UTEST expects members to share their knowledge. Should we, as a community, enforce and encourage participation (and if so, how)? Or should we continue with our present policy that it’s not an issue if some members don’t post much—if at all?
- • UTEST was formed to be safe, closed, private, and exclusive. We have assumed that the safety of UTEST is jeopardized when people talk about UTEST in places where anyone can pick up the URL (for example, through their blogs or portal sites). Should we continue our present policy? Or should we consider that brief factual references to our website are not harmful?
What We Heard Overall
We received 40 analyzable responses, which represent a return rate of about 40% of community members who posted that month.
We found pretty strong consensus on the first two issues.
- It is fine to have new members. Many comments said the entry procedures as set up were fine and that the barrier to entry should not be made higher and more restrictive. UTEST should be open to all who are interested and the variety of perspectives in the community was refreshing: too homogeneous a community loses value. Some respondents were very clear that entrants to the usability field should be welcomed.
- It is fine to have lurkers. Lurking can be seen as a form of responsible group participation. Lurking should not have a pejorative association. People who listen also build the community. Some respondents said that lurking is part of a knowledge exchange mechanism. They said that if someone really doesn’t know enough to add an intelligent comment, it is okay for that person not to post. And it is good to belong to a community where people don’t feel they have to post trivial things just to be part of the conversation.
We found mixed feelings on the third issue.
- Good signal-to-noise ratio is important. Exclusivity makes the community valuable. But being secret is ironic for UTEST. Respondents realized that exclusivity makes the community valuable to belong to, and the quality of membership contributes to the good signal to noise ratio. But many questioned the policy about being so strictly discreet about the existence of UTEST.
What We Heard in Detail
Twelve major themes emerged from the responses. Within each theme, we typically found comments relating that theme to other themes. The mind map in Figure 1 shows how the themes relate to each other.
Figure 1. A mind map of the themes that emerged from our survey
Three of the themes are fairly self-descriptive:
“Not Broken”
The UTEST special policies are working and have been shown to work. Don’t change them.
“Advanced Technology”
Why not use more modern technologies to run the community, such as Second Life or a wiki?
UTEST still uses email with a digest option. Tharon actually built a trial version of the community in Drupal to test out this suggestion, but ultimately we did not change despite this theme. The change from the current push technology to a pull technology seemed to change the interaction of the community too drastically and conflicted with the “Not Broken!” theme we were hearing from members.
“Role of AC”
Respondents wrote appreciations of the facilitation that Tharon and the advisory council provide, which we believe speaks to the importance of our earlier recommendation that people starting new communities consider creating advisory groups with roles similar to those of UTEST’s council members.
We present eight of the nine other themes here in alphabetical order and briefly describe each. They each focus on one of UTEST’s features. We then cover the last theme separately because it is not so much about features of UTEST as a personal reaction from some respondents.
“Archives”
Although some respondents commented there was a lot of value in UTEST material that could be kept for reference, most respondents favored the current no-archive setup. They liked being able to take new approaches to ideas that had been discussed previously and felt that archives discourage that type of discussion.
“Entry Policies”
Respondents said that a closed list like UTEST creates a sense of belonging, enhances community spirit, and adds to the value of the list.
“Exclusivity”
Respondents appreciated that UTEST’s discussions were sharply focused on the needs of practitioners, and they appreciated that the questions that members asked in the community were often answered by some of the thought leaders in the field. Although there were a few respondents who were in favor of having liberal criteria for membership in the community, there was wide spread support for screening applicants who sought membership in the community.
“Lurkers”
Respondents said that lurking is okay. They saw a value in listening. Because members do not have to contribute if they don’t wish to, they felt that lurking keeps a good signal to noise ratio and adds value to the community.
“Secrecy (policy)”
Respondents were in favor of liberalizing the policy of not mentioning UTEST in published articles or in public. However, they understood that the more liberal we were, the greater the burden would be on reviewing applications.
Over the years, we have changed the policy somewhat. We now invite members to tell others by email or face to face about the community. We allow members to list UTEST on their résumés (without its full address). We allow members to ask Tharon and the advisory council for permission to mention UTEST in other situations.
“Signal-to-Noise Ratio”
Respondents thought the signal-to-noise ratio was good and many attributed it to the policies discussed in this essay.
“(UTEST) Special Policies”
Although UTEST’s special policies may seem to be long and look strange at first to some people, respondents said that the policies are a good summary of acceptable behavior for the community. They said that UTEST is a safe place for discussions because community members adhere to these standards of behavior.
In the years since the survey, we have revised policies to clarify them, make them more readable, and deal with new issues as they arise.
“Value”
There were many attestations to the value of UTEST in the professional lives of the members that echo Tharon’s concept of “Belonging” as an important attribute of an online community.
“Fear”
One more theme emerged from this survey. We labeled it “fear.” It was different from the other 11 themes in that it seemed to speak from the personal experiences of some community members.
Some community members said they were intimidated by the erudition, expertise, and depth of experience of others, and they preferred not to post for fear of being shown up as uninformed.
The advisory council has since been more vigilant in using offline gentlegrams to ask members who adopt an abrasive attitude to be more careful of their choice of expression and to consider the spirit of UTEST as a welcoming and safe community.
Summary of themes
To summarize: In this survey, UTEST community members said they value UTEST’s exclusivity, its signal-to-noise ratio, and the substantive conversations they have within the online community.
Some respondents particularly indicated their appreciation of the relationship between UTEST’s goals and specific policies, such as not advertising the community in public blogs and websites, not archiving messages, restricting the community to people who are involved in usability (user-centered design, user experience broadly defined), and so on. They saw these aspects of the way the UTEST community operates as making their participation in the community a valued and unique experience.
Conclusion
We hope that this brief history of UTEST and our discussion of the RIBS principles that we have used to manage the community serve well as a model to build other online communities either within companies or in public spaces. As the survey of UTEST members suggests, the RIBS principles seem to work well and to be good reasons why the community has lasted for more than 20 years when it’s estimated that only 1.3 percent of online communities are successful (Leahy 2013).
If you are a UTEST community member, we hope you agree with these survey results.
If you are an active contributor on UTEST, we thank you and appreciate your contributions to UTEST discussions.
If you have been lurking on UTEST, we hope you will now feel more comfortable and begin to contribute more.
If you are not a community member and are active as a UX practitioner or researcher, we invite you to join the community by reaching out to tharon@clemson.edu.
We plan on continuing UTEST for many years as a safe space for the UX community to ask, discuss, and share.
References
- Crane, D. (1972). Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.
- Howard, T. (2010). Design to thrive: Creating social networks and online communities that last. Waltham, MA: Morgan Kaufmann / Elsevier.
- Leahy, A. (2013, July). Why do so many communities fail? Retrieved from http://cultivate.ning.com/ning-blog/conceptualizing-a-framework-for-communities-video.
