The idiom of “don’t let the fox guard the hen house” warns us about the potential danger of giving someone responsibility for overseeing something that he or she shouldn’t be involved with, particularly when there is an inherent interest in the outcome. Unfortunately, that idiom is all too fitting in the world of user experience (UX) research and practice, especially with respect to usability testing. I have seen all too often designers who evaluate their own design, or the design agency that is responsible for evaluating their own work. This is an inherent conflict of interest that results in poor quality research and, perhaps more importantly, undermines the credibility of our profession. The good news is there is an easy fix.
Let us start by stating that I do not mean to offend anyone, especially the very talented UX researchers and designers I work with every day or the design agencies that produce world class products across every industry. I have tremendous respect for their skill and professionalism. I know that they want to produce and deliver great user experiences to their clients. But even with the best of intentions, wrong decisions about how to test the usability of those products and services are sometimes made.
Even though there has already been a fair amount written about the risk of designers evaluating their own designs, this continues to be a problem, particularly with the lean UX approach. The general consensus is that this approach is not a good idea because designers have great difficulty in maintaining objectivity. It is widely understood that it is just too hard to maintain objectivity. But, what are the risks of design agencies evaluating their own design work?
In this editorial I focus on the inherent conflict of interest that design agencies have when they are responsible for evaluating their own design work, what can be done to mitigate this problem, and the implications for the UX community. I define a design agency as a consulting firm that is hired to design (from a visual and interactive perspective) digital products.
What Do I Mean by Design Work?
As we all know, there are many different types of design work. Design work might be in the form of basic wireframes, low-fidelity prototypes, or high-fidelity prototypes. I believe that the risks with design agencies evaluating their own design work increase as the fidelity of the artifact increases. The risks are directly tied to the time and cost of iterating the design. Basic wireframes or low-fidelity prototypes are relatively easy to modify. If the major changes to the navigation or interactions need to be changed, the implications for the project timeline are minimal. However, a high-fidelity prototype or complete prototype with visual design is much more costly to modify. A tremendous amount of work has gone into designing a high-fidelity prototype, particularly if all the visual design is included. A design agency would be very reluctant to make significant changes or even to scrap a complete prototype based on a poor usability evaluation. This is especially true when there is only one usability test, and it is performed too late in the design process to get major changes into the product.
Even though I argue that design agencies should not evaluate their own design work, I think it really applies to high-fidelity prototypes or fully functional prototypes. I take no issue with design agencies who are self-evaluating basic wireframes or low-fidelity prototypes as a process to uncover potential prototype designs. Because basic wireframes or low-fidelity prototypes are relatively quick and easy to iterate, they do not pose the same risks to slowing down the project plan.
What’s the Problem?
Many design agencies claim that they can set aside any conflict of interest and run independent, objective usability evaluations and deliver unbiased results to their clients. This is a claim that I find a little dubious. I contend that the pressure to look good in the eyes of the client is simply too strong to ignore and claims of complete independence and objectivity are questionable.
So, what does this conflict of interest actually look like? How does it rear its ugly head? There are three ways that a conflict of interest manifests itself within a design agency: in the decision to evaluate, in the method(s) chosen, and in the interpretation of the findings.
All too often design agencies make the decision to not go through formal, third-party usability evaluations. It is common for agencies to cite factors such as slowing down an agile schedule or the additional costs to the project. While these concerns are understandable, they are not well founded. Some design agencies even claim that they understand the product or the target user so well that they don’t need to get feedback from the end user. Not only is this wrong, but it ultimately leads to poor designs reaching the market, which is sometimes called “marketplace testing.” Or, more to the point, we can call it a type of UX arrogance.
Once the decision to not independently evaluate a design is made, a conflict of interest can influence the testing protocol. I have seen many times that a design agency will bias the protocol towards receiving positive feedback from end users. This typically involves only showing visual designs to the end users, with little to no interaction with the prototype. Another example that comes to mind is including leading questions in the prototype test that are likely to produce positive responses, or loading the test with easy tasks. I have also observed that many decisions that go into the creation of a study protocol can easily be manipulated. For example, many design agencies typically don’t compare their proposed designs against the current design or the competition’s design. Granted there are some designs in which this may be difficult to do; however, in most cases it is not only easy to do but also highly informative. Many design agencies might argue that it doesn’t make sense to evaluate something that they know will change. But, how do you know your design is better if you don’t measure it against the competition and/or previous designs? After all, what would the agency say if their proposed design did not fare well against the current design or the competition? No doubt it would make for a difficult conversation, and usually result in going back to the drawing board and cost the design agency additional time and money.
Some design agencies’ UX researchers sometimes bias findings often without even knowing it. I have seen that UX researchers feel hesitant to criticize their own agency’s work in front of the client. For example, a UX researcher may report serious usability issues as minor, or a researcher may provide design recommendations that are based on easy implementation rather than the right design solution to optimize the user experience. After all, no one wants to give their colleagues more work, especially when the project is based on a fixed-cost. Also, a researcher may think that a meeting with the client is the time for an agency to shine—to validate the design approach and move forward with development. When the client relationship manager is in the room, it can be very difficult to report any serious issues for fear of upsetting the client and slowing down the project timeline.
Why Does This Happen?
This conflict of interest happens for three reasons. Perhaps the most obvious reason is because of the financial arrangement between the design agency and client. The client is typically paying the design agency for their services to design a new website, mobile app, piece of software, etc. on a fixed-costs basis. Any activity, usability testing or otherwise, that potentially delays the project poses a financial burden on the design agency. Usability tests typically reveal many issues, some of which might be unexpected. Some of these issues might demand significant changes to the design, or even a completely new approach. Most design agencies do not build in extra time to address these issues, nor do they have the budget to do so. Therefore, some people may feel that it would be in the best interest of the design agency to avoid usability testing or conduct testing in a way that minimizes the need for any project delays.
Another source of this conflict of interest is client perception. Clients are hiring design agencies based on their expertise. Expertise means coming up with great designs quickly and within budget. I know usability tests are an excellent way to identify problem areas with any design. But the last conversation any design agency wants to have with the client is why their own design fared so poorly in usability testing. The client might say “I thought you were the experts, that’s why we hired you, so why are our target customers having difficulty with the product?” Usability testing has the potential to expose flaws not only with the design but the design process, and even perhaps the design team as a whole. Many client relationship managers would strongly prefer to avoid this situation if at all possible.
Lastly, it is important to remember that the UX researcher wants to be seen as a “team player.” This means that the researcher doesn’t want to upset the designers, developers, or client relationship manager within their agency. Sounding the alarm on a product might have a significant negative impact not only on how others within their team see him/her, but could also potentially affect his/her performance measures.
What About Other Conflicts?
Design agencies are not the only ones susceptible to this conflict of interest. Internal UX teams can also have a conflict of interest, particularly when the research and design teams are part of the same reporting structure. Internal UX teams that are under very aggressive timelines or require funding from their stakeholders can also show similar biases in their prototype design decisions—favoring approaches that are faster and yield more positive results. Internal UX research teams might also find it hard to be critical of their design colleagues, especially if there are performance issues involved. All things considered though, the conflict of interest by internal UX teams pales in comparison to those by the design agencies.
Bigger Impact
The obvious result of these conflicts of interest is sub-optimal design and money poorly spent. I would argue that there might be a bigger, more significant impact of this conflict of interest—credibility. When we as a UX community fail to make the right decisions or bias our findings (consciously or not), we undermine the credibility of our profession as researchers and practitioners. At the end of the day, our credibility is essential. Without credibility our value is diminished: Our arguments are weakened when we make a recommendation on the best research approach or try to convince management on the importance of UX research.
As UX researchers we strive for objectivity—to ground the truth. This is the best and only way we can deliver value to our stakeholders. These conflicts of interests that I’ve presented get in the way of that goal. Once we start sacrificing our objectivity, our value is diminished, and our presence might be questioned. I believe that the lack of objectivity is more of a job threat than standing behind our principles. By not upholding our principles, we stand on a very slippery slope, and one we should desperately avoid.
A Simple Solution
It’s not all doom and gloom, there is a simple solution. Design agencies can (and should) outsource the evaluation of a design to another team that specializes in UX research and evaluation. This sends a very strong signal to the client about an agency’s confidence and objectivity related to the design. Essentially the design agency is telling the client “Don’t’ take our word for it; let’s get objective data from an independent third party.” I have seen that this message is very effective, especially during the early scoping discussions with the client. I believe it is easier to win work when the recommendation is made for an independent, third-party evaluation. Not only does outsourcing the evaluation produce better designs in the end, but it also may translate to an increase in revenue for the design agency because the agency that chooses to be objective and honest, is the agency that clients will choose.
I expect that some UX researchers and designers may not agree with my position in this editorial. That is fine, and what I would expect. I hope it opens up a dialogue about the best way to deliver long-term value to our clients and mature our profession. In the end we all want to create great designs. Let’s identify those things that get in our way and move forward. So, yes, it’s time to stop having the fox guard the usability lab.
