Is it Realistic to Shift Testing Left?

In a recent Applause webinar, Vodafone’s Lead UAT Manager, James Mortlock, explained how Vodafone made the move to Agile and shifted testing left. Together with Applause’s UK Telco Practice Lead, Agim Isufaj, James discusses what it takes to make shift-left testing a reality. Here’s what they had to say.

Agim: Loads of companies have the aspiration to change and are looking for ways to shift not just testing, but their entire SDLC left. They ask themselves: if we are at point A and we want to get to point Z, how do we go about it? Before we get onto that, something you mention a lot at the start of the webinar is the importance of the customer perspective in testing. How did you drive the customer-centric approach in your teams given the scale of Vodafone?

James: How to incorporate the customer voice in testing is a question I constantly get asked, even within Vodafone. For me, it’s about recruitment and team ethos. You’re not going out there looking for hardcore test engineers or automation gurus. You’re looking for people with the aptitude to see different points of view… empathy, basically, and honesty. I tend to look for people who have been involved in UX, or completely different environments like gaming. In terms of ethos: We are not in engineering, we are in UAT. We are there for the users, like guardians of the user experience.

Agim: I bet there are lots of engineers listening to this who are thinking: come on, the engineering piece is important because you need to ensure features are functionally sound. What would you say to those people?

James: My role at Vodafone is the first role I have ever had where I am not an engineer. If you snapped me in half, I would say engineer on the inside. I started at the bottom as a technician, trained formally, studied defect methodology, kept my Kepner Tregoe, if anyone remembers that! There is value in this stuff. However, testing is not the silver bullet that everyone thinks it is. Quality is the silver bullet. A lot of the functional piece can be covered by really good foundations in development. It’s making sure you have a targeted and robust automation pack and having additional testing phases where you do something different, because you won’t catch anything always running the same tests. Functional is the baseline — UAT goes further.

Agim: You talked a lot about the ideal scenario. If you designed a transformational software development process from start to finish, fully shifting left with Agile embedded, how would you structure that?

James: It’s circumstantial. I can’t give you an off-the-shelf solution that is going to mean great software, fewer bugs and loads of money piling in. But you can have a toolbox. Sometimes you will need the sledgehammer and sometimes… a small hammer! Waterfall is like hammering a screw into the wall: it will work, but it won’t look great. It’s about having all the tools you could use and then tailoring them to delivering a great customer experience. You want to make it as lean as possible while supporting the organization with any governance needs. It’s a balancing act between not taking so much out of the toolbox that it becomes a joke.

Agim: Thinking back to when you started this whole journey… What was the trigger that made you realize that things could be better and want to improve them?

James: When I was working in the digital frontend department on the handset team, we were doing gigantic releases. I was an Agile evangelist from a previous role and knew there was a much better way to do things. The tipping point was when we started releasing much more frequently and I realized that my team was still shouting about and celebrating releases even when we were releasing multiple times per day, as opposed to every six months. I thought: yeah, we need to do more of this.

Agim: So that was the moment you knew you had hit on something. What did that do for the other teams?

James: Oh, it just made it into a competition. I am aggressively competitive. My team would bust the limit of emojis you can use to reply to a release post as soon as possible. (As an aside: not that it was a real competition, but my team always won on that). Other teams started tracking how many releases they were doing per week, month, year. It was the proof we needed that you can pull all of the governance and over-paperworking out and deliver something really lean.

Agim: Something I imagine people would scream right now if they could is: just because you are releasing so much, doesn’t mean the quality is there. We can all reach a certain release cadence. How did you maintain quality?

James: It’s about the focus. It’s not about delivering faster, it’s about delivering faster to a high level of quality. The whole department needs to be on board that it’s not about hitting a magic number, but delivering a feature without compromising.

Agim: Most companies rely on a CSAT score, or some kind of customer satisfaction metric, to do a sanity check that the quality is still maintained. How do you measure quality?

James: You need to use classic metrics like error rates in production, feedback from users, and most expensive bugs. In terms of maintaining quality, we also make sure that there is someone keeping an eye out at each deployment stage, because we don’t want to know that the last time we touch a feature is in development and the next time we see it is in production. The voice of the customer needs to be at each deployment stage.

Agim: I don’t mean to plug Applause, but do you see this as the opportunity for crowdtesting to come in and speed up delivery while including the voice of the customer?

James: There are so many uses for crowdtesting depending on the size of your organization. Crowdtesting can be exceptionally scalable, up or down, when you need it. The most advantage you get is the point of view. There are so many different ways you can use it. You can carry out UAT in sprint and then get a final perspective from users, or, transversely, use the testers to rip apart the journey right at the start, before you have built the wireframes.

Agim: To piggyback off of that, you mentioned automation. It’s a hot topic and something Applause has seen growing requirements for at different stages of the development process. The challenge we’ve seen is that sometimes people deploy automation solutions for the sake of it. What role does automation play in your team?

James: My first question is always why are you doing automation. Is it because the business has asked for it? Or is there a reason? Automation can often take over something for you that a person is already doing. If you remove that workload for them, what additional value can that person add? What additional pieces can you bring in, like more user testing, UX research or new tooling? It gives you time to experiment with the value-add stuff that nobody has time for. The exponential benefit is huge.

Agim: You mentioned the importance of recruitment at the start of this webinar, ie. bringing in the right people with the right mindset. A big challenge companies of Vodafone’s size face is that they have an internal workforce that does not have the internal capability. What do you recommend they do from a resource perspective?

James: I’m going to use another buzzword: the growth mindset. It’s about adding “yet” to the end of sentences you think are impossible.

Agim: An argument people may have is: but you’re Vodafone — you’ve got endless resources. Am I right in thinking that if you could do this all over again, you’d start with a much smaller team?

James: Definitely. The advantage with smaller companies and teams is that you tend to be more dynamic. You can change and experiment. It’s not all about budget, it’s about the growth mindset and the failing fast mindset. My favorite analogy is two people pushing a cart with square wheels and another person says: guys, I have an idea. They produce two round wheels and the guys pushing the cart say: we’re too busy. It doesn’t cost anything to stop and try something else that might actually make you go faster.

Watch the full webinar with Vodafone below.