Requirements engineering in an agile environment.

Marcel: Hello and welcome to a new episode of Querverlinkt. Maybe some of you already know me from previous episodes. My name is Marcel, I am one of the heads of software development project managers at consistec. Today I've brought my colleague Alexandra Bernhard along to talk to me. Alexandra, please introduce yourself briefly. 

Alexandra: Yes. Hello, my name is Alexandra Bernhard. I've been at consistec for two and a half years. Before that I worked in software development and requirements engineering and now that's my current focus.

Marcel: Alexandra, good to have you here. Today's episode is about 'requirements engineering in an agile environment' and you have experience in the role of 'requirements engineer'. It's the process for defining, documenting and changing requirements. Explain how that relates to the agile environment, whether it is even possible to do requirements engineering in such an environment.

Alexandra: Exactly. First of all, 'requirements engineering' itself is not just about documentation, but also about eliciting requirements. That means first of all recording the customer's requirements and discussing methods and then writing them down. And then, very importantly, to validate them. That means talking to the different stakeholders and finding compromises if, for example, requirements contradict each other. Among other things, that's a big part of requirements management. Management is about things like prioritising, assessing a risk and later, when requirements change, making the whole thing comprehensible and adapting it overall.

Marcel: How does software that you want to develop differ from a product that is developed on a production line, such as a sofa?

Alexandra: What is often different about a software project is that it has a bit of an innovative character. That means I might not know exactly what I want at the beginning, I might just have an idea. And then I work my way forward. I think this innovative character is the main difference. It is not something that has already been produced a thousand times, but simply something new. It's due to the vagueness of the requirements. And it only gets better the further you get on and the further you are in the development. Because then the idea simply works better. And that's the special thing about the agile approach, that I simply know that I don't know everything exactly at the beginning. I don't define anything in detail if I don't know how it will work at all. The closer it gets to development, the more precisely I know how I want it. And then I specify it in more detail. 

Marcel: It's the environment alone. You have existing systems and the software development has to work with these existing systems. That means you don't have a software development from scratch, like with the example of the sofa. I can now specify exactly how it should look, because I always have some problems, conditions, constraints that I don't know in advance. They only arise and are determined when the developer reaches a certain point. And then the whole thing has to be rolled up. Then this condition, which is noticed in advance - this problem - has to be fed back to the requirements manager in order to re-evaluate it. To check whether it is still compatible with the goals.

Alexandra: That is also an important point. That is something that has to be done very early in the process, that you think about what your constraints are. Do I have restrictions? For example, in the form of systems that should work together with my systems. Perhaps there are already constraints that come from the customer or perhaps from the team. For example, we want to develop with Java, then that would also be a constraint, that is important to consider. You should include that very early in the project.

Marcel: Exactly. But practice has often shown that these conditions can rarely be defined and recorded from the beginning. And that this often becomes apparent during the work, during the software development. Especially to the developer. He notices it at a late stage and then you can incorporate the whole thing again.

Alexandra: You do a lot to keep track of everything. But you are not aware of everything. You have to write things down, so that you don't forget them. Even if you realise at the beginning that you have such and such constraints and then write them down immediately. Because that is often something that is forgotten and then it can be very difficult later if you haven't taken it into account. Then nothing fits anymore. 

Marcel: I know the project management method 'Scrum' from practice. And I know the agile values. And what degree of requirements engineering is not mentioned there? How is it in practice? Is requirements engineering simply left out or how is it formed, how is it presented?

Alexandra: That is a misunderstanding that sometimes occurs in practice, that people think in principle they have already given everything they need through Scrum. In principle, you have different possibilities to break down your requirements into epics, user stories and so on. But this is by no means sufficient, or rather it should be supplemented. Also through classical methods or adapted classical methods.

Marcel: You just used the terms epics, user stories and task. Can you explain them briefly?

Alexandra: Well, the idea is that you go from coarse requirements and a coarse granularity to finer and finer. This means that at the top level I have the epics, which can contain several user stories. And user stories in turn can be broken down into tasks. And if you need another level in between, you still have features and a bracket to form the user stories accordingly.

Marcel: You just talked about misunderstandings. One misunderstanding was that in practice documentation is not considered as important in the agile environment. Is that the case?

Alexandra: Yes, one of the guiding principles is communication is more important than documentation. That doesn't mean that documentation is not important, but that the communication level is even more important.

Marcel: And the second misunderstanding I have now interpreted is that the developers expect epics, user stories and tasks with a certain level of detail. And that sounds like it doesn't fit the idea of requirements engineering in the agile environment. How is that meant exactly?

Alexandra: It is the case that in Scrum, for example, the different levels of detail are given by epics, as the coarsest element, down to the tasks, in order to zoom in and out of requirements accordingly. But this is also how the developers, or in requirements management, proceed relatively intuitively. And they use other possibilities for documentation. These include, for example, process modelling, which is first needed to get an overview of the subject matter, or how the later process should look in the software. Down to the UI design in the form of wireframes and the software design is a part of that as well. Or context models, in which you can simply see how the developed software interacts with the environment.

Marcel: Exactly. So documentation is a central artefact in requirements engineering. That's how I understand it now. It gives you an insight into different perspectives, it gives you an overview of the big picture. You also mentioned examples of different types of documentation. What is the purpose of this, or rather I now know-. I have experienced a lot that many people expect the documentation to be done in order to have proof of what was recorded afterwards. They say: "This has now been requested. This is in the budget. And that has to be developed. Are there other purposes and goals of documentation?

Alexandra: Yes, unfortunately, the focus is sometimes mainly on being able to initially estimate the requirement. But there are many purposes for the requirements definition. For example, legal purposes, when I actually have to secure myself. That is one thing. But you also want to facilitate later maintenance, for example. That means it also helps the software developers if the system is documented accordingly. And initially, for example, you might want to bring everyone up to a common standard first and would spend more time on documentation than you might do later. This can then be designed very individually. So with diagrams or completely freely. Anything that helps understanding is useful.

Marcel: Well, I think what should be clear in any case is that documentation cannot be all-encompassing. Now, with the client there is a given budget and a fixed time frame is agreed upon. And in the end, the client expects the requirements to be estimated. How do you see this in the agile environment?

Alexandra: In the agile environment, the approach is usually a little different or the focus and the idea behind it is different. The idea is that you first formulate a vision. That's the most basic thing, an overarching vision and then you derive your goals. The goals must then be measurable in some form, for example, so that you can determine whether you have actually achieved them. And well understandable. And the goals then first serve as guard rails for the definition of requirements. And at the end of the day, it is more important that the customer achieves his business goals. And it is more important for software development to support these rather than to adhere exactly to requirements that were originally defined. Of course, you don't keep the buisness goals completely undefined, you would define them initially so that you have an idea, but the objective is fundamentally different. 

Marcel: So do I understand that correctly? It doesn't matter whether it's in a classic project environment or in an agile one. The time frame and the budget can be fixed in both cases. The big difference is then in the requirements. In the classic environment, the requirements are fixed from the beginning, including the number of requirements. And in the agile environment, one is more oriented towards the goals. There can be an initial set of requirements, but you also have the option of dropping requirements if the budget is exhausted or the time frame cannot be met. If the overall goal can still be achieved. Is that how I see it?

Alexandra: Unfortunately, even in the agile context, it is usually the case that the customer actually wants to fix a budget. But you have more levers than in the classic approach. In the agile approach, you would rather go and agree on the business goals, the fulfilment of these goals and fixate less on the requirements. If the budget really is a bottleneck, one or the other requirement will be omitted. Or maybe even swap out requirements to get closer to the business goal and to turn less on the budget screw and less on the quality. Quality is the thing that usually suffers a lot here, because most of the time you haven't defined exactly how qualitative you want it to be. And that leads to customer satisfaction in the end. This means that the agile approach helps to stay closer to the budget on the one hand and to maintain high quality on the other. 

Marcel: I would like to emphasise quality in particular: There is a difference between internal and external quality. And the external quality is what the customer sees, what is easy for him to understand. Like, for example, the user mask, which the end user then sees, or perhaps some speeds. He sees the reaction time. But what we as software developers have to pay attention to is the inner quality. That means we have to make sure that the software is maintenance-friendly or that certain things are taken into account, such as architecture design, which are not so obvious to the customer.

Alexandra: Exactly. You can't really talk your way out of it as a software developer in this day and age. There are now a lot of standards and it is actually expected and assumed that they are adhered to accordingly. If you haven't already fixed it in the contract anyway. So it's very important nowadays.

Marcel: Now you mentioned that in the agile environment it is more important to define the overall business goals and that you can also drop requirements when the budget is tight or the timeframe comes to an end. I can imagine that this leads to displeasure with the client. That he is not happy at all to make such compromises. What is that like?

Alexandra: Well, mutual trust is really important here. It also helps a lot if you already know each other a bit better in a customer-supplier relationship. Or it also depends on whether the customer has already been involved in agile projects and knows the procedure a bit. Most of the time, there is the fear that perhaps no attention is paid to actually pursuing the business goal. On the other hand, it is important for the software developer or the software company, to rely on the customer. Because if, during the project, the customer comes up with thousands of new feature requests and practically wants his software to be gold-plated, you're a bit lost if that hasn't been worked out in detail.

Marcel: Now I can imagine that a customer comes in with the motto "Trust is good, but control is better". But I think there's a bit of-.  Most customers don't understand that especially in an agile environment the possibility of control is greater, because the customer has the possibility to look at the content early on, because it is an iterative process, and to intervene in time, if there are problems or faulty implementations. It's often not seen that way.

Alexandra: Exactly. Actually, the big advantage is that the sprint reviews give the customer the opportunity to get really concrete feedback. In other words, the software developers receive concrete feedback. And that the customer can see what has been developed for him. That means they can say whether it's going in the right direction, whether something is still missing, whether something should be changed again. So he actually has the means to change a lot. And that can be supplemented again and again. It is important that the client is very close to the development process. And in practice that isn't the case as much as one would like it to be.

Marcel: Exactly. Not only in the development process. The customer should be involved in the requirements engineering documentation process, even iteratively from the very beginning. The documents have to be validated, they have to be checked, they have to be agreed upon. And I also think in large projects that involve large business processes, business logic, perhaps also portals that are built, where a lot of customer interaction take place that practice is necessary. It is important that the customer is really involved from the beginning, starting with the goals. And then he is continuously involved and thus also has control over it.

Alexandra: Yes. So it is actually a great advantage for the client that he can actually see what is happening. It's always difficult to actually have an overall view of everything. That's why it helps to have the requirement described again at a higher level, so that the customer doesn't just see: "Hey, this was actually done in detail", but that he can see that, somewhere in the overall context, everything is going in the right direction. So here, too, you can use requirements methods or documentation, to support this even more.

Marcel: Earlier you talked a lot about the goals, that should be agreed with the customer. Can you be a bit more specific about what this means, or what it looks like? I have a hard time hearing the difference between goals and requirements and maybe visions.

Alexandra: Again, it's basically a matter of first getting from the rough to the fine. So first of all from a coarse granularity, also from the business level, what do I actually want to achieve, there I have the vision as a superordinate element. Then, on the next level, the goals, which are then measurable. So that would be a criterion that is important for defining goals. They should be clearly defined so that they can be understood somewhere. One goal should be achievable somewhere, because if I define something that I can't achieve at all-. Is it realistic? And I have to know by when I want to have achieved it. And so when I have set these goals, I have my guard rails. And when I set requirements, for example, then I look at is it actually achieved? Have all the goals been achieved? And if not, I would discard such a requirement if necessary.

Marcel: Now I can imagine that in practice you often have to deal with old systems. The user or the customer already works in certain systems, in existing systems. And there are already digital processes, semi-digital, semi-automated processes, manual processes. Certain tools are already used. And I can imagine that there are often misunderstandings when it comes to achieving and defining the goals. That the customer might want to stick to the tried and tested.

Alexandra: It often happens that the client already has very detailed ideas. Maybe how the masks should look. Depending on whether the customer is a bit technically experienced, this sometimes goes down to the database and then it is the task of the requirements engineering, or the requirements manager, to question this and then say: "Let's go back one more level. Let's first look at the broad picture and then break it down again into what the goal actually is. Sometimes there are better or other solutions that one might not have thought of in the first step, because when one is in one's own field of expertise, one already has a lot of experience. On the other hand, it is very important what the experts can provide as information. It is important to get to the core of the whole thing.

Marcel: That is a bit of an art in requirements management, because the technical expert speaks from his own point of view. He talks about what he is used to, what he knows. Also what he already knows. And then we have to support the whole thing across many departments within the framework of digitalisation. And that's a bit different than only the individual seeing his goal, but they dont see the big picture.

Alexandra: Exactly. So it is part of the validation of the requirements that you address such things and discuss them together with the stakeholders. And sometimes you have to prioritise between different requirements. It really helps to question the why. Why should this be implemented in such and such a way? If it is very concrete right from the start, it helps to ask again.

Marcel: Exactly. Now it is so often the case that a customer wants to have a quicker implementation and skips the requirements management and then it often comes to problems with developers. And then there is a lack of clarity and understanding of the requirement. Now the literature talks about a 'sprint zero'. Can you briefly tell us something about that?

Alexandra: To start directly with Scrum without having clarified the framework when you start a new project actually leads to the fact that the first sprint is probably very messed up. And it doesn't look that great to the outside world. That means in reality we would first clarify the framework. This starts with the working conditions, or you have to clarify whether employees still need training. All the way down to designing the system.

Marcel: It can happen that a 'sprint zero' is started and afterwards you come to the conclusion that it's not ready yet, we actually have to do more requirements engineering to be able to start with the implementation. 

Alexandra: That could happen and then you have to think about how you can counteract it in the early phase. The good thing is that you will recognise it very early before the damage is already there or has intensified.

Marcel: You have already gained a lot of experience in the field of requirements engineering. What can you generally recommend to customers who have no idea and no knowledge of larger software projects and have never done anything like this before? What are your two or three tips that you would give them to get off to the right start?

Alexandra: First of all, it is important, or rather extremely helpful, if the developers are very close to the technical processes or to the customers. And that they understand what should actually be implemented, no matter how much investment you have to make at the beginning. Especially at the beginning of a project, there is a great lack of clarity in the definition of requirements, because the developer does not know the domain well or does not know the specific customer. And that's why it's so important that the customer accompanies the process and also gives feedback again and again. And because you don't yet know what you'll actually achieve in the implementation of the requirements. You first have to record the requirements in detail and then work on them with the customer when you implement them. Otherwise you have actually done all the work for nothing, everyone has to-.

Marcel: Exactly. This also brings to mind the beautiful term "domain-driven design", which helps developers to understand the business logic of the customer's business processes. I would like to refer to the podcast again. We have an episode called: "Domain-driven design"

Alexandra: So it is very important to clarify initially: who are the relevant stakeholders? Who provides information on requirements in the first place? Who needs to be considered later? Who are the actual users? You have to clarify these things so that you don't forget anyone. If someone is not directly involved in the project, you should at least record this in the form of personas in some way. And then think about it again later so that you don't completely leave this person out. 

Marcel: We are now slowly coming to the end. At the end, I would be interested to know what you would want from a client ideally?

Alexandra: Of course, that they are very close to the development project and provide as much input as they can. That active communication takes place. A bit of a partnership is also very important, that you have a certain basis of trust, that the other person doesn't implement nonsense for you. Developers would like to understand-. At least the developers in my environment would like to understand what they are developing. That's why it's particularly important.

Marcel: That's exactly why developers work in sprints. A sprint is a fixed period in which the developers implement the software. And at the end of a sprint, there are always review meetings in which the product that is being created is evaluated again, to see if we are still working on the goal that we defined at the beginning. And then the client can take a look at that.

Alexandra: Maybe an important aspect is to be patient. I mean that you understand that when you are new to the project or maybe you don't know the domain that well, you have to question accordingly, so that the solution that comes out at the end is actually good. It also depends on what the client needs at that point. You always have to ask: why is that the case? Why do you want to go exactly that way? And also look at alternatives and that is the most valuable thing, if you get the time.

Marcel: Maybe we can conclude by saying what exactly the requirements engineering process is. We have now established that there is no simple recipe, like from a recipe book. You can't just follow the recipe and in the end the project will work because you followed it. It just varies from project to project. You really have to be in the process and depending on what the problem is, you have to adapt the processes.

Alexandra: Exactly. That's very important, that in principle-. The procedure cannot be a recipe, because the environment can be very different.

Marcel: Now we have reached the end. Alexandra, thank you for your participation and the nice conversation and the time you took. Thank you very much.

Alexandra: Thank you as well. 

Marcel: Bye.

Alexandra: Bye.

So that's it again for today. We hope you enjoyed the topic and that you could learn something. As always, you can find further links to the current episode in the show notes. And if you are interested in the wonderful world of software development, we would be happy if you subscribed to us. For our episode on the seventh of July, we have consistec CEO Dr. Thomas Sinnwell and a very special guest in store for you. 

We look forward to seeing you again!