Productivity measurement in Agile teams – why is it so difficult?
Many organizations have moved from traditional methods of project management (waterfall) toward agile and or DevOps approaches.
Agile software development empowers teams and puts the product to be (further) developed centrally instead of having a project-like focus. This has led to less grip and less control on a management level. The main reasons for this are:
- it has become harder to measure productivity, mostly because of the adoption of story points on a team level.
- the inability of many organizations to implement a standardized method to use on a higher level for productivity measurement, productivity improvement, benchmarking, long term estimation and contract KPI’s.
In this short paper, it is explained why many organizations have lost grip on these crucial activities after transforming to the ‘Agile world’. At the end of the paper, as a recap a few often-heard myths are “busted”.
Productivity is one of the most important concepts in economic science and in the business world.
Productivity measurement and improvement is an important activity in most industries, often crucial for their survival.
It is very important for organizations to understand their productivity against industry peers (benchmarking). However, organizations engaging in application development (AD) usually don’t objectively measure productivity. They are not able to demonstrate their productivity and they are not able to understand their capabilities against industry peers.
Customer organizations often don’t know it is possible to measure supplier productivity. They don’t realise hat they can actually use productivity in their supplier selection process. They outsource Application Development based on rate cards and time and material, completely ignoring the concept of productivity.
Why is productivity measurement so important in all industries, but not in the software development industry? The answer is: it’s hard to measure… or at least it is perceived to be hard.
Why is it hard to measure productivity?
Productivity is universally defined as Output / Input.
Input is usually measured by the number of effort hours per discipline that was spent on the realization of a new development project, release or (series of) sprints.
As most organizations use effort administration systems to log effort hours and the associated activities, this part is usually not that hard. There may be some discussion regarding the activities in scope of the measurement and the way overtime is handled in the system, but this is usually not really an issue and easy to deal with. So, the input part of the equation is easy to deal with.
The hard part is measuring the output. How much software has the team delivered in the new development project, release or (series of) sprints? This is difficult, as software is intangible and therefore not easy to measure in a standardized way. However, many different attempts have been made to measure output.
Through time, a lot of different measures have been proposed and used in the industry. A few examples:
Lines of code (Number of Lines of code produced)
- This may look like a good measure, but in fact it’s an industry bad practice. There is no universal standard for Line of code and different code counters result in different numbers when measuring the same code.
- It’s easy to manipulate. If you outsource AD based on price per LOC you know one thing for sure: You will get a lot of lines of code!
Subjective method to assign a number of usecase points to usecases, based on the number of actors and flows in a usecase. Especially the determination of the Technical Complexity Factor and the Environmental factor multipliers are highly subjective.
Method proposed by IFPUG to measure the non-functional requirements of a software project. Not standardized, and does not measure all non-functional requirements that are considered to be cost drivers.
Widely used in software development teams that use an agile software development method.
This is not really a size measure, but an effort measure. It’s a subjective method that is very suitable to estimate the number of sprint backlog items that the team should be able to realize in a sprint. But this is where its use ends.
Story points are very subjective and metrics based on story points cannot be used to compare between teams, departments or organizations. This method is not suitable for the estimation of a longer period than 1 to 1,5 months. It does not help to manage a contract and it does not provide the possibility to benchmark. In contracts where metrics based on story points are used, a concept named story point inflation has been observed many times.
The industry is still struggling to measure the output of software development in a standardized way.
The need for standards
The main problem is that the industry needs a standardized, objective, repeatable, verifiable way to measure the output of Application Development, preferably a method that can be applied easily or that can be automated.
Until now, the international ISO/IEC standard for functional size measurement is the only useful standard to measure the output of software development in an objective, repeatable, verifiable and therefore defensible way.
IFPUG, Nesma and COSMIC methods all comply to this standard and maintain standards of their own. These methods can usually be applied easily in most agile teams as they can measure user stories without a problem and don’t cost a lot of time or money. However, there are several reasons why the use of function points is not popular in agile teams:
- Function Point Analysis, whether it is IFPUG, Nesma or COSMIC, is a very specialist expertise and many organizations don’t have knowledge to understand the value of function point analysis or the skills and expertise to carry out this type of analysis.
- Function point analysis is perceived to be expensive, while in fact recent studies and experiences of many project managers/Scrum masters is that it’s very easy and quick to measure (COSMIC) function points based on user stories.
- Recent studies show that estimation based on (COSMIC) function points are more accurate than studies based on story points.
- However, agile teams often consider function points to be an activity of the ‘old traditional world’ that should not have a place in the modern world. However, function points is a standard for the universal concept of size. Just like the meter or the foot, size is independent from the development methodology, materials used or other non-functional requirements.
As agile teams use Story points, they don’t see the need to use other measures and they have somehow convinced management that this is the way to go. However, management now lacks the ability to control the functionality delivered and they are not able to understand the team capabilities compared to the outside world.
Thus, crucial activities like productivity measurement, productivity improvement and benchmarking are not carried out any longer, which poses a big risk to many organizations.
Recapturing, just a few myths are busted in the next table:
|Velocity burn-down charts show the progress of the team in an objective way||As story points is not a standard, they can be manipulated easily, especially when used in contract KPI’s. This is usually referred to as the concept of Story Point Inflation.|
|Function Points were suitable to measure Cobol software running on mainframe computers, back in the eighties.||Function Point Analysis is an ISO/IEC standard to measure the functionality that software provides to the user, regardless the technical environment and regardless the non-functional requirements. Just like the meter was used hundreds of years ago and is still used to measure length, function points can always be used to measure the functionality provided by software to the users.|
|It’s expensive to use function points and the accuracy is low.||The high-level variant of function points (Nesma or COSMIC) can easily be learned in one or two days by Scrum masters, requirements analysts or other team members that understand functional requirements. FPA costs less time per sprint than the sprint planning sessions and produce more accurate and reliable results.|
|Measuring function points should not be done in an agile team, as it is considered waste (in Lean terms).||For the team, FPA may be considered waste, as story points metrics give them enough grip on performance on the team level. Management however needs grip on budgets, contracts, performance and need to make decisions on team size. As it is impossible to base this on story point metrics, FPA is definitely not waste on a management level.|
|Even if we have the function points, there is no data available to compare to and it will take a long time to build up metrics.||The ISBSG repository has data of over 8000 Application Development projects, releases and sprints in Excel format to start with. As sprints are usually quite short (2 or 3 weeks) building up data will go quite fast.|