Software penetration testing has become a common practice in software companies. Penetration testers apply exploratory testing techniques to find vulnerabilities, giving developers feedback on the results of their security efforts—or lack thereof. This immediate benefit is mostly uncontested, although it comes with the caveat that testers find only a fraction of the present vulnerabilities and their work is rather costly.

Security experts commonly recommend that developers should respond to the results of a penetration test not merely by fixing the specific reported vulnerabilities, but rather analyze and mitigate the root causes of these vulnerabilities. Just as commonly, security experts bemoan that this does not seem to happen in practice. Is this true and what prevents penetration testing from having an impact beyond defect fixing?

Studying Software Developers

We studied this question by observing a product group of a software company over the course of a year. The company had hired security consultants to test one of its products for security defects and subsequently train the developers of this product. We wanted to know how effective this intervention was as a trigger for change in development. To this end we conducted two questionnaires, observed the training workshop, analyzed the contents of the group’s wiki and issue tracker, and interviewed developers and their managers.

The product group in our study comprised several Scrum teams, development managers, and product management. Scrum is a management framework for agile software development. Scrum defines three roles—Development Team, Product Owner, and Scrum Master—and artifacts and ceremonies for their collaboration. The Development Team is responsible for and has authority over technical matters and development work; the Product Owner is responsible for requirements and priorities; and the Scrum Master facilitates and moderates their collaboration.

The participants of our study appreciated the penetration test results as feedback and the training workshop as an opportunity to learn. They managed to address the particular vulnerabilities reported by the consultants. They also felt that security needed more attention in their development work. Yet, after addressing the vulnerabilities uncovered by the penetration test, they returned to their familiar ways of working without lasting change.

Analyzing our observations through the lens of organizational routines, we found three key factors inhibiting change in response to the penetration test and training: successful application of existing routines, the organizational structure of roles and responsibilities, and the overall security posture and attitude of the company.

(1) Existing Routines

To address the immediate findings of the penetration test, our developers used an existing bug-fixing and stabilization routine. Defect reports arrive asynchronously and sometimes require quick response; developers therefore commonly dedicate some—variable—percentage of their working time to defect fixing in response to reports. The penetration test fed the team dozens of new defects at once, but developers hat a routine to deal with it. Moreover, management tracked the number of open defects, so that the sudden increase raised attention and created pressure on the team to get this number down.

Feature development, on the other hand—where change would have to occur—remained mostly unaffected. Feature development followed the process of Scrum and the penetration test neither had a specific impact here nor did it feed requests or ideas into this routine.

(2) Organizational Structure

Following the ideas of Scrum and agile development, a strong division of labor and responsibilities characterized the organizational structure in the setting of our study. Product management and product owners were responsible for the direction of the development work, whereas development teams enjoyed a certain autonomy in technical questions. This structure worked as a social contract: managers expected developers to take care of security as a matter of quality and developers were keen to deliver the features requested by management. However, the penetration test had little impact on the manager’s priorities beyond the pressure to reduce the number of open defects. The developers thus found themselves in a situation where security was not explicitly required and additional security work could not be justified.

(3) Business Role of Security

Finally, security had limited perceived importance for the business of the studied company, which thus far had not experienced any public security disaster and did not actively sell security. The company therefore lacked a security narrative that could have been used to justify security efforts beyond defect fixing. This, together with the inherent low visibility of security and insecurity, shaped priorities. Product managers knew that features sell their products—new features are easy to show and explain, whereas security improvements are not. Security was perceived as contributing little to the success of the product and the company, making it a low-priority requirement.

Our study points to some of the complexities of managing software development and of triggering change by interventions. While it would be tempting to assign blame to a single factors, such as agile development or negligent management, the problem is really more complicated. Organizational structures and routines exist and they are shaped by business needs. Scrum, for example, is highly beneficial for the studied company. One might even ask whether the company’s dealing with security is a problem in the first place. Are they perhaps doing the right thing for their market and customers?

—

Our Paper:

A. Poller, L. Kocksch, S. Türpe, F. A. Epp; K. Kinder-Kurlanda: Can Security Become a Routine? A Study of Organizational Change in an Agile Software Development Group. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW’17), February 25–March 1, 2017, Portland, OR, USA.
DOI: 10.1145/2998181.2998191