Developers Are Not a Cost Center

For far too long the security industry has treated developers the same way they treat the security team, and this is a mistake. This is not really the fault of security teams around the world but rather security vendors. They have taken a paradigm that has existed for a long time and copied and pasted it on developers. This paradigm is “We the vendor will find bad stuff and you the customer can figure it out from there”. In many cases that is exactly what you want, but I will argue that’s not what you want for developers.

Why Developers Are Not a Cost Center
No one can deny that we have gone through an explosion of growth of all kinds of software and software enabled businesses. Behind each business tool that we rely on, behind each app on our phones, and behind countless companies that provide services that make our modern world convenient is a team of developers. These developers must move quickly to fix issues, fight off competitors, optimize for cost and performance, and perform a million other tasks. If you have ever worked in this environment, you know that developers are one of the core value generators of the business. As such, saving them just a few percentage points of time or giving them productivity in any way drives actual business value to the company. This is the fundamental reason developers CAN NOT be seen as a cost center for any business that has software as a core part of its offering. 

Incentives Create Behavior
For the longest time, any security solution in the AppSec space was purely a detection solution. In order to understand the legacy AppSec space, it is important to understand what sells for these tools: finding as much as possible! I have been in the detection space for over 20 years and the sale usually comes down to an assumption that the vendor which finds the most is the best. This creates an incentive for each vendor to optimize as much as possible on finding more, even at the cost of accuracy. These vendors also do not really have the resources to understand each company's nuance and unique tech stack. This breadth first approach is useful for sales but when it comes to operationalizing the tools much of the work ends up falling on developers.

It is Not the Security Teams Fault
None of what I have said so far is the security team’s fault. The size disparity between security and development means there will often be 100 developers for every AppSec engineer. To make things even more complicated, AppSec engineers usually support teams that use different languages, tech stacks, ci/cd pipelines, databases, and various frameworks. As there is no way AppSec engineers can support up to 100 developers with individualized help, they need to rely on tooling. As mentioned earlier, the tooling is designed to detect as much as possible at the cost of being accurate. The team who purchased the tool is usually the AppSec team, whose goal is to help the company manage risk. AppSec teams generally think these tools will be helpful to developers!

SOC vs Developer Workflow
The Security Operations Center (SOC) for most organizations is seen as a cost center. It is something that is crucial to a business and you need to pay top dollar to get the best one you can afford. The SOC will help protect you from breaches and help you respond to one if it ever happens. In general, a company will have a lot of security tooling around network, endpoint, and cloud detections. These detections flow into a SOC team, which is usually staffed in tiers. Tier 1 usually sees all the tickets and decides if things should get escalated or acted on. To facilitate the workflow, there is a lot of automation and tooling to give each SOC analyst as much context as possible to help make decisions. If there are ways to completely automate a task, those are always preferred. The key takeaway is that a company must pay for a SOC team and their job is to operationalize the SOC and take the correct actions.

For developers, the workflow is completely different. Since most developers work for a company that sells a software enabled product, their job is to write code that the company will turn around and sell to customers. If a developer must do security related tasks, it is taking away opportunities to write code that can produce value for the business. With this in mind, do you think developers should be looking through noisy tickets to figure out which are real security issues and which are not? Even worse, consider that context switching is one of the biggest killers of developer productivity and nothing is more of a context switch than triaging and investigating a security issue that ends up being a complete waste of time. With these factors in mind, I suggest we completely rethink how developers should deal with security issues. We should prioritize as much automation as possible to lower the cognitive load on developers and allow them to do what the business needs: ship great code and ship it fast.

One question that might come to mind is: Why can’t the SOC just handle developer security issues? This would seem like an appropriate solution. However, it doesn’t work.Why it doesn’t work boils down to two reasons: talent and context. First, it’s extremely difficult to staff a SOC already and if you must add to the skills requirement a bunch of modern development tech stacks your talent pool will shrink to near zero. Second, even if you could staff the SOC with people who know all the tech stacks your developers use, at the end of the day the application context is key to decision making. If developers are writing and releasing code at a rapid pace, the context of the application is constantly moving, evolving, and changing. There is no way someone in the SOC can keep up with that rate of change on top of the fact they are already outnumbered 100 to 1. I have seen, in some situations, all tickets are directed to the AppSec team first and then assigned to developers after triage. One advantage of this approach is it cuts down on obvious false positives for developers. It comes at a price which may not be worth it for some developers. The price you pay is latency. The AppSec team has to burn down the triage queue and by the time they get to a developer’s ticket and assign it to them it could be weeks or months from the time they actually wrote the code. This would be a deep context switch for a developer to remember the details of that code and then design a remediation. This workflow also interrupts the normal sprint flow of a development team and usually means less velocity for the features and bug fixes product managers are relying on.

Conclusion
In this blog I only scratch the surface of how we should start thinking about security for developers. I also tried to show the difficult position security teams are in regarding how they deploy tools for application security. I also want to mention that I have met a lot of security teams that go the extra mile and make developer lives easier. These people are the rockstars in the security space and find ways to add real value to the business. To the developers out there, I want you to know things are changing and help is on the way!