DevSecOps: The good, the bad, and the ugly

DevSecOps offers benefitsbut it also has its challenges. Learn why companies are making the shift and why its not always easy.

DevSecOps is the practice of integrating security into every stage of the DevOps pipeline. It unites development activities, operations support, and security checks, and coordinates the teams involved in the software development life cycle (SDLC). The synergy between the teams is helped by automation.

But DevSecOps isnt a quick fix or a temporary solution. Its a long-term implementation that helps ensure that an organization can achieve and maintain secure SDLC practices. It requires development teams to follow a standard SDLC process to guarantee that security issues are addressed early on, when they are easier to fix.

In the DevSecOps paradigm, developers maintain versions of their code and follow a peer review process before it can be moved to other environments. No one person or team has full control over how updates are made in the code/environmentseparate teams are responsible for development, testing, deployment, and so on.

The operations team supports the entire development process, including maintaining and updating the operating environment, defining and implementing a deployment process, and logging every detail of the DevSecOps process.

The security team identifies and eliminates any vulnerabilities. Should a vulnerability reach production, DevSecOps processes provide a clear trail of when and how it got there. In traditional coding processes, the process of deploying an application to production starts with changes to code, but in DevSecOps, building, testing, and security scans start earlier and require other activities to complement it, such as design reviews and postproduction monitoring.

There are many tools that offer various types and combinations of services, but there is no single tool that can provide a DevSecOps process. Some vendors that offer static application security testing (SAST) tools are now adding software composition analysis (SCA) tools (and vice versa), but DevSecOps is more than just performing scans.

Its also important to note that no one tool fits in all environments, and often no one tool fitsall companies. In addition to application testing tools, DevSecOps processes require reporting tools, defect tracking/management tools, environment building tools, and more. Also please note that security, build, and metric collection activities are not restricted to just the tools available in the market. Even scripts (Shell, PowerShell, Python, etc.) offer various capabilities.

In DevSecOps any changes to code will trigger activities such as SAST or dynamic application security testing (DAST) review, architecture review, pen testing, and so on, which in turn trigger scans, which in turn generate metrics and reports that can make or break deployment. And all that happens in minutes or even secondsand that speed enables teams to scale up.

Security teams are often short of resources, but they still hold responsibility for stopping bad actors from taking advantage of vulnerabilities. As development teams move toward more multifaceted, multipronged agile methods, DevSecOps processes help security teams share the responsibility of building security into the CI/CD workflow with everyone involved in the SDLC process. Introducing security testing earlier in the SDLC enables developers to fix security issues in their code in real time to avoid costly delays.

DevSecOps lets you do a lot of work quickly. Speeding things up, reducing delays, and enabling scalability are some of the biggest advantages. With global teams dispersed across many different time zones, organizations need processes and frameworks that foster collaboration while reducing dependencies to help teams achieve their goals.

Often, different teams work in independent silos, and these fragmented environments make it difficult to ensure consistency while still allowing each team the independence their processes require. How can teams get aligned on a goal if theyre unable to change their entire development process? Is gradual change better than full-blown change in implementation?

DevSecOps can address these issues and more. No matter how mature an organizations security level isor how fragmentedDevSecOps makes it possible to initiate and implement security activities and adapt to different functional teams.

You may have heard the saying, Security isnt one persons responsibility, its everyones responsibility. DevSecOps involves everyone in the process and practices of ensuring security. Developers, app managers, ops teams, security teams, reviewers, and testers all have an important role to play.

The fast-paced world needs fast-paced solutions. DevSecOps eliminates manual steps and dependencies, so the entire process is completed faster and sooner.

The power of acceleration in DevSecOps can be seen in the example of a company that received a request to onboard and scan 30 microservices two days before production. By leveraging automation, the team was able to complete this request in two hours. Think of that: onboard 30 new microservices on two tools, run the scans, evaluate them, and triage the scan resultsall in two hours.

With often less than a week to move through the entire SDLC, there is little time to address security processes. Thats why many security tools today have improved in terms of how quickly a scan can be run, and many provide capabilities to customize a scan so you can select the checks to run, further optimizing scan time.

One of the reasons DevSecOps is so popular is that it enables security teams to scale with limited bandwidth. The automation inherent to DevSecOps is critical to a firms ability to support many applications even with a limited security team. For example, a team of four was tasked with SAST reviews and signoffs, but since it was done manually, it could only support 200 apps. But with automation and security integration, the team was able to scale up to 700+ apps in a few months and support reviews for each of them.

Organizations may want to transition from one tool to anotherand sometimes that involves 1,000 apps or more. Does that sound like a nightmare to you? In DevSecOps, jobs are run through a common library of scripts, and because those scripts are shared across all jobs, you can transition easily from one tool to another. Updating a common set of instructions with the new tasks or replacing existing tasks makes it easy to propagate these changes across all applications instead of making changes in each job.

In DevSecOps, processes are interconnected and automated, so its easy to see if a change results in an issue or problem. This level of traceability makes it easier to hold people accountable for their changes. It also encourages developers to be more careful about writing secure code, helping prevent the pipeline from breaking. Projects can also be set up to send emails to the entire development team at specific milestones, such as when the job is completed or when the team has met or failed to meet the security requirements. When a change triggers a scan, the emails can specify who checked in or created the changes that triggered the scan.

Automation can be used to trigger builds, scans, deployment, evaluations, and approvals. When these tasks are automated, security teams can focus on other important activities rather than the operations of it all. For example, if an organization has 700 apps, it would be difficult for a security team of four to monitor regular releases manually. But automation can greatly reduce the workload.

As mentioned earlier, organizations often have teams spread across several different time zones. The security team doesnt have the budget nor the bandwidth to support all those time zones, but DevSecOps can help. For example, it can monitor code being reviewed by developers in India during local business hours but late night / early morning New York time. It can also help by providing portals through which developers can run on-demand scans for on-prem tools. DevSecOps also helps by enforcing standardization, which makes every step of the process clear and understandable for everyone. Standardization also makes it easier to scale the process and make updates and additions as needed.

Every organization has to prioritize its activities, and DevSecOps may not be everyones top priority. In some cases, organizations may not be able to integrate security into their DevOps process because they are dependent on some environment and script changes. Or the team may not have the capacity to take on these changes due to other priorities.

Often teams support legacy apps because they simply dont have a plan to transition them yet. There are security tools that dont integrate easily or automatically with other tools, and they require a layer of abstraction in order to be used in the DevSecOps process. For example, until recently Burp didnt have a CI plugin, so it wasnt easy to integrate a Burp scan into an automated process.

Another issue with legacy applications is that they can be critical to functionality, but because they were written so long ago, no one is able to or willing to make changes. Assigning resources to automate such apps doesnt make sense. But these applications still have to be scanned by the security team on a regular basis (especially when there are updated testing methodologies). So they need to be adapted to the standardized process of testing.

DevSecOps requires patience and tenacity. Any DevSecOps implementation takes a minimum of a yearanything less than that is incomplete. It will involve a lot of planning and designing before you start setting up the solution. You must first identify the gaps in your current process and then determine the tools required to support the process you intend to implement. You will need to coordinate with a variety of teams to get buy-in and instruct them to implement the required changes. None of this happens overnight.

Making changes to your process affects all people involved in the process and all applications following the process. If all your applications are being scanned using a common set of libraries, any change in these libraries will impact all apps unless you put in specific conditions.

Adding a new application to this process may take a long time. Onboarding .Net applications usually take a lot more time because they must build correctly. Visual Studio tends to hide a lot of build errors and provides dependencies at runtime; this is less true for MSBuild. In cases when the app team built an application using Visual Studio and checks it in, an automated process using the MSBuild command line can break due to a variety of reasons (incorrect directory structure, missing dependencies, incorrect dependencies, and so on).

Sometimes the CI tool itself is not up to the task. Organizations often start their CI process using Jenkins because its free, open source, and popular. But the number of bugs in Jenkins and its plugins can be staggering, and they can result in the need for messy workarounds. And a lot of plugins on Jenkins are no longer maintained or supported. Which is not to say its completely badits still very useful. And its worth noting that although there are other CI tools on the market, they have limitations as well.

Its also good to remember that tools dont always have the level of maturity to do everything thats needed. Every tool has its limitations, especially in an automated process. For example, Jenkins probably cannot allow conditional parameterization. And there may be some plugins that offer workarounds, but not the actual requirement.

Of course, the biggest headache any security team has to deal with is false positives. Without properly customizing your security tools, you could be overwhelmed with false positives. Organizations need to be vigilant about customizing the tool to the application, language, technology, or framework being used to narrow down your results.

Despite the challenges, there are many advantages to adopting DevSecOps. Not the least of which is how it helps address the ongoing lack of resources in security teams. DevSecOps enables teams to work more efficiently and keep up with an ever-expanding environment.