DevOps in Finance: Best Practices and Case Studies

The financial services sector has had a mixed relationship with DevOps. Denial — No, it’s impossible. Anger — Why is it that everyone but us can do it?! Bargaining — Okay, maybe we should at least try CI/CD? Depression — No, we are not making enough progress. And, finally, acceptance — Yes, the financial sector can do DevOps and do it well!

Financial businesses can do DevOps so well, in fact, that State of DevOps Report by Puppet and CircleCI ranked financial services among the leaders in DevOps practice maturity, alongside tech industry players.

Indeed, many FinTech and traditional financial companies have made admirable progress in DevOps adoption over the years. Because they are similar to tech players, companies in the financial sector have similar IT demands and constraints:

• Growing data volumes
• An expanding cross-channel presence
• Rapid rates of digitization
• Secure, low-latency data exchanges

The difference is that many tech companies (at least the cloud-born ones) have headroom because they are starting from scratch. Traditional financial institutions have the baggage of legacy infrastructure as well as cultural resistance, which may slow down transformations.

But the twenty twenties look to become decisive for plowing ahead with DevOps adoption. We can now observe the financial sector more and more utilizing AWS platform potential to build new capabilities and deliver meaningful business value. While doing so, the organizations have to balance their existing processes, systems, processes, and toolset in combo with the AWS DevOps culture of transformation while innovating at speed of market demands.

Why DevOps in finance is a priority

According to the Atlassian Survey 2020 – DevOps Trends, 99% of respondents say DevOps has had a positive impact on their organization. That’s a not-so-surprisingly nearly unanimous result.

At this point, the benefits of DevOps are well-described:

• Better release cadence. Top DevOps teams deploy new code 208 times more frequently than WHOM.
• Faster deployments. 49% of teams using DevOps report faster time to market for new applications.
• Increased reliability. Mature adopters have a 7X lower rate of failure.
• Improved security and compliance thanks to better-documented development and testing processes, plus clear frameworks for application governance, risk management, and security.

Ultimately, the promise of DevOps for finance is that it will help your software teams get better at what they do — writing, testing, securing, and integrating new code — through better processes and supporting tools.

So why now for DevOps?

In one buzzword — digital transformation. Amidst the pandemic, many financial companies had to triple down on their digital service to meet consumers’ shifting preferences. Frontend transformations such as digital account opening, KYC, mobile banking, online customer support, and self-service had to be done fast.

This major push for a better digital customer experience (CX) has had a ripple effect on backend systems and supporting IT processes. A growing digital footprint, stretching through on-premises software and the cloud, requires better cadence for managing new deployments, target infrastructure, and legacy cores integrated with new digital add-ons.

DevOps is a people, technology, and process equation aimed at finding the optimal cadence for managing, optimizing, and expanding software development capabilities in the context of overall IT services delivery. It’s just the glue many financial teams need to bind together their product delivery services.

DevOps best practices for financial companies

While DevOps can offer fast results, the route to adoption may take several bends. In a survey by Atlassian, over 85% of teams reported at least one roadblock to implementing DevOps:

• Lack of skills/knowledge among team members
• Legacy infrastructure
• Immature delivery culture

The good news? These obstacles are manageable with some DevOps adoption guidance and a commitment to new ways of working. Here’s a lowdown on the DevOps best practices for financial companies (with benefits and case studies).

CI/CD for finance/financial institutions

CI/CD stands for continuous integration and continuous delivery (or deployment). Continuous integration means that developers frequently integrate their code into the main branch. Every change to the application first passes a number of automated tests and only then gets into the build. This helps you avoid integration hell and find problems in software early on.

What’s more important, continuous integration makes continuous delivery possible. CD is a set of practices that speed up the software lifecycle. In other words, CD allows product releases to happen more frequently. It pushes the build to delivery environments. And just like CI, CD is a largely automated process that involves quantifiable risks. But continuous delivery doesn’t necessarily mean continuous deployment. Putting the release into the hands of users is often still a manual process. By contrast, continuous deployment is fully automated. Essentially, with continuous deployment, every update to the build gets into production right away.

Continuous integration, delivery, and deployment explained

The business benefits of CI/CD for finance:

• Faster time to market. Automation means that more code gets deployed to production sans scheduled maintenance or other service disruptions.
• Flexibility and responsiveness. CI/CD lets you build fast and fail fast in a sandbox environment. This allows your teams to rapidly test products, troubleshoot bugs and integration issues, and release fully functional software rather than half-baked code.
• Increased productivity. Implementing CI/CD allows the development team to stay more productive. CI/CD for FinTech eliminates rework and wait time. Thanks to the automation of routine processes, software developers can focus on other more crucial tasks, such as ensuring code quality and security.
• Product quality. Since the development process is largely automated with CI/CD, it’s also more reliable. Errors are detected early in development and potential risks are calculated, making it easier for developers to control them.

To sum up, continuous integration (CI) and continuous delivery (CD) for FinTech and financial institutions pave the way to faster, more stable, and more predictable code deployments. In essence, CI/CD creates a step-by-step production line with quality checkpoints.

Capital One, one of the earliest DevOps adopters in finance, designed their software pipelines using the principle of “16 Gates” — checkpoints that help ensure top code quality. These include:

• Rigorous source code version control and optimum branching strategy
• Static code analysis
• Minimum acceptable code coverage of 80%
• Ongoing scans for vulnerabilities
• Artifact version control
• Automated resource provisioning
• Immutable servers
• Integration and performance testing
• Automated build deploy testing for every commit
• Automated rollback and change orders
• Zero-downtime releases
• Feature toggling

This approach has enabled Capital One to reduce the time needed to build new application infrastructure by over 99%. Plus, it has significantly improved the company’s software release timelines.

Continuous testing

Continuous testing (CT) is the process of introducing automated testing at different stages of the software development lifecycle to enhance end product quality. In other words, CT encourages you to test early, often, and fast.

Continuous testing accomplishes several things at once:

• Assesses risk coverage. Learn which code is production-ready and which release candidates are not ready to move to the delivery pipeline (e.g. due to integration issues).
• Acts as a safety net. Prevent subpar products from creeping into releases and undermining the user experience. Early bug detection also reduces bills for bug fixes.
• Ensures comprehensive coverage. Test data and test cases are written in advance and always available for instant runs. CT also covers everything from “shift left” (e.g. unit and component testing to prevent costly debugging at later stages) to “shift right” (load testing, application performance monitoring, production testing, etc.) to ensure stable product performance, high failure tolerance, and a stellar customer experience.
• Augments product quality. Mature teams run software tests at the right time and stage of the SDLC (without creating bottlenecks for other processes) and use the insights obtained from tests to improve code quality.

To sum up, continuous testing encourages you to blend testing activities into your development process instead of postponing most till the last stage so that all produced code is covered by end-to-end tests from the get-go through deployment.

Mastering this “shift left” often requires not just better testing tools but more tectonic changes in QA processes and team mindsets. These can be challenging to execute. Here’s some extra data to help you make your business case.

Enterprises can save up to $7 million in operating costs over 3 years after adopting continuous testing. With the help of automation and test case number minimization, teams can also shave off 80% in execution efforts, resulting in $4 million cost savings in test case execution.

Forrester Total Economic Impact Study on Continuous Testing

Furthermore, Forrester indicates that teams with mature continuous testing practices:

1. Minimize unplanned work by 28% and spend 50% less time on test case design
2. Spend up to 80% less time on test case automation and execution
3. Save up to $13 million in testing environment infrastructure costs
4. Increase release management productivity by as much as 40%
5. Reduce cut-over calls in critical code deployments from 8 hours to 30 minutes

Forrester also reports that better requirements formalization — a positive side effect of CT — also reduces the volume of unnecessary QA work and increases team performance.

Infrastructure as code (IaC)

Infrastructure as code (IaC) refers to the automated provisioning and management of infrastructure for new deployments so you can develop, test, deploy, and scale new cloud applications faster and with less risk.

Instead of programming configurations manually, your teams can automatically create and replicate the same environment each time they start a new project. IaC helps to ensure that all infrastructure configurations are well-documented, version-controlled, and homogenous, yet flexible — meaning you can divide your infrastructure into modular components to suit an array of projects while ensuring that all of them use the versioning the DevOps team uses for source code.

IaC documents the full topology of your infrastructure — from networks and VMs to load balancers — and incorporates all steps required to configure the right environment to run your applications.

In short, infrastructure as code acts as scaffolding. In one click, you can build up the environment you need to develop and test new products.

Benefits of infrastructure as code for financial application development include:

• Faster development and deployment. Having the ability to set up infrastructure on command accelerates team performance at every stage of the software development lifecycle. Your team can provision CI/CD and testing environments in a matter of moments, plus streamline deployments since the application and production infrastructure are neatly packaged in one block.
• Consistent product quality. The biggest issue with manual infrastructure provisioning is configuration drift: when your dev, testing, and deployment environments are mismatched. Ad-hoc configurations can cause issues during deployment and result in security vulnerabilities and non-compliance — the least desirable scenarios for finance.
• Enhanced testing. IaC lets you test apps in a production-like environment at any stage of the SDLC. Doing so helps prevent common deployment issues caused by configuration drift, missing dependencies, or integrations.
• Cost optimization. The cloud computing bill can run high when left unattended. IaC helps contain resource consumption through targeted optimization and the dynamic provisioning and teardown of environments.

Case in point: McKinsey notes that one European bank managed to reduce the size of its IT infrastructure team by 45% after moving to standardized, automated infrastructure provisioning products.

What do these infrastructure team members do now? Probably play in the automatically provisioned sandbox as Barclay’s team does. Brian Simpson-Adkins, the product owner of IaC at Barkley’s, has shared how IaC has helped Barkley’s get better at innovation without breaking any regulatory barriers:

Such an approach has allowed the heritage financial institution with operations in 40 countries to remain compliant while also being up-to-speed with adopting emerging technologies and responding to shifting customer demands.

DevSecOps

DevSecOps is a newer trend in town that mature financial organizations are now eager to explore.

Short for development, security, and operations, DevSecOps is aimed at infusing automated security best practices at every stage of the SDLC, from design to deployment.

Source: McKinsey — Agile, reliable, secure, compliant IT: Fulfilling the promise of DevSecOps

Similar to continuous testing, DevSecOps encourages the development team to incorporate security continuously into every iteration instead of treating it as an afterthought. DevSecOps helps ensure that deployment-ready software meets all compliance, reliability, and security requirements.

It’s no secret that the financial industry is a prime target for cyber attacks — to the extent that financial services providers spend three times more on cybersecurity compared to other businesses. While DevSecOps cannot fully replace proactive security measures, this movement helps ensure you are not leaving any loose ends an attacker might explore. In addition, DevSecOps promotes the establishment of automated governance and compliance in the development process — something 63% of mature DevSecOps teams in the financial sector already do.

The World Bank Group recently set a course for DevSecOps after successful adoption and scaling of the underlying DevOps practices. They broke down the adoption into two acts:

• Act I: First, the team decided to automate cloud security testing. After assessing different scenarios, the team managed to automate over 400 security decisions. However, automation didn’t eliminate operational security support. The DevSecOps team remained in charge of security monitoring and response.
• Act II: Next, the financial institution decided to codify security controls for their infrastructure as code using the AWS Service Catalog.

Finally, they combined the above practices into a secure pipeline built to match their operational and security requirements:

Source: Sonatype — The World Bank Group’s Cloud Journey with DevSecOps presentation

Ultimately, the World Bank Group plans to move towards pipeline as code as well as security as code approaches to further strengthen their product security.

Why DevOps on AWS is THE choice

An always newfangled AWS platform makes the shift from the old-school gatekeeping to modern integrations that lead to secure financial management practices in perspective. The AWS has proven its capabilities in the real world, from optimized banking ops to security-sensitive payment services, to capital markets enablement to new level of insurance agility.

According to Statista, Amazon Web Services (AWS) dominates the DevOps market, with a 40% market share and is expected to remain the DevOps market leader in the coming years.

Moving behind the scenes of business operations, productivity at speed, quality at scale and time to market – modern technology ecosystem does not accept the term ‘compromise’. Yesterday’s digital product release can’t be today’s buggy snag, because tomorrow’s new feature release is already in the pipeline.

The adoption of DevOps on AWS model reaches far beyond cost optimization; nowadays it is a combination of timely software delivery, system’s reliability, scalability, top security and improved cross-team collaboration.

Implementing DevOps on AWS enables automation at scale which powers finance teams to deliver new applications and features to market and respond to customer-driven sector’s dynamics faster. With AWS platform’s superpowers in security-by-default features, DevOps and AWS DevSecOps teams leverage cloud-based infrastructure to maintain integrity of sensitive financial data, confidentiality and compliance with regulations. Collaboration is the very core of DevOps culture, when combined with developed network and AWS community, it encourages even deeper cooperation and agility between technology and business helping to break down silos and improve workflows for better business outcomes.

DevOps tools in the finance industry

Many think that tool adoption is the key to a successful DevOps transition. Yes and no.

Indeed, DevOps tools can deliver much-needed automation and repeatability — the key principles of DevOps. But the best software will fall short if it doesn’t support the respective operational processes described above.

Once you’ve set the above on the rails, you can start cherry-picking the missing tools in your chain. Some common tools include:

• Version control and source code management tools
• CI/CD automation servers such as Jenkins, TeamCity, Bamboo, or GitLab CI/CD
• Continuous testing tools such as Selenium, Appium, and Test Sigma
• Automated release and deployment tools (pipelines)
• Infrastructure as code tools for IT infrastructure automation (Puppet, Chef)
• Containerization and virtualization tools such as Docker, Kubernetes, and Vagrant
• Workload orchestration and scheduling tools

How to approach DevOps adoption?

DevOps is a cultural shift, requiring targeted action and dedication. First and foremost, DevOps is a practice that requires paced changes in how your teams approach software development, testing, security, and delivery.

Without strong Agile practices, adoption will likely stall. But then again, a rapid and imposed shift to the Agile way of doing things may not drive immediate results.

Choose your battles. Start small and adopt one best practice at a time. Begin with introducing CI/CD and polish your baseline act. Then consider adding a greater degree of automated testing and self-service infrastructure provisioning. Assess and iterate on your performance and foster further improvements. DevOps isn’t a destination — it’s a journey.

Contact Intellias DevOps practitioners to receive preliminary guidance on DevOps adoption and more insights on how financial institutions can rethink their approach to software development.