DevOps and Cloud InfoQ Trends Report – July 2021 – InfoQ.com

Key Takeaways

This article summarizes how we currently see the "cloud computing and DevOps" space, which focuses on fundamental infrastructure and operational patterns, the realization of patterns in technology frameworks, and the design processes and skills that a software architect or engineer must cultivate.

Both InfoQ and the QCon conference series focus on topics that we believe fall into the "innovator, early adopter, and early majority stages" of the diffusion of technology, as defined in Geoffrey Moores book "Crossing the Chasm." What we try to do is identify ideas that fit into what Moore referred to as the early market, where "the customer base is made up of technology enthusiasts and visionaries who are looking to get ahead of either an opportunity or a looming problem." We are also looking for ideas that are likely to "cross the chasm" to broader adoption. It is perhaps worth saying, in this context, that a technology's exact position on the adoption curve can vary. For example, microservices are widely adopted amongst Bay Area companies but maybe less widely adopted and perhaps less appropriate elsewhere.

In this edition of the cloud computing and DevOps trend report, we believe that hybrid cloud approaches have evolved to become more "cloud native". In late 2019 all the three prominent public cloud vendors brought new hybrid cloud products to the market and over the last two years they have continued to invest heavily in them - Google with Anthos, Microsoft with the Azure Arc and Stack offerings, AWS with Outposts, and more recently, Amazon ECS Anywhere. For enterprises, for instance, it is about not only bringing workloads to the Cloud but also running them on-premise or both, or on multiple clouds. Thus, managing the infrastructure for the workloads centrally with a service like Arc or Anthos delivers value. Furthermore, these products allow enterprises to extend their platform.

There has been increasing adoption (and technological evolution) in the space of "edge cloud" and "edge computing", and so we believe this topic should move to the early adopter stage of our graph. There is a fair amount of traction here from specific vendor tools, such as Cloudflare Workers, Fastlys Compute@Edge, and Amazon CloudFronts Lambda@Edge.

The participants of this report have also identified an emerging trend named "no copy data sharing." This can be seen in data management services such as Snowflake, which do not copy or move data, yet enable users to share data at its source. Another example is the Azure Synapse service, which supports no-copy data sharing from Azure Cosmos DB via Azure Synapse Link. The recently announced Delta Sharing open standard is also contributing to the upward trajectory of the no-copy data sharing tendency.

Observability continues to be a popular topic within DevOps and SRE. While most organizations have begun to implement some form of observability stack, as Holly Cummins notes, the term is overloaded and therefore should be broken down into its various components. Ideas such as centralized log aggregation are currently commonplace in most organizations, however, logs only make up one of the three pillars of observability.

The increasingly popular OpenTelemetry project provides a consistent framework for capturing not just logs, but also traces and metrics. The consistency provided by adopting a single framework helps with capturing data across hybrid and heterogeneous environments and also monitoring tooling. The use of service level objectives (SLOs) as a tool to communicate the desired outcome of monitoring and observability is also gaining popularity as seen with the first ever SLOConf earlier this year.

"DevOps for Data" has seen increasing adoption over the past year with the rise of both MLOps and DataOps. MLOps focuses on using DevOps style practices (such as CI and CD) to implement continuous training for machine learning models. Open source tooling and commercial services exist to help in this area, such as KubeFlow for deploying ML models on Kubernetes, and Amazon SageMaker Model Monitor for automating monitoring of ML models. DataOps looks to shorten the cycle time of data analytics by applying similar concepts used by DevOps teams to reduce their own cycle times.

In the people and organisational space of DevOps we have seen the Team Topologies book, from Matthew Skelton and Manuel Pais, become the de facto reference for arranging teams within an organization to enable effective software delivery. Team Topologies describes four fundamental team types and three team interaction patterns, and dives into the responsibility boundaries of teams and how teams can communicate or interact with other teams.

There is also increasing focus on post-incident "blameless postmortems" in becoming more akin to "healthy retrospectives", from which the entire organisation can learn from. Key leaders in the computing domain of resilience engineering and the "Learning From Incidents" community have been influential in driving this discussion.

For context, here is what the topic graph looked like for the first half of 2019. The 2021 version is at the top of the article.

The following is a lightly edited copy of the corresponding internal chat log between several InfoQ cloud computing DevOps topic editors and InfoQ contributors, which provides more context for our recommended positioning on the adoption graph.

Lena Hall - Director, Large-Scale Systems @ Microsoft, who contributed the recent 'Can We Trust the Cloud Not to Fail?' and 'Evolution of Azure Synapse: Apache Spark 3.0, GPU Acceleration, Delta Lake, Dataverse Support' articles:

Moving to Early Majority

Moving to early adopters

New topics in early innovators

Furthermore, Hall discussed the evolution of the hybrid cloud model with InfoQ:

Cloud-Native Hybrid Approaches (e.g., Azure Arc, Google Anthos). In its traditional understanding, a hybrid cloud means using on-premises infrastructure for parts of the company's workloads and using the public cloud for other parts of it, where relevant. For example, a hybrid cloud strategy could mean running some applications in the cloud (e.g., serverless processing) and some applications on local infrastructure (e.g., business ERP system). The hybrid data approach could look as simple as storing backups of data in the cloud, or using more advanced cloud gateways and precisely-tuned data sync, caching, or movement patterns.

In the recent months and years, hybrid options have evolved beyond the traditional definition. They have expanded to enable the functionality of cloud services to run outside of the cloud, allowing for a much more seamless and smooth experience. We can think of the new cloud-native hybrid options as an extension of cloud services to our on-premises or multi-cloud environments. Azure Arc and Google Anthos are perfect examples. One of the ideal examples is Azure Arc. It allows the running of Azure SQL Managed Instance and Azure Database for PostgreSQL Hyperscale in hybrid and multi-cloud environments. It also provides the management of applications consistently on-premises and in the cloud with Azure App Service, Functions, Logic Apps, and more. Another fantastic example is Google Anthos powering BigQuery Omni, enabling its query engine to be deployed and managed on multiple clouds.

It's important to note that these new cloud-native hybrid approaches like Azure Arc and Google Anthos are different from more standard hybrid cloud approaches, such as Azure Stack or Amazon Outposts. To understand it more clearly, both Azure Arc and Azure Stack offer hybrid solutions (and can be used together). Azure Stack is a hardware approach to run Azure environment on-premises. On the other hand, Azure Arc is a software approach treating the on-premises and multi-cloud resources, consisting of virtual or physical servers and Kubernetes clusters, as something natively managed and accessible by Azure Resource Manager. Similarly, we can make a distinction between Google Anthos/Azure Arc and AWS Outposts. Even though AWS Outposts extends the AWS cloud platform to on-premises, it has specific hardware requirements and only works with hardware devices designed and supported by AWS.

Hall also shared an interesting insight into the emerging trend of "no copy data sharing":

No Copy Data Sharing (Snowflake,Google BigQuery, Azure Synapse, Presto/Starburst, etc.)There are quite a few approaches to working with data when the same data needs to be accessed or shared between different services or different environments. There might be a variety of scenarios when it comes to data sharing.One scenario could be storing the same data (or parts of the same data) in several services running in the same cloud. Often, the compute and storage of such services aren't independent. To work with the data from another service, we'd have to copy or move it.

In the emerging no-copy data-sharing approach, and with separation of compute and storage, it's not necessary to move or replicate the data to be able to access it from different services. As a result, it enables better scalability, cost-efficiency, and direct access to data. As an example, this can be enabled by features such as Azure Synapse Link for Azure Cosmos DB, BigQuery external data sources, or Snowflake data sharing.

Recently announced Delta Sharing is an open standard, contributing to the upward trajectory of no-copy data sharing tendency. According to its website, "Delta Sharing is the industry's first open protocol for secure data sharing, making it simple to share data with other organizations regardless of which computing platforms they use."

Holly Cummins, an innovation leader in IBM Corporate Strategy who has spent several years as a consultant in the IBM Garage, shared and contributed the "Cloud-Native Is about Culture, Not Container" talk and article shared with InfoQ:

Wow, some things in tech change so fast, and some hardly change at all. Most of the introduction of the 2019 DevOps and Cloud InfoQ Trends Report feels like it could have been written now.

I don't think there's been any progress on undoing the confusion between continuous integration (CI) and continuous delivery (CD) tooling and CI/CD practices, and I don't think there's been progress on adopting CI and CD practices.

One thing which seems to be missing in the discussion is observability. Like "AIOps", it's overloaded and buzz-wordy, but it feels a bit like how "cloud-native" was in 2018. So if your product is anywhere remotely in this space, you have to throw in the word 'observability.' Then other people will point out that it's not really observability because observability means something distinct, and then there's an extensive discussion. 🙂

In the chart itself, breaking it down into things like "centralized log aggregation" seems wise because then it avoids the argument, but it would be worth mentioning in the discussion. Another subcategory to include might be OpenTelemetry, in the innovator section.

I feel like service meshes have slipped out of the conversation a bit.

FinOps should be in the innovator section. A related topic is cloud cost optimization. There's an exciting conversation started by James Governor from the RedMonk team "Shifting cost optimisation left: Spotify Backstage Cost Insights" where he makes a distinction between FinOps (real-time information flow for cost analysis, aimed at finance team) and "shifting cloud cost optimization left" (aimed at engineering team).

Sustainability accounting is another innovator category. Again, it's very early - I can't think of many products with enough maturity in this area - but the conversations are happening, and I think it's coming.

I think DevOps for data and ML at the edge have moved to early adopters. DevOps for Data has two subcategories, MLOps and DataOps.

I'd say site reliability engineering (SRE) is now the early majority. I'm seeing a lot of banks adopting it.

I see a lot of conversation around GitOps, but I'm not sure that's translating into implementation in prod, or at least not enough to push it to the early majority. Likewise, I feel like momentum around ChatOps has slowed, and it hasn't pushed into Early Majority.

Renato Losio, Cloud architect, remote work enthusiast, and speaker:

Jared Ruckle, Cloud Editor @ InfoQ; Product Marketing @HashiCorp:

Changes in the Innovator bucket

Changes in the Early Adopter bucket

Changes in the Early Majority bucket

Aditya Kulkarni, Blogger, Reader, and Iteration Manager at Tenerity:

Moving to Early Majority

Moving to early adopters

New topics in early innovators

Steef-Jan Wiggers, Technical Integration Architect at HSO and Microsoft Azure MVP:

Moving to Early Majority

New topics in early innovators

Daniel Byrant, Director of DevRel @ Ambassador Labs | InfoQ News Manager | QCon PC:

Innovators

Early Adopters

Early Majority

Late Majority

Helen Beal, DevOps speaker, writer, and strategic advisor:

1)

2)

Rupert Field, Delivery Lead @Duffle-bag Consulting and DevOps Editor @InfoQ:

Shaaron A Alvares, Editor at InfoQ for DevOps, Culture & Methods | Agile Coach

Is Continuous Verification contained in "Shift left on Sec/InfoSec"?

Add to Innovators: Sociotechnical architecture DevOps value stream management platforms Developer Velocity (or it can be contained in DevEx?)

Add to Innovators or to Early Adopters: Low Code No Code (LCNC) Hybrid Cloud

Move: AIOps and MLOps move to Early Adopters

Matt Campbell, Lead Editor, DevOps | Engineering Director @ D2L

Observability practices and tooling continue to mature. I think we should be looking at splitting the topic of observability into some newer sub-topics such as OpenTelemetry and service level objectives (SLOs). As noted by others, the logging portion of the three pillars of observability is a relatively well-adopted piece. Monitoring of metrics is probably close to the same level of adoption at this point so I'd say EM for that. The tracing pillar of observability is still a less adopted portion and there are new advancements recently, especially with the more widespread adoption of OpenTelemetry.

SLOs as a tool for communicating outcomes and goals have started to see a resurgence with the recent SLOConf leading the way. I expect to see the various observability platforms start to add SLO-creation and tracking tooling soon. I would put this in late innovator/early EA.

Security continues to be a hot topic as more and more high-profile attacks make the news. While infrastructure as code is probably within EM or later, tooling and techniques to automate IaC vulnerability scanning are not as well utilized (maybe EA). Newer approaches such as Policy as Code (as promoted by Open Policy Agent (OPA)) and remote access management tooling (such as HashiCorp's Boundary) are pushing forward identity as code and privacy as code, so these are probably within the innovator space. These trends are a continuation of the "shift left for security" approach or the newer DevSecOps ideas. With the continued push for early security practices I think we will see more tools and processes introduced in the upcoming year.

The InfoQ editorial team is built by recruiting and training expert practitioners to write news items and articles, and to analyze current and future trends. Apply to become an editor or to contribute articles and get involved with the conversation.

Lena Hall - Director of Engineering: Big Data at Microsoft. She is leading a team and technical strategy for product improvement efforts across Big Data services at Microsoft. Lena is the driver behind engineering initiatives and strategies to advance, facilitate and push forward further acceleration of cloud services. Lena has more than 10 years of experience in solution architecture and software engineering with a focus on distributed cloud programming, real-time system design, highly scalable and performant systems, big data analysis, data science, functional programming, and machine learning. Previously, she was a Senior Software Engineer at Microsoft Research.

Holly Cummins - Innovation Leader, Corporate Strategy SPEED @IBM and spent several years as a consultant in the IBM Garage. As part of the Garage, she delivers technology-enabled innovation to clients across various industries, from banking to catering to retail to NGOs. Holly is an Oracle Java Champion, IBM Q Ambassador, and JavaOne Rock Star. She co-authored Manning's Enterprise OSGi in Action.

Renato Losio - Cloud architect, remote work enthusiast, speaker, and Cloud Editor @InfoQ. Renato has many years of experience as a software engineer, tech lead and cloud services specialist in Italy, UK, Portugal and Germany. He lives in Berlin and works remotely as principal cloud architect for Funambol. Location-based services and relational databases are his main working interests. He is a AWS Data Hero.

Jared Ruckle - Cloud Editor @ InfoQ; Product Marketing @HashiCorp. Jared has over 20 years experience in product marketing and product management. He has worked at numerous IaaS, PaaS, and SaaS companies, including VMware, Pivotal, and CenturyLink. Currently, Jared is director of product marketing at HashiCorp.

Aditya Kulkarni - Blogger, Reader, Iteration Manager at Tenerity, and DevOps Editor @InfoQ. Starting from the developer role, Aditya has evolved into the management domain. Having worked with organization on their journey of agility, Aditya has kept in touch with the technical side of things.

Steef-Jan Wiggers - Technical Integration Architect at HSO, Microsoft Azure MVP and Cloud Lead Editor @InfoQ. His current technical expertise focuses on integration platform implementations, Azure DevOps, and Azure Platform Solution Architectures. Steef-Jan is a board member of the Dutch Azure User Group, a regular speaker at conferences and user groups, writes for InfoQ, and Serverless Notes. Furthermore, Microsoft has recognized him as Microsoft Azure MVP for the past eleven years.

Daniel Byrant - DevRel @ Ambassador Labs | InfoQ News Manager | QCon PC. His current technical expertise focuses on DevOps tooling, cloud/container platforms and microservice implementations. Daniel is a leader within the London Java Community (LJC), contributes to several open source projects, writes for well-known technical websites such as InfoQ, O'Reilly, and DZone, and regularly presents at international conferences such as QCon, JavaOne, and Devoxx.

Helen Beal - DevOps speaker, writer, strategic advisor, and InfoQ DevOps Editor. Herfocus is on helping organisations optimise the flow from idea to value realisation through behavioural, interaction based and technology improvements.

Rupert Field - Delivery Lead @Duffle-bag Consulting and DevOps Editor @InfoQ. He loves learning about technology and helping people harness it to fix problems. His experience includes defining, designing and delivering technology team strategy. This includes delivering coaching, training, operating model design and business transformation.

Shaaron A Alvares - Editor at InfoQ for DevOps, Culture & Methods | Agile Coach. She is Certified Agile Leadership, Certified Agile Coach from the International Consortium for Agile, and Agile Certified Practitioner, with a global work experience in technology and organizational transformation. She introduced lean agile product and software development practices within various global Fortune 500 companies in Europe, such as BNP-Paribas, NYSE-Euronext, ALCOA Inc. and has led significant lean agile and DevOps practice adoptions and transformations at Amazon.com, Expedia, Microsoft, T-Mobile. Shaaron published her MPhil and PhD theses with the French National Center for Scientific Research (CNRS)

Matt Campbell - Lead Editor, DevOps | Engineering Director @ D2L, an education technology company, responsible for their Infrastructure and Cloud platform teams. His area of focus is DevOps and SRE and implementing these at enterprise scale. He also instructs programming courses with Conestoga College.

Read the rest here:

DevOps and Cloud InfoQ Trends Report - July 2021 - InfoQ.com