The Cloud has been a significant contributor to the digital optimisation and transformation of businesses and institutions globally since the 2010s. It seems almost an eternity ago, when the IT department was essentially a support function, with KRAs around design and delivery of Information Technology Architecture encompassing Infrastructure, Data Centres and constituent servers, personal computers, software, networking and security systems, along with the associated, vendor evaluation, outsourcing, contracting, commissioning and finally aligning with business systems and goals, as this pre-millennium Research Gate paper indicates.

The one and a half decades since the advent of the millennium saw the rise of many trends, besides the cloud, such as shift from integrated to business specific applications, resulting data management and insights, globalisation, adoption of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), implosion of telecom, mobility and Mobile Backend as a Service (MBaaS), other technologies such as social media, E-commerce, Extended Reality, Digital Twins, AI/ ML, RPA, Internet of things, Blockchain and Chatbots and lastly, the growing skill-gaps and demands of talent.

The cloud has now taken over a major chunk of responsibilities pertaining to infrastructure, data centre, hosting, SaaS and architectural applications, platforms, networking and security functions thus freeing up the IT and Business Teams for leveraging technology for more strategic tasks related to operations, customers, R&D, supply chain and others. The cloud, hence enabled companies and institutions to leverage the amalgamation of technology, people and processes across their extended enterprises to have ongoing digital programmes for driving revenue, customer satisfaction, and profitability. The cloud can potentially add USD 1 Trillion of Economic Value across the Fortune 500 band of companies by 2030, as this research by McKinsey estimates.

Even before the pandemic, although the initial adoption of Cloud was in the SaaS applications, IaaS and PaaS was surely catching up, thus shifting away from the traditional Data Centre and On-premise Infrastructure. Gartners research way back in 2015 predicted a 30 % plus increase in the IaaS spending with the public cloud IaaS workloads finally surpassing those of on-premise loads. In the same year, Gartners similar paper highlighted significant growth in PaaS as well : both for Infrastructure and Application iPaaS.

The Cloud is adding significant value across Industry verticals and business functions, right from remote working with online meetings & collaboration tools, automated factory operations, extended reality, digital twins, remote field services and many others. The cloud has also been adopted as the platform for deploying other new technologies such as RPA and Artificial Intelligence/ Machine Learning (AI/ ML) Depending on industry best practices, business use cases and IT strategies, it became feasible to leverage infrastructure, assets, applications, assets, and software, in a true Hybrid / Multi/ Industry Cloud scenario with separate private and public cloud environments covering IaaS, PaaS, SaaS and MBaaS. As Platforms were maturing, organisations were furthermore transitioning from Virtual Machine and even from IaaS based solutions to PaaS based. Gartner had predicted in this research that by 2021, over 75% of enterprises and mid-sized organisations would adopt a hybrid or multi-cloud strategy.

There was also a clear transition from the traditional lift and shift to the cloud native approach, which makes full use of cloud elasticity and optimisation levers and moreover minimises technical debt and inefficiencies. This approach makes use of cloud computing to build and run microservices based scalable applications running in virtualised containers, orchestrated in the Container-as-a-service applications and managed and deployed using DevOps workflows. Microservices, container management, infrastructure as a code, serverless architectures, declarative code and continuous integration and delivery (CI/CD) are the fundamental tenets of this cloud native approach. Organisations are balancing use of containerization along with leveraging the cloud hosting provider capabilities especially considering the extent of hybrid cloud, efforts and costs of container infrastructure and running commodity applications.

From the architecture standpoint, cloud-based composable architectures such as MACH- Microservices based, API-first, Cloud-native SaaS and Headless and Packaged business capabilities (PBCs) are increasingly being used in organisations for enhancing Digital Experience Platforms enabling customers, employees and supply chain with the new age Omnichannel experience. These architectures facilitate faster deployment and time to market through quick testing by sample populations and subsequent full-fledged implementations. These composable architectures help organisations in future proofing their IT investments, and improve business resilience and recovery with the ability to decouple and recouple the technology stacks. At the end of the 1st year of the pandemic, Gartner here highlighted the importance of composable architecture in its Hype Cycle of 2021 especially in business resilience and recovery during a crisis.

Intelligently deploying Serverless Computing in the architecture also enhances cloud native strategies immensely, thus enabling developers to focus on triggers and running function/ event-based computing, also resulting in more optimised cloud economics. Also, access to the cloud service providers Function-as-a-service (FaaS) and Backend-as-service (BaaS) models significantly reduce the IT environment transformation costs. This Deloitte Research illustrates the advantages that Serverless computing can bring about to retail operations.

To enhance their cloud native strategies, to further encourage citizen development, reducing over reliance on IT and bridging the IT-Business Gap, organisations are also making use of Low Code No Code (LCNC) tools and assembling, clearly shifting from application development. Citizen developers are making use of LCNC functionalities such as Drag and drop, pre-built User Interfaces, APIs and Connectors, one-click delivery and others to further augment their containerisation and microservices strategies. This Gartner Research predicts that 70% of new applications developed by organisations will use LCNC by 2025, well up from less than 25% in 2020.

Infrastructure and Data Management in the cloud are being immensely powered up by Automation and Orchestration. Especially in minimising manual efforts and errors in processes such as provisioning, configuring, sizing and auto-scaling, asset tagging, clustering and load balancing, performance monitoring, deploying, DevOps and CI/ CD testing and performance management. Further efficiencies are brought to fruition through automation especially in areas such as shutting down unutilised instances, backups, workflow version control, and establishing Infrastructure as Code (IAC). This further hence value-adds to robust cloud native architecture by enhancing containerisation, clustering, network configuration, storage connectivity, load balancing and managing the workload lifecycle, besides highlighting vulnerabilities and risks. Enterprises pursuing hybrid cloud strategies are hence driving automation in private clouds as well as integrating with public clouds by creating automation assets that perform resource codification across all private and public clouds and offer a single API. This McKinsey research highlights that companies that have adopted end-to-end automation in their cloud platforms and initiatives report a 20-40% increase in speed of releasing new capabilities to market. A similar report by Deloitte, mentions that intelligent automation in the cloud enables scale in just 4-12 months, compared to the earlier 6-24 months period, through streamlined development and deployment processes.

CIOs are also increasingly turning to distributed cloud models to address edge or location-based cloud use cases especially across Banks and Financial Institutions, Healthcare, Smart cities, and Manufacturing. It is expected that decentralised and distributed cloud computing will move from the initial private cloud substation deployments to an eventually Wi-Fi like distributed cloud substation ecosystems, especially considering the necessary availability, bandwidth and other operational and security aspects.

These rapid developments in the cloud ecosystems especially for hybrid and multi cloud environments have necessitated infrastructure and data management to encompass dashboards for end-to-end visibility of all the cloud resources and usage across providers, business functions, and departments. Governance and Compliance, Monitoring, Inventory Management, Patches and Version Control, Disaster Recovery, Hybrid Cloud Management Platforms (HCMP), Cloud Service Brokers (CSB) and other Tools aid companies in better Infrastructure Management in the Cloud, while catering to fluctuating demands and corresponding under and over utilisation scenarios, while continuously identifying pockets for optimisation and corrections. For companies with customers spread across diverse geographies, it is important to have tools for infrastructure management, global analytics, database engines and application architectures across these global Kubernetes clusters and Virtual Machines.

The vast increase in attack surfaces and potential breach points have necessitated CIOs and CISOs to incorporating robust security principles and tools within the Cloud Native ecosystem itself, through Cloud Security Platforms such as Cloud Access Security Broker (CASB), Cloud security posture management (CSPM), Secure Access Service Edge (SASE), DevSecOps and incorporation of AI and ML in their proactive threat hunting and response systems. This is also critical in adhering to Governance, Risk and Compliance (GRC) and Regulatory compliances, in with Zero Trust Architecture and Cyber Resilient frameworks and strategy. This McKinsey article highlights the importance of Security as Code (SaC) in cloud native strategies and its reliance on architecture and the right automation capabilities.

This EY article highlights the importance of cybersecurity in cloud native strategies as well as the corresponding considerations in processes, cyber security tools, architecture, risk management, skills and competencies and controls. Data Encryption and Load Protection, Identity and Access management, Extended Threat and Response Systems (XDR), Security Incident and Environment Management (SIEM), and Security Orchestration and Response (SOAR) tools that incorporate AI/ ML capabilities ensure more of proactive vis--vis a reactive response. Considering the vast information to ingest, store and analyse, organisations are also considering/ deploying Cyber Data Lakes as either alternatives or in conjunction to complement their SIEM ecosystems.

There is an increasing popularity of Financial Operations (FinOps) which is helping organisations to gain the maximum value from the cloud, through the cross functional involvement of business, finance, procurement, supply chain, engineering, DevOps/ DevSecOps and cloud operational teams. Augmented FinOps has been listed by Gartner in the 2022 Hype Cycle for emerging technologies here. FinOps immensely value-adds to infrastructure and data management in the cloud through dynamic and continuous sourcing and managing cloud consumption, demand mapping, crystallising the Total Cost of Ownership and Operations with end-to-end cost visibility and forecasting to make joint decisions and monitor comprehensive KPIs. Besides the cloud infrastructure management strategies listed in this section, FinOps also incorporates vendor management strategies and leveraging cloud carbon footprint tools for their organisations Sustainability Goals.

What about Data Management through the Cloud?

The 2nd half of the 2010s and especially the COVID-19 period have also resulted in an implosion of IoT, social media, E-commerce and other Digital Transformation. This has made organisations deal with diverse data sources residing on cloud, on-premise and on the edge, diversity in data sets across sensor, text, image, Audio-Visual, Voice, E-commerce, social media and others, and the volume of data that is now required to be ingested, managed and delivered on real-time and batch mode. Even before the pandemic, this implosion of unstructured data, necessitated companies to leverage Hadoop and other Open Source based Data Lakes besides their structured data residing in Data Warehouses. According to this Deloitte Article, for their surveyed CXOs, Data Modernisation is even a more critical aspect than cost and performance consideration for migrating to the cloud.

This research by Statista estimated the total worldwide data amount rose from 9 Zettabytes in 2013 to over 27 Zettabytes in 2021, and the prediction is this growing to well over 180 Zettabytes in 2025. Decentralised and distributed cloud computing, Web 3.0, the Metaverse and rise in Edge Computing will further contribute to this data growth.

Many organisations are looking at the Cloud as the future of data management as this article by Gartner states. As the cloud encompasses more and more data sources, this becomes more pivotal for data architects to have a deeper understanding of metadata and schema, the end-to-end data lifecycle pipeline of ingestion, cleaning, storage, analysis, delivery and visualisation, APIs, cloud automation and orchestration, Data Streaming, AI/ ML models, Analytics, Data Storage and Visualisation, as well as Governance and Security.

Data Architects are hence leveraging cloud computing in their strategies including scaling, elasticity and decoupling, ensuring high availability and optimal performance with relation to bursts and shutdowns, while optimising cost at the same time. The Data Teams are also deploying Automated and Active Cloud Data management covering classification, validation and governance with extensible and decoupling. There is also careful consideration for ensuring security for data at rest and in motion, as well as seamless data integration and sharing

It is also important to choose the right MetaData Strategy and consider options of Tiered apps with push-based services, pull-based ETL, and Event based Metadata. It is also worthwhile to stress upon the importance of having a robust Data Architecture/ DataOps culture as well, especially considering business and technology perspectives of the end-to-end data lifecycle right from data sources and ingestion, meta data and active data management, streaming, storage, analytics and visualisation. Deploying elasticity, AI/ ML and automation bring about immense benefits to the cloud native strategies.

Considering these aspects in Data Management, organisations have looking at ML and API powered Data Fabrics along with Data Lakes, Warehouses and Layers to manage this end-to-end data lifecycle by creating, maintaining and providing outputs to the consumers of this data, as this Gartner article on technology trends for 2022 highlights.

This article by McKinsey summarises the major pivot points in the data architecture ethos which are fundamentally based on Cloud with containerization and serverless data. These cover hybrid real time and batch data processing, shift from end-to-end COTS applications to modular best in function/ industry, move to APIs and decoupling, shift from centralised Data Warehousing to domain-based architecture and lastly from proprietary predefined datasets to data schema that is light and flexible, especially the NoSQL family.

For BFSI, Telecoms and other industry verticals which need customer data to reside locally, CXOs have been deploying Hybrid Data Management environments, that leverage Cloud Data Management tools to also automate, orchestrate, and re-use the on-premise data, thus providing a unified data model and access interface to both cloud and on-premise datasets.

Application of Automation and Orchestration in Data Storage also ensures prioritisation of processes, tasks and resources to balance speed, efficiency, usage and cost along with eliminating security vulnerabilities. This is especially applicable for tasks such as provisioning and configuration, capacity management, workflows and data migration, resource optimisation, software updates and data protection and disaster recovery. This World Economic Forum report right before the pandemic highlighted the fact that the conventional optical/ magnetic storage systems will be unable to handle this phenomenon for more than a century. CIOs and Leaders are hence leveraging automation and cloud, Storage-as-a Service (STaaS), decentralised Blockchain powered data storage and storage on the Edge, besides alternates to conventional electromagnetic/ optical data storage mechanism

What is the role of people and culture in this cloud powered data and infrastructure management ecosystem?

People, talent pool and organisation culture play a pivotal part in successful FinOps, cloud native and cloud data management strategies. In this dynamic and uncertain world, it is of paramount importance to have uniformity, alignment and resonance of business KPIs to best practices for Enterprise and Data Architecture, DevOps as well as those of Engineering, Finance, and Procurement. This environment of continuous evolution, and optimisation can be only brought about by an ethos of Communication, Trust, Change Management, Business-Finance-IT alignment, which are equally important cloud native strategies, Architecture, DevOps, DataOps, Security and other Engineering Talent Pools.

The continuing trends of the Great Resignation, Quiet Quitting and Moonlighting necessitate a combination of having the best employee and vendor engagement strategies, a readily available talent pool of architects, analysts, engineers and other skillsets, as well as upskilling.

Winding up?

The Cloud has deeply impacted and revitalised Infrastructure and Data Management in all aspects in the workplace. As per this Deloitte research, it is ideal to leverage an equal mix of people, tools and approaches to address cloud complexity, and have a powerful, agile, elastic, secure and resilient virtual business infrastructure deriving maximum value from the cloud.

Cloud-centric digital infrastructure is a bedrock in the post COVID world, aligning technology with business to support digital transformation, resilience, governance along with business outcomes through a combination of operations, technology and deployment as mentioned in this IDC paper. This is so important in the increasing complexity of todays world across Public Cloud Infrastructure, On-Premises and on the Edge.

With continuing business uncertainty, competitiveness, customer, supplier and employee pressures and stringent IT budgets, organisations are looking at the Cloud to revitalise their Infrastructure and Data Management and gain maximum value.

See the article here:

Revitalising data and infrastructure management through cloud - ETCIO South East Asia