How are IT leaders using automation technologies? What are the keys to success? Whats next? Our comprehensive guide shares advice from CIOs and IT automation experts

Why is IT automation a growing trend?

How can I get started with IT automation?

What best practices should we use for IT automation?

How can I make the case for IT automation within my organization?

What are some common mistakes with IT automation?

What about IT automation and security?

Whats next in IT automation?

Where can I learn more about IT automation?

In enterprise IT shops, automation has become more than a trend: It has turned into a necessity. Thats due to the current mix of technologies in use, the speed at which IT must run, and talent concerns.

On the technology side, cloud, containers, and microservices lead the list of factors driving increased use of automation. Containers and microservices help tame the complexity of hybrid cloud environments. As you scale up your use of containers and microservices, automation soon becomes a core need.

You could manually deploy and configure 50 or even 500 servers, but when you hit 5,000 servers that is no longer an option, says Ned Bellavance, director of cloud solutions at Anexinet. Automation gives the administrator tools to effectively deploy scalable workloads, without a commensurate increase in staff.

Thats where automation tools like Ansible and orchestration platforms like Kubernetes come to the rescue: They help IT leaders reap the potential of cloud, containers, and microservices in a scalable manner without constantly throwing new bodies at growing and evolving operational needs. (Read our full article, 5 factors fueling automation in IT now.)

The speed at which organizations now work, and the concurrent rise of the DevOps way of working, also fuel the need for automation. DevOps requires paying down technical debt, such as time-sucking manual patches, so people can focus on more important problems.

If adding manpower is the only way to grow your business, then scalability is a pipe dream."

The result of automation is scalability less effort per person to maintain and grow your IT environment," as Red Hat VP, Global Services John Allessio recently noted. If adding manpower is the only way to grow your business, then scalability is a pipe dream. Automation reduces your manpower requirements and provides the flexibility required for continued IT evolution. (See his full article, What DevOps teams really need from a CIO.)

Finally, theres the IT talent factor. As weve noted, If your engineers and other talent spend the bulk of their time on painful, manual tasks, are you getting their best work? Probably not, and thats a killer when youre constantly challenged to do more with less.

In the enterprise IT space, one of automations biggest use cases is ensuring businesses have the right people performing the right tasks, says Mark Kirstein, VP, products at BitTitan.

IT leaders need talent working on strategic priorities today, not drudge work that can be automated. Also, IT people whose companies keep them on drudge work are more likely to leave - for a more innovative environment where they can grow their careers. Automation, in this regard, can help with retention.

Perhaps you have already started down the road to automation and are trying to increase its use. Or, perhaps you are just beginning to transition from older, manual ways of working to more automated ones. One of the hardest tasks is knowing what task makes a good automation candidate and what doesnt.

Before you start automating, or automating more, make sure you are not automating a business or IT process that no longer makes sense, notes Red Hat chief technologist E. G. Nadhan. IT leaders working with automation must keep an eye on processes that can be sunsetted instead of automated.

Automation is an effective trigger to revisit existing processes and determine their applicability in todays market, he writes. While it is true that repeatable processes warrant automation, it is important to make sure theyre being executed the right way. Automating the wrong processes proliferates chaos.

After examining your processes, identify types of low-skill work that vacuum up your IT teams time.

As ServiceNow CIO Chris Bedi recently wrote, IT teams continue to spend too much of their time managing the day-to-day tasks involved in keeping the business up and running.

Bedi sought to free up more of his teams time for innovation after finding his team was

To be a 'no service' organization will take some time, effort, and a plan, Bedi writes. Service automation cant just be about quick wins and incremental improvement. Its about creating competitive advantage over the long run: Its a marathon, not a sprint. You wont jump from manual to machines completely managing every service in one step.

Start with end-to-end processes with a lot of structured tasks and where automation will alleviate the most workload for the IT team. Provisioning VMs, patching machines, and installing are good candidates. Automate these entire processes the provisioning, the management, the reporting, the scaling up and down. Learn from them, then tackle more.

Bedis team boosted its operations resources spent on innovation from about 25 percent to 43 percent and Bedi wants that number to keep rising. Our IT team now has the time to experiment with how the rest of the organization can benefit from intelligent automation applied to their business processes, Bedi notes.

Youll also need to audit IT workflows, identify the success metrics for your work, and help others in the organization help themselves using automation. For detailed advice, see the fullarticle:Getting started with automation: 6 tips.

As Bedis experience shows, automation success is far from automatic. Also, if you focus just on tools, youll soon see that people and process are key.

Here are some best practices to keep in mind:

Get buy-in from your team: Of course, this advice applies to any significant shift in IT strategy, but its particularly true with automation, which sometimes carries negative connotations, notes Ned Bellavance, director of cloud strategy at Anexinet. As an IT leader, youll need to explain how the automation strategy will benefit IT and the greater business, both at an organizational and individual job level.

Although you may already have some tools in mind, listen closely to what your team suggests. They are the ones who will be using it the most, and they wont use it if they don't believe it can do the job, Bellavance says. Some other team members may have misgivings about automation and the amount of work it will take. Put their fears to rest by identifying and making some quick wins that simplify the teams life.

Choose flexible tools: When selecting an ideal automation toolbox, I would focus on three key attributes: flexibility, simplicity, usability, Bellavance advises. Any automation tool should be flexible enough to cover 90 percent of your use cases.

Think long-term and begin incrementally: Don'ttryto tackle your entire software development life cycle (SDLC) or a similarly large workflow all at once.

Ellucian CIO Lee Congdon, following this strategy, has taken a measured approach to automation in his organization. As he writes, For us, the first step in automation is thinking about what we can consume as a service. As we rely more and more on partners to do the low-level and routine work, we can also rely on them to automate that work for us, rather than do it ourselves.

Secondly, as we shift more of these technology tasks to cloud providers, and we gain more time to work more closely with the business, were naturally seeing more business automation opportunities. We can see where business people are doing manual and repetitive tasks and find new ways to assist them with tools for automating processes. (See the full article,Ellucian CIO: How we took a measured approach to automation.)

Consider a dedicated automation lead on the IT team: Theywill champion efforts and document success over the long haul.

Insist on strong, clear documentation for automated processes:You dont want Rube Goldberg projects that are complex masquerading as simple.

Dont be afraid to pass on automation for projects where it doesnt fit. Possible example: Services that require significant customization or one-off deliveries, whether to different business units, partners, or external customers.

Automation sounds scary to some people, since it is often mentally associated with job loss, and tied to some lingering misconceptions. Helping people understand the what, why, and how of your companys automation strategy is a necessary step to achieving your goals.

We asked a variety of IT leaders for their advice on making the case for automation in your organization - up and down the ladder. Here are some tips:

Show people whats in it for them.Showpeople how your automation strategy will benefit them and their jobs. Will automating a particular process in the software pipeline mean fewer middle-of-the-night calls for team members? Will it enable some people to dump low-skill, manual tasks in favor of more strategic, higher-order work the sort that helps them take the next step in their career?

Convey whats in it for them, and how it will benefit clients and the whole company, advises Vipul Nagrath, global CIO at ADP. Compare the current state to a brighter future state, where the company enjoys greater stability, agility, efficiency, and security.

Paint a before-and-after picture to help people see the upside, he advises.

You want to paint a picture of the current state that people can relate to, Nagrath says. Present whats working, but also highlight whats causing teams to be less than agile. Then explain how automating certain processes will improve that current state.

Connect automation to specific business goals.The case for automation needs to be driven by a business demand signal, such as revenue or operating expense, says David Emerson, VP and deputy CISO at Cyxtera. No automation endeavor is self-justifying, and no technical feat, generally, should be a means unto itself, unless its a core competency of the company.

Break the plan into manageable pieces. This will help people digest it - and reduce skepticism. It will also give you some flexibility to tweak plans as you go.

Promote your success. Talk up your small wins. Eric Kaplan, CTO at AHEAD, says that the value small wins reveal can actually help you sharpen the big picture for people. Kaplan points to the value of individual and organizational time as an area everyone can connect with easily.

The best place to do this is where you can show savings in terms of time, Kaplan says. If we can accelerate the time it takes for the business to get what it needs, it will silence the skeptics. (For more, see the full article:IT automation: How to make the case.)

Like any IT effort, automation has its oops moments. Thankfully, use of automation has progressed to a point where you can learn plenty from the earlier mistakes of other people. Here are some common trip-ups to watch out for and avoid, as you automate more.

Botching your estimate of automation results: Vipul Nagrath, global CIO at ADP, says skimping on due diligence is a key automation pitfall to avoid. This is particularly true, Nagrath says, when it comes to defining the goals and expected outcomes of your automation plan.

Be realistic. If you overestimate the benefits, you risk not achieving your stated goals, Nagrath explains, adding that theres a flip side: If you underestimate the benefits, you end up underselling the program, and that can cause analysis paralysis. Carefully estimate what the automation effort is truly going to yield.

Thinking too narrowly about the benefits: You may have a particular IT task in your sights to automate, but you need to think broadly for the business. Probably the most common mistake I see today is myopia, says David Emerson, VP and deputy CISO at Cyxtera. Automation has so many benefits.

For example, you may be able to use automation to realize peripheral benefits such as reduced compliance audit complexity, improved security posture, and fewer manual controls and processes that hamper engineering output, he says.

Expecting too much in the first few weeks: One of the most common mistakes Bruno Attore, CTO and co-founder at Uru, has seen is under-estimating the difficulty of the early stage of implementation. Expect some trial-and-error as you iterate and optimize, he stresses.

If you are transitioning from a pure manual operations and QA process to an automated one, the first few weeks will be hard.

If you are transitioning from a pure manual operations and QA process to an automated one, the first few weeks will be hard, Attore says. It's important not to give up and keep pushing forward. In the end, everyone needs to know that this is not a flip of a switch, but instead, a continuous process that will only get better and better.

Other common mistakes includebelieving automation will fix a bad process, failing to ensure governance, and ignoring the culture change aspect. For much more detail on these issues, see our article:8 IT automation mistakes to avoid.

Since you will hear security worries from others in your organization (remember the early days of cloud security worries?), its important to be able to articulate how your move to automation will affect your security strategy.

The good news: Automation tools and their frequent partner, containers, can actually help improve your organizations security posture (as many organizations have found is true of cloud services).

Orchestration tools like Kubernetes not only help you manage container deployments at scale, but they also manage related security tasks. As Red Hat security strategist Kirsten Newcomer shared in a related podcast, You really want automation, orchestration to help manage which containers should be deployed to which hosts; monitoring host capacity; container discovery knowing which containers need to access each other; managing shared resources, and monitoring container health.

Newcomer encourages people to think of container security as having ten layers including both the container stack layers (such as the container host and registries) and container lifecycle issues (such as API management). (For complete details on the ten layers and how Kubernetes fits in, check out this whitepaper: Ten Layers of Container Security. )

At the same time, more organizations are baking security into each step of the development and operations pipeline (rather than bolting it on right before teams deploy code.) Some organizations call this approach DevSecOps.

Enterprises are finding ways to move security left in their application development lifecycles, Newcomer told us. Theyre adopting DevSecOps by integrating security practices, tooling, and automation throughout the CI/CD pipeline. (Read the full article: Why DevSecOps matters to IT leaders. )

Containerization can also better protect against some existing threats and help you react quickly to emerging security issues.

The good news is that most containers are stateless and replaceable, which makes it easy to roll out a newer version of the image across a deployment and improve your security posture quickly, Bellavance says. They should also be immutable, in that they are replaced rather than changed.

When youre dealing with IT technologies that involve this much change, its valuable to get a peek at whats coming in the not-so-distant future. Were monitoring several trends that IT leaders should keep on the radar screen.

Machine learning and AI will mature and play bigger roles:When it comes to machine learning, its still very early days for most organizations in terms of actual implementations. However, machine learning is expected to play a significant role in the next waves of IT automation.

With the data that is developed, automation software can make decisions that otherwise might be the responsibility of the developer, says Mehul Amin, director of engineering for Advanced Systems Concepts, Inc. For example, the developer builds what needs to be executed, but identifying the best system to execute the processes might be [done] by software using analytics from within the system.

Machine learning can also do things like enable automated systems to provision additional resources when necessary to meet timelines or SLAs, or retire resources when theyre no longer needed.

IT automation is moving towards self-learning, says Kiran Chitturi, CTO architect at Sungard Availability Services. Systems will be able to test and monitor themselves, enhancing business processes and software delivery.

Chitturi points to automated testing as an example; test scripts are already in widespread adoption, but soon those automated testing processes may be more likely to learn as they go, developing, for example, wider recognition of how new code or code changes will impact production environments.

Scripting and automation tools keep evolving:The way people view and use scripting or automation tools (sometimes referred to as configuration management tools) is evolving with greater use.

There are many processes in the data center environment that are repetitive and subject to human error, and technologies such as Ansible help to ameliorate those issues, says Mark Abolafia, chief operating officer at DataVision. With Ansible, one can write a specific playbook for a set of actions and input different variables such as addresses, etc., to automate long chains of process that were previously subject to human touch and longer lead times. (Want to learn more about this aspect of Ansible? Read the related article: Tips for success when getting started with Ansible. )

Also, the tools themselves continue to become more advanced.With advanced IT automation tools, developers will be able to build and automate workflows in less time, reducing error-prone coding, says Amin. These tools include pre-built, pre-tested drag-and-drop integrations, API jobs, the rich use of variables, reference functionality, and object revision history.

Automation opens new metrics opportunities:Automation paves the way for new ways to measure IT performance. As more and more development activities source control, DevOps pipelines, work item tracking move to the API-driven platforms the opportunity and temptation to stitch these pieces of raw data together to paint the picture of your organization's efficiency increases, says Josh Collins, VP of architecture at Janeiro Digital.

Collins thinks of this as a possible new development organization metrics-in-a-box.

Whether measuring individual resources or the team in aggregate, these metrics can be powerful but should be balanced with a heavy dose of context, Collins says. Use this data for high-level trends and to affirm qualitative observations not to clinically grade your team.

For more issues to watch, see the full article: Whats next in IT automation: 6 trends to watch.

Want more detailed advice on automation? Dig into these articles, papers, and Ebooks:

The Automated Enterprise (Red Hat Ebook)

Understanding automation

Tips for success when getting started with Ansible

Ansible in depth (Red Hat whitepaper)

DevOps metrics: Are you measuring what matters?

5 TED Talks on AI to watch

Want more wisdom like this, IT leaders? Sign up for our weekly email newsletter.

See more here:

Automation: The IT leader's guide | The Enterprisers Project