Insights

Addressing the CI/CD paradox: What experts say

UST Cloud team

UST participated in this year's cdCon + GitOpsCon, a two-day conference hosted by cd.foundation and Cloud Native Computing Foundation in Vancouver, British Columbia, Canada. UST's Global Head of Cloud Advisory spoke on a panel covering current topics in continuous integration and continuous deployment/delivery (CI/CD). Alongside Clark was Garima Bajpai from the Canada DevOps Community of Practice and Carl Coryell-Martin, CTO at Delving; the panel was moderated by Gautham Pallapa from VMware.

Given the setting of this conversation, it was no surprise that the panel hosted an animated discussion around some of the most prescient topics in contemporary DevOps, especially CD. While each member of this panel brought with them considerable expertise, the tone of the conversation was generally broad and philosophical. These industry veterans were able to compare and contrast some of the more introspective questions on what makes for good metrics of CD performance, how to achieve the best continuous integration and delivery/deployment, and how to make your teams' work – and lives – the best possible.

The panel started with a question for Bajpai, given her background in the greater Canadian DevOps community, asking 'what are some common mistakes teams make when adopting continuous delivery, and how can they be avoided?'

Bajpai responded on a note that would remain constant throughout the discussion, stating that in the hierarchy of priorities, teams often put tech above people. While, of course, technology is the raison d'etre for any CI/CD pipeline, it is people who build it and the people whom it serves. When teams lose track of their business goals or an organization's greater goals, the technology is distant from this understanding.

Clark chimed in with a keen insight that the technology is important as well, precisely for these reasons. CI/CD pipelines should be "declarative and automated" to the greatest extent possible"; according to Clark, pipelines that are not self-sufficient and require developers to handle minute tasks like building images are failing their purpose.

In the way that a pipeline that does not offer the value intended is a failed investment, Coryell-Martin added that when teams do not build pipelines with security in mind, they also fail from an investment perspective. If, at the end of building the pipeline, developers realize they need to add security features, they usually need to get approval from teams under the CFO's purview. Given the cyclical nature of corporate budgeting, this key feature often gets bumped until the next budget since, from the finance team's perspective, it is an additional cost, yet from the DevSecOps vantage point, it is a core competency.

The conversation between the panelists never lulled, and from this topic, with the help of Pallapa's deft moderation, the group seamlessly pivoted into an exploration of the value of measuring the "speed" of continuous delivery pipelines.

The group generally agreed that speed is not the best success metric for CD teams, as it does not convey a logical link back to the business. Coryell-Martin distinguished speed from a vector, which has both velocity and direction, highlighting this as a more comprehensive way of assessing progress.

Bajpai added by saying that CD teams need time to fail, and an obsession with speed can deny the valuable experience of reflecting on mistakes and correcting them. Pallapa put it succinctly: "Fail often, fast, and cheap." This is the recipe for bringing the most thoroughly built product to market and for building the most comprehensive CI/CD pipeline.

UST's Clark shifted the conversation poignantly by assessing the value between "how many times you deploy versus the quality of your deployments." This led Pallapa to ask the group what are the best metrics for assessing CI/CD performance: is it adherence to DORA metrics? Percentage of automation in a pipeline? The number of deployments?

Clark's experience with helping large-scale organizations adopt comprehensive CI/CD practices was the first to respond. According to Clark, there is no one answer; "what are the problems you're trying to solve as a business?" This question, Clark was adamant, is the starting point for determining the metrics needed.

"If your OKR is the number of times we need to deploy… you'll have a quality problem." The common divisions between the business team and developers lead to a problem on a larger scale. When the OKR is simply the number of deployments, Clark stated, you'll see people find ways to inflate the deployments. "People behave as you incentivize them to behave," meaning that workers just looking to check boxes, get paid, and go home will fulfill the OKR presented. But to Clark's point, "How often you release a piece of crap doesn't tell me anything."

Bajpai reinforced this point by adding that "context is everything." She broke down the stakeholders in CI/CD into "producers, consumers, and practitioners." She highlighted that understanding how each of these stakeholders played into the construction of a CI/CD pipeline would allow teams to build the best metrics. In that vein, teams need to go beyond DORA": it's time for "the next generation of metrics."

Coryell-Martin compared DORA to brushing your teeth. As a general hygiene metric, it's helpful, but it doesn't answer the question of if you are building a happy life. Similarly, the number of deployments can be helpful, but it needs to inform if the product being built is really in line with an organization's needs and goals.

Clark added a final appraisal of this situation that CI/CD pipelines can add enormous returns, but to build useful automation in CI/CD, working in small steps in the early stages of development allows teams to see what is broken and fix it before organizations run into major issues.

After a little more discussion, Clark added, "The most important thing is connecting things back to the business." Technology needs to make lives and work easier, and that benefit needs to push things in a vector in line with the business goals. The work going into building CI/CD should always answer the guiding question: "why we're doing this."

To watch the complete panel discussion held at cdCon + GitOpsCon between these experts, click here.