How Big Data and AI Can Help the Developer Toolchain

Developers today have a fantastic assortment of tools and technology available to them, which they use to construct the digital world around us. However, the sheer number of choices in the DevOps and CICD toolchains introduces a vast amount of complexity, which leads to multiple inefficiencies. Now a new discipline called developer productivity engineering (DPE) is emerging to tackle this problem, and advanced analytics and AI play big roles.

While the advent of DevOps and continuous integration, continuous deployment (CICD) methods has made developers lives simpler in many respects, they have also unleashed new forces that hinder developer productivity, according to Hans Dockter, the CEO of Gradle, the for-profit company behind the leading open source build tool of the same name.

For starters, testing is critical to ensure software is bug free and doesn’t pose a security risk. Thanks to build tools like Gradle, Apache Maven and Bazel, developers no longer need to manually execute the various convoluted steps required to push a new feature or bug fix into production. With more than 3,000 integrations maintained by its open source community, the Gradel build tool can eliminate much of that drudge work.

But for some large enterprise or Web applications, there may be 10,000 tests that need to be run before code can be promoted into production. That means that even for the smallest code changes, it may take 24 hours to run all the checks . Multiple that times a thousand developers or so, and you quickly run into a development quagmire.

“We have many companies where the wait time waiting for the tool chain adds up to multiple hours a day, pretty much,” Dockter says.

About 20% of the time, the tests will come back and alert the developer to a problem in the code. That’s a good thing, Dockter says. After all, you don’t want to deploy buggy code. But taking the next step is not always obvious or simple.

“For most engineers in the industry, it’s very hard to figure out the root cause,” he says. “They have a lack of data to understand, am I responsible for this problem? Is my colleague responsible for it?”

The third issue is that there’s no observability across the toolchain, Dockter says. When companies want to get some basic visibility into their developer’s day, such as how much code they’re pushing into production, they are typically forced to resort to manual methods, such as developer surveys.

“So all the machinery that developers are using, day after day, is causing them a lot of problems, is completely not observable,” Dockter says. “The industry that has made all the other industries so observable and yet it has no observability when it comes to its own machinery–it’s almost ironic.”

Gradle focuses on the developer experience (Image courtesy Gradle)

Irony makes for a lousy business model. But shining a light into the inefficiencies of the software development industry using connected data and machine learning has the potential to save companies billions of dollars, and could therefore be quite a lucrative one.

This is the basic premise behind Gradle’s enterprise products, as well as the DPE discipline as a whole. Gradle hosted the first annual Developer Productivity Engineering Summit in San Francisco in October, and Dockter reports that attendance exceeded expectations. Many of the companies at the cutting edge of software development, including Netflix and LinkedIn, participated in the DPE Summit.

The way Dockter sees it, DPE has the potential to transform software development by introducing the element of data-based rigor, engineering, and reproducibility, as other industries have already done.

“If you look at other industries, like chemical factories, they have the discipline of chemical process engineering. Chemistry has automation engineering. You can do a PhD in those disciplines,” he says.

Similarly, a car manufacturer likely has a better grasp on its parts supply chain than its software supply chain, even if both are critical to survival. “If I would go to one of your factories and someone would ask me how long it took to get part A from B to C, that would be absolutely not acceptable,” Dockter says.

Dockter sees big data and AI playing big roles in the future of DPE. The company has established a data science team, and rolled out the first AI-based product. Predictive Test Selection uses machine learning to predict which parts of the codebase are sensitive to change, and which tests can be safely excluded from the DevOps lifecycle.

“We can tell you, oh, you changed that part of the software. 9,000 of the 10,000 tests you don’t need to run, because we know from our data that those tests are completely insensitive to those areas of the code,” Dockter says. “Only 1,000 tests are sensitive to this area, so let’s only run 1,000 out of 10,000 tests.”

Advanced analytics and AI are critical to making sense of observability data, Dockter says, and Gradle will have more AI and analytics products to help customers soon.

“We have our first data science product. But they have many more to come,” he says. “We think at scale, only with advanced analytics and machine learning, can you really get the full benefits from that data for your developers.”

The company uses the data is collects from 3,000 community-managed integration points to surface insights about developers via KPIs and dashboards, which can help to inform management about the current state of application development. It’s all about transforming development into a data-driven discipline, he says.

Keeping developers happy is a priority at many companies. Many companies are reactive when it comes to managing development and development teams. They’re just waiting for problems to happen, and then dealing with them when they occur. That isn’t ideal, Dockter says

“Emotionally [there is] so much frustration in this area,” he says. “And because there is no data you can imagine the finger pointing all over the place.”

DPE could provide the mechanism by which managers of development teams can provide data-driven insights that describe the current state of her staff. That, hopefully, leads to more proactive decision-making, and happier, more productive developers.

“It’s a huge maturity step for the software industry, as you can imagine,” Docketer says. “We’re now making progress. We’re still not in paradise, but [to say] ‘We have 7% less flaky tests over the last four weeks’–it’s still way too many, but we’re making progress.”

OpenTelemetry Gains Momentum as Observability Standard

Companies Drowning in Observability Data, Dynatrace Says

The post How Big Data and AI Can Help the Developer Toolchain appeared first on Datanami.