CI/CD and DevOps are all about the processing of automation steps to build and secure software applications. Various tools need to be integrated tightly to make sure all of these steps work fluently together. These integration points use Rest APIs to communicate back and forth. This is seen from a system-to-system perspective. The developer perspective seeks integration points between technical tools and their collaboration counterparts. These include tools like Slack, Github, MS Teams, or Jira. All of them use webhooks and/or (custom) notifications. Once you integrate your supporting tools with your CI/CD or DevOps workflow, you boost your software development activities. Effective webhooks and notifications to optimize your DevOps workflow.
A quick intro about webhooks and notifications
To put the rest of the contents into perspective, the reader of this article should have a good understanding of what webhooks and notifications really are. The article on hookdeck.com provides a solid (technical) base on what they are. Simply said: webhooks are a way to send information from one system (the source) to another based (the destination) on an activity that happened at the source. The destination system receives the notification in a certain shape or form. Examples are a notification in a chat channel, an SMS, or an email.
Practical use cases
Many DevOps teams already use webhooks to send notifications to themselves or other systems. For example, send out a notification when their CI/CD build pipeline has finished. Or when their integration tests are finished. Another example would be to create an issue on a ticketing board. A final example could be to send a notification in case your Kubernetes-based applications scale out or when the number of Pods on your Virtual Machines reaches the maximum amount they can handle.
Problem area
The above-mentioned examples are great to notify DevOps team members and/or other persons, but sometimes there can be an information overflow. Since the number of notifications and alerts becomes a bit too much, people are distracted from their day-to-day work. Therefore, it would be good to optimize your webhooks and notifications so they become more context-specific and “business-driven” instead of purely “technology-driven”. Then notifications transform from purely informative to ones that lead to actions based on informed decisions. It optimizes your DevOps workflows and makes your organization more effective.
Create your tickets at the right place
It might be tempting to create a ticket or work item as soon as there is a build or release issue is discovered by the CI/CD pipeline. People might think that creating an issue and assigning it to the author might be the best case. There are more effective methods:
- Create a ticket and assign it to the Product Owner if it requires a big change that needs to be estimated. For example: in case an issue is found that requires refactoring of a specific challenging component. If the number of story points is too high, the Product Owner might decide to put it lower on the backlog.
- Automatically place important issues on the backlog based on the right priority. This means you must classify the type of issues and assign the right priority to them before invoking the Rest API of the ticketing system.
- Consider a notification that is about an external system: create the ticket at the ticketing board of the external party with a reference back to the calling project. This way, there is no initial communication needed between the two teams.
These tips help to direct tickets to the right place and to the right persons at the right moment. Avoid extra layers and overhead of sorting and duplicating tickets to dispatch them to the right persons. It all requires time and effort to overthink these options. After that, they also need to be implemented. Perhaps this is a great concept for the dedicated CI/CD pipeline team so every team can benefit from it.
Please respect my time!
Some DevOps teams who are experimenting with Webhooks send notifications to their chat channels on every build- successful or not. This might look good in the first phase, but once more build cycles happen every hour, the number of notifications quickly increases. It would be much more efficient to only send out notifications when you need to react to them. So this leads to narrowing down the total stream.
- Only send out notifications when the CI/CD build pipeline fails, perhaps just in some specific steps, not when there is a syntax error or a missing file. For example, only alert the team if unit tests are failing or when integration tests are broken, or when the targeting system has an error. Ignore successful builds on feature branches but DO get notified when the master branch is successful since this is the most important branch.
- Don’t send every message to the entire team. Suppose someone is testing out a new feature or he/she learns to explore the system, then you can introduce a flag “committer-only” that sends a message to the person who committed that code and no one else. This way, not everyone is disturbed when a bunch of notifications arrives in a relatively short time period.
- If you really want to trace all builds, it’s also an option to add an @-tag to every notification. This way, only the people that are tagged do receive a special notice and the others can just watch the stream of notifications/build trends a bit later.
And last but not least, include a configuration file that has a mapping with the team members and their days off. Suppose team member X is off on Wednesday, don’t include that person in Wednesday’s notifications.
Focus on true security-related issues
There might be no debate over fixing security issues. However, not every security issue is an immediate threat to an application or its precious data. So we can also narrow down on that topic.
- Highly critical security issues might require the intervention of the CISO or security expert team. So why not send out an alert to the DevOps team itself and also to the contact person of CISO of the team in charge? This helps to keep them aligned and they are also informed automatically in case other teams face the same problem(s). They can even help to review solutions brought in as a Pull Request. This requires an active team that has the practical skills to collaborate on the operational level.
- Send out notifications to the Product Owner about security issues that have been there for a long time (say 6+ months) and do not have a fix (yet). He/she can take action and start having meetings about risk acceptance or other (non-technical) remediation issues. Be sure to include the ticket number in your commit message so everyone got an understanding of the right context.
- First, look up the “false positive” database. This might be an internal database filled with CVEs that has been constructed based on previous builds. By filtering issues using this list, you greatly reduce the number of (invalid) tickets.
All of these ideas help to dedicate your time to the things that really matter.
Identify trends and attract the management
Organizations need to learn based on the knowledge and experience they gain. Therefore it would be good to put the spotlight on trends that span multiple projects and departments. Think of the following to pinpoint issues in an early stage:
- Build a CI/CD pipeline that scans other pipelines. If specific pipelines did not run for a long time on the master branch (this means there is no recent activity), then inform notify the Product Owner or the last developer that has committed a change. This requires a meta-data file in every project which should be structured identically as the other projects.
- Another case would be no recent runs of the master branch but with a lot of feature branches that actually do run. This means, the DevOps team does not merge their source code fast enough, and changes crop up. In the end, this leads to more time that is needed to resolve conflicts or to merge the source code later on.
- Spot CI/CD pipelines that run for very long during off-hours. This can mean a lot of (integration) tests or other blocking issues. Extra help might be the solution for this team.
Involve the management
- Create a dedicated management channel in your online collaboration tool. Publish trends and useful statistics every week so that everyone is informed. This way people can start discussions with each other if things are not solved or improved in time. It might lead to fewer discussions and less communication overhead plus it adds transparency to the organization of what is actually going on.
- Mark specific tasks as “pilot” or “Proof of Concept” and notify people who are the budget holders for these items. This way they can follow the outcomes of these new and innovative initiatives pretty quickly.
These concepts will hopefully provide more powerful insights to the management so they can react to trends and act accordingly.
Wrap up
Webhooks and notifications are a great way to inform team members and other persons about specific events. It helps to integrate various systems and it reduces the communications overhead to otherwise react to things manually. Due to the fast nature of DevOps, the number of notifications can quickly get out of control. The tips and tricks in this article helped to streamline and optimize your DevOps workflow so everything becomes a bit more effective and less distracting.
If you have questions related to this topic, feel free to book a meeting with one of our solutions experts, mail to sales@amazic.com.
