In software development what matters the most today is fast feedback cycles and flawless deliveries, which exactly match the customer’s requirements. Do you agree? The benefits of fast feedback can be observed on many levels.
Looking at the software product as a whole and its requirements, releasing working software to the customer early often reduces the gap between understanding where the value for the customer lies and how the development organization interprets it.
By enabling faster feedback in testing and integrating code, developers can find problems sooner and can react upon them quicker. This enables them to correct any errors in their code while they still have it in close memory, increasing their productivity. Less context switching between development tasks is needed as feedback gets faster.
Wondering where to find such a holistic solution?
Continuous integration (CI) is an enabler for these benefits on both levels – faster delivery to the customer and quicker feedback about code defects.
Adopting the process in your organization, you would be able to see amazing results in terms of reducing merge conflicts, ensuring frequent and flawless deliveries, increased quality and productivity and customer satisfaction.
While most organizations would have heard of CI and many must have been using it already, for those unfamiliar with it, let’s first understand the concept.
What is Continuous Integration (CI)?
In practice, continuous integration is achieved through a high level of automation of code compiling and testing activities. All changes made by the developers to the software should trigger an automated process to verify that the new changes successfully integrate with all existing code. Developers regularly check in their work to a central version control system, which is monitored for changes by the CI system. For each incremental piece of code, the CI system automatically checks the integrity of the software against preset criteria, and gives feedback on whether the increment meets the criteria.
Thus, the CI system enforces quality of all changes made to the software and alerts developers to issues right as they occur.
Continuous integration can also be extended to the point where software that passes all tests is automatically deployed to production. This extended case is an example of continuous delivery. Regardless of the fact that CI relies heavily on specialized tools and automated testing, it cannot be thought of simply as a set of technical tools. Successful adoption of CI requires conformance to certain practices and principles. Adopting the practice of CI means more than using CI software.
Establishing practice, selecting/ finalizing tools & reporting are formally called “building a CI pipeline”.
Prime Benefits of Continuous Integration
The benefits of adopting continuous integration are numerous.
Benefits to teams
- Debugging becomes easier, as errors can be associated to small increments in code. Fixes can be applied before a large number of errors start to accumulate, interfering with each other. Risks of a large integration phase towards the end of the project are averted, as software is kept constantly in a verified working state.
- Development feedback cycle times are reduced and frequent deliveries become possible. By merging code frequently, less merge conflicts arise, as individual code merges are less complex.
- High emphasis on build and test automation can contribute to increased quality and productivity. Also, a good automated test coverage means that new developers have a safer environment to learn in, which makes it easier for them to catch up to speed with the project.
- As developers are not needed to spend time in debugging, merging, deployment etc. this time can be utilized in ensuring quality & completing features ensuring value addition.
Benefits to management
- Reducing risks or errors, delivery delays and inadequate productivity.
- Reducing overhead expenses across the development and deployment process.
- Enhancing the reputation of the company by providing quality assurance testing and a foolproof delivery.
Benefits to customers
- The practice of CI is quite compatible with the lean notion of flow smooth value delivery to the customer. Because a continuously integrated product is constantly maintained in a demonstrable state, customers can easily evaluate its value at any time.
- With Agile and established CI, customers can have a shippable product at the end of any sprint.
- All the builds are delivered from an established pipeline thereby ensuring quality benchmarking and predictability.
- Bringing down GTM.
What All It Takes To Implement CI
A minimal implementation of a CI system has two logical components: a method to execute automated workflows, or CI builds, and a means to give feedback on the results of the workflows. You must anlayse and invest on Jenkins (for CI build system), SonarQube (for reporting), Selenium (for automated testing) and Zap (for security vulnerability testing).
For comprehensive analysis of code and reporting the same on SonarQube, you must use following plugins on Jenkins;
Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans of this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard.
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage
- Drupal Developer Plugin
A plugin to review code and run tests on Drupal.
- FindBugs Plugin
This plugin generates the trend report for FindBugs, an open source program which uses static analysis to look for bugs in Java code.
This plug-in generates reports static code violation detectors such as checkstyle, pmd, cpd, findbugs, codenarc, fxcop, stylecop and simian
For building the code, you need to use Gradle plugin, Maven plugin SBT plugin, MSBuild, XCodeBuild. For vulnerability scanning (web projects), resort to ZAProxy Plugin.
It would be a great idea to use the default quality profiles provided by SonarQube (per language). These profiles have been strengthened with additional profiles available for specific languages (like OClint, FindBugs, Android Lint, Drupal, PSR-2, Swift 7 SwiftLint). In course of time you will come across instances where we had to change the severity of rule as well as disable some rules in mentioned profiles. This will enable you to better gauge the code quality and be realistic with analysis reports.
A CI build should be fast to provide timely feedback during product development. On the other hand, testing should be comprehensive to give confidence on code quality. These two goals are conflicting in nature, because adding more tests increases the run time of the build.
You should split builds in CI build into multiple stages. For any given project the build pipeline could be collection of 1 to 4 builds that run in sequence. Together, these builds perform comprehensive verification of code, but alone they are fast enough so that feedback time to developers is kept to minimum as possible level. If any build stage fails, the complete pipeline is considered failed.
Feedback about the whole pipeline’s success can be given at the time of the first detected failure. If feedback for all stages is not desired, processing resources can be saved by not running any further build stages that would not change the success state of the pipeline. Running all stages may still be desirable, even if some stage fails to verify. Unless the failure of one stage prevents the running of later stages, the test results of later stages can provide additional information about the failure to the developer. This, in turn, can help in fixing the problem.
How To Do IT
You need to segregate the CI pipeline into
- Code analysis & reporting
- Automated Testing
- Scan vulnerability (for web projects)
If any of the build fails, it can lead to failure of the whole pipeline for the given change. As you are moving into continuous deployment using Ansible and Docker but are not yet there, triggering of task 3 and 4 is not automated.
The reports generated at various stages are accessible to stakeholders on SonarQube, Jenkins project dashboard and Emails.
Brace Up Yourself For The Following Challenges In CI
Adopting continuous integration is not easy, nor is it free of cost. Continuous integration is an integral part of continuous delivery, and as such there should be plan to implement continuous delivery via continuous integration.
A source management system – A CI pipeline cannot be created without an established source repository and an established source versioning process.
Test automation and testing technique – CI relies on automated testing, which can take considerable effort to implement at an acceptable level. Unstable tests, sometimes referred to as flaky tests, lower developers’ confidence in build results. They might even cause developers to ignore build results altogether, assuming problems to lie with the flaky tests when there in fact may be real problems with the code. The existence of manual testing steps may prove problematic, as it slows down the process of producing verified increments of software.
Efficient communication – Efficient communication is crucial for successful team work. In a CI environment, communication is an essential requirement. Effectively using the CI system and producing a working, integrated software products are both collaborative efforts that require efficient communication. The effects of lacking communication on the collaborative aspect of software integration are highlighted by Damian.
In studying reasons behind failed CI builds, they have traced failure back to communication gaps between developers whose code was involved in the broken integration. Another kind of communication takes place between the developer and the CI system. The CI system communicates with the developers in the form of build result feedback. Developers need to be aware of this feedback, and actively follow it for it to have any value to them. Also, the produced feedback should be understandable to the developers.
Change resistance – Old habits are hard to change. Developers may feel skeptical about the benefits of CI. The process of CI requires developers to submit code for others to see earlier than they might be used to, as changes are constantly integrated. The increased transparency in programming work can feel threatening to developers. Developers may feel hesitant to expose their code early on, in fear of being judged for its unfinished state. Also note resistance to change in management to prove problematic, possibly preventing the success of process improvement initiatives such as adopting CI.
Software Architecture – Software architectural decisions may prove to be a hindrance to efficient implementation of CI. To give an example, tight coupling of components may necessitate building all parts of a large system from scratch during CI builds, instead of building a small subsystem more closely related to the changes made. This in turn increases build time and thus limits the possible frequency of builds.
Domain Applicability And Size Or Complexity Of Product – Existing development processes in companies may not be suitable for the practices of CI, especially in companies that are working in close relation to hardware. Large size and high complexity of the product being developed may make CI process adoption difficult. Also, the speed at which deliverable releases of software can be made is necessarily slower in larger projects, already because of longer compile and test times. This slows down the feedback loop.
Understanding Of CI/CD Process – A shared understanding of the process across the whole organization is needed, as well as a high degree of expertise. Initially setting up the required practices and infrastructure demands deep knowledge and investment of resources. Developers require a good grasp of the pipeline to perform their daily work. Developers’ lack of awareness of the CI system can diminish its value altogether – if its feedback is not followed or its process not understood, what use is it? Furthermore, management and development goals need to be aligned, and this requires that the managers also understand the processes of CI and CD.
Requirement Definition – Requirement definition for software became more difficult when a CI process was adopted. Requirements had to be broken down to smaller increments to allow for more frequent integration, which added to the amount of prioritizing and decisions that needed to be done for each requirement. With smaller requirements, a desire to deliver visible value to the customer raised the question, whether very small increments were worth integrating at all.
Dependency Management – Because the process of CI requires that code is integrated in small increments, the number of integration necessarily is high. Multiple developers contribute to the integrated software product in parallel instead of a single effort to integrate components of code. This makes management of dependencies and coordinating their integration more difficult. Component interfaces need to be more clearly defined. Sources of integration errors may be harder to track because of more developers contributing to the integration of a software product. Find especially transitive dependencies to be often the reason for broken CI builds. Because transitive dependencies are not explicitly specified for a software module, the developer is less aware of their effects.
How To Overcame These Challenges
It is a great idea to use SVN, TFS for quite some time and establish a process around source control and versioning. This will facilitate the initiation of CI pipeline.
Agile teams – Agile brings about a lot of communication in teams. Teams with agile background can be catalysts for CI push. It is a good idea to start CI initiatives with teams running on agile projects.
Setting up of adhering to coding practices – The learning from CI are made part of learning/ training cycles & are formulated as coding practice. This way the learning from one project may contribute towards quality of another project & overall product quality is sustained.
Training on CI (tools, practice and results) – CI training and awareness sessions should be a part of training calendar. This will increase awareness levels by training members on working of CI pipeline. What results are on SonarQube and what to analyze.
Management support – Management has always supported CI initiative and have provided resources (hardware and human) for the CI cause.
Introducing continuous integration to the Software Development process changed the way software’s are developed. It requires effort from all team members, and a cultural shift in the organization. The changes required to establish CI are to be done gradually and all teams and stakeholders need to be on-boarded after awareness and training. Educating team members about the practice of continuous integration practice and building the automated tests suite is to be done systematically.
If you are looking for any help on building any digital solution for better customer or employee engagement, please contact us at [email protected]