VITech Software Development Life Cycle Manual
This document is created to represent our SDLC. Our development process is effective, modern, scalable, and lightweight. Our methods make us competitive compared with other top-performing companies, we can work fast without losing efficiency. We can deliver value to customers minimizing the failures and creating a blame-free environment. We keep our employees motivated, satisfied, and inspired by using modern and effective tools and processes. We save money by reducing the development period. We collect, share and reuse knowledge. We create an atmosphere of trust and constant self-improving.
Inspired by Accelerate |
Overview
Current solution is built on top of Jenkins X.
Jenkins X aims to automate and accelerate Continuous Integration and Continuous Delivery for developers on the cloud and/or Kubernetes allowing them to focus on building awesome software instead of figuring out how to use the cloud or manually create and maintain pipelines. |
CI/CD flow
Pull Request flow
-
conventional commits scan - Commitsar
-
scan for secrets - Gitleaks
-
build - including Small Tests & Medium Tests
-
dependencies check - OWASP Dependency-Check
-
container build - using Paketo Buildpacks or Kaniko
-
static code analyse - sonar
-
preview environment creation - JX (preview folder in imported project)
Stack
-
Jenkins X - CORE
-
Tekton - CI/CD
-
Keycloak - Open Source Identity and Access Management
-
WebdriverIO - Large test framework
Setup Cluster & Environment Repository
Prerequisites
-
Create a git user for a bot (different from your own personal user account) and generate a personal access token, which will be used by Jenkins X to interact with git repositories.
-
Terraform CLI (version
== 13.5
). -
Jenkins X CLI (version
== 3.1
).-
Create file with GitHub credentials for JX: create file
~/git/credentials
with following content:https://<username>:<access_token>@github.com
-
-
kubectl (version
== 1.20
). -
helm (version
== 3
). -
-
Ensure that you have enabled the IAM Service Account Credentials API for the project. (Implement only the first item of the provided guide).
-
Login
gcloud auth application-default login
. -
Make sure, the user has full access to GCP resources.
-
-
Generate Sonar cloud token.
-
Generate Slack Incoming Webhooks
optional
. -
Generate Snyk token
if you have license
optional
. -
Create GitHub Oauth application for Pipeline Visualizer: https://jenkins-x.io/v3/admin/setup/ingress/oauth/#prerequisites
After a successful login into GCP, the user should have full access to cluster resources including the ability to create IAM roles. |
Setup
1. Create Infrastructure Repository.
Create infrastructure repository for GKE: https://github.com/jx3-gitops-repositories/jx3-terraform-gke/generate.
Do not clone the created repository on your local machine, it will be done automatically by the install script. |
2. Create Environment Repository.
Create environment repository: https://github.com/vitech-team/jx3-gke-vault/generate.
3. Prepare install script.
#!/usr/bin/env bash
set -e
export INFRA_REPO_NAME="demo-infra" (1)
export ENV_REPO_NAME="demo-environment" (2)
export BASE_REPO_URL="https://github.com/vitech-team"
export INFRA_GIT="$BASE_REPO_URL/$INFRA_REPO_NAME.git" (3)
export ENV_GIT="$BASE_REPO_URL/$ENV_REPO_NAME.git" (3)
export TF_VAR_jx_bot_username=XXX (4)
export TF_VAR_jx_bot_token=XXX (4)
export CLUSTER_NAME="demo-time" (5)
export GCP_PROJECT="XXX" (6)
export ZONE="europe-west1-c" (7)
export MIN_NODE_COUNT="4" (8)
export FORCE_DESTROY="false" (9)
export green="\e[32m"
export nrm="\e[39m"
if [ ! -d $INFRA_REPO_NAME ]; then
git clone $INFRA_GIT
fi
if [ ! -d $ENV_REPO_NAME ]; then
git clone $ENV_GIT
fi
cd "$ENV_REPO_NAME" || exit
git pull
jx gitops update
git add .
git commit -m "chore: gitops update from upstream"
git push
cd "../$INFRA_REPO_NAME" || exit
rm -f values.auto.tfvars
(10)
cat <<EOF >values.auto.tfvars
resource_labels = { "provider" : "jx" }
jx_git_url = "${ENV_GIT}"
gcp_project = "${GCP_PROJECT}"
cluster_name = "${CLUSTER_NAME}"
cluster_location = "${ZONE}"
force_destroy = "${FORCE_DESTROY}"
min_node_count = "${MIN_NODE_COUNT}"
EOF
git commit -a -m "fix: configure cluster repository and project" && git push || echo "Nothing to push"
terraform init
terraform apply
echo -e "${green}Setup kubeconfig...${nrm}"
gcloud components update
gcloud container clusters get-credentials "${CLUSTER_NAME}" --zone "${ZONE}" --project "${GCP_PROJECT}"
echo "Taling logs..."
jx admin log
echo -e "${green}Okay, now we are creating new key for service account...${nrm}"
gcloud iam service-accounts keys create keyfile.json --iam-account "${CLUSTER_NAME}-tekton@${GCP_PROJECT}.iam.gserviceaccount.com"
SECRETNAME=docker-registry-auth
kubectl create secret docker-registry $SECRETNAME \
--docker-server=https://gcr.io \
--docker-username=_json_key \
--docker-email=sdlc@vitechteam.com \
--docker-password="$(cat keyfile.json)" \
--namespace=jx
kubectl label secret $SECRETNAME secret.jenkins-x.io/replica-source=true --namespace=jx
cd "../$ENV_REPO_NAME" || exit
git pull
echo -e "${green}Okay, now we need populate secrets which is related on other services...${nrm}"
echo -e "${green}Creating proxy to vault...${nrm}"
jx secret vault portforward &
sleep 3
jx secret verify
jx secret populate
jx secret edit -f sonar
jx secret edit -f oauth2-proxy
echo -e "${green}Killing proxy process...${nrm}"
kill %1
echo -e "For destroy open infrastructure folder and execute: ${green}terraform destroy${nrm}"
echo -e "For vault root token use: ${green}kubectl get secrets vault-unseal-keys -n secret-infra -o jsonpath={.data.vault-root} | base64 --decode${nrm}"
1 | Infrastructure repository name. |
2 | Environment repository name. |
3 | Infra. and env. repo URLs. |
4 | GitHub username and token. User should have settings permission to all repositories. |
5 | Cluster name to be created. |
6 | GCP project id. |
7 | Cluster zone: https://cloud.google.com/compute/docs/regions-zones#available. |
8 | Default node count. |
9 | If buckets and PVCs should be deleted in case of terraform destroy command. |
10 | Store cluster configs in file. For more configs, see: https://github.com/jx3-gitops-repositories/jx3-terraform-gke#terraform-inputs. |
Put the script into the folder, where you want to clone your repositories, and execute it.
4. Populate secrets.
4.1 Create vault proxy.
First, we need to start the vault proxy
jx secret vault portforward
4.3 Populate required secrets
jx secret edit -f slack
jx secret edit -f snyk
jx secret edit -f sonar
Secrets can be populated via Vault UI as well: Vault. |
5. Create application based on SDLC quickstart.
5.1 Spring?
If you need some REST API backend service, use a template name: vitech-sdlc-backend
.
jx project quickstart --pack="spring-gradle"
5.2 Angular / React?
If you need a frontend application on Angular, use: vitech-sdlc-frontend
.
jx project quickstart --pack="angular"
After setup, you need to edit default configs in the environments
folder.
-
keycloak url:
kubectrl get ingress -n keycloak
. -
change backend service name in
nginx.conf
.
Setup nexus npm proxy.
How to setup Nexus proxy for NPM, see: https://blog.sonatype.com/using-nexus-3-as-your-repository-part-2-npm-packages.
NPM_CONF="~/.npmrc"
USER="admin"
# possible to pull pass from k8s secret.
# secret: nexus
PASSWORD="admin123"
JSON_PATH="{.spec.rules[*].host}"
NEXUS_URL=$(kubectl get ingress -n jx nexus -o jsonpath="$JSON_PATH")
ENC_PASS="$(echo -n "${USER}:${PASSWORD}" | base64)"
echo "registry=${NEXUS_URL}" >> "$NPM_CONF"
echo "_auth=${ENC_PASS}" >> "$NPM_CONF"
Customization
Keycloak
If you need some auth server, in our case it’s Keycloak execute following commands inside environment repository folder.
-
add Keycloak chart
jx gitops helmfile add --chart=vitech-sdlc/keycloak --namespace=keycloak --repository=https://vitech-team.github.io/sdlc-charts
-
resolve main helmfile
jx gitops helmfile resolve
-
create new branch and create PR
-
wait for green pipelines
-
merge PR
Oauth access for Pipelines dashboard
Default setup is configured for GitHub, if you need change your auth provider check: https://jenkins-x.io/v3/admin/setup/ingress/oauth/
Pipelines
Pipelines catalog and pack
All shared tasks and packs are stored in: https://github.com/vitech-team/tekton-pipelines-catalog.
Packs
All tasks, packs and pipelines are managed by syncing to environment repository via Kpt.
For more information about tasks and pipelines, check Tekton docs. |
For more information about pipelines on JX, see JX Pipeline Docs. |
Large Tests
Currently, we have only Large Tests implementation based on WebdriverIO.
We have added a few steps to release
and pullrequest
pipelines:
-
Check if the large test has been executed on Staging before promote it to Production environment.
-
Execute Large Tests after changes applied on Production environment.
Enable
-
Open
.lighthouse/large-test/triggers.yaml
and change:always_run: false
,optional: false
totrue
. -
Open
.lighthouse/jenkins-x/release.yaml
and uncomment commented tasks: ` large-test-prepare-and-check` andlarge-test-execute
.-
Change large test image name property:
LARGE_REPORTS_IMAGE
. -
Change your app URLs properties:
APP_URL_STAGING
,APP_URL_PRODUCTION
. If you have more environments, just add additional property like:APP_URL_XXX
.
-
Selenium
For selenium hub config, use:
-
charts/dev/largetests/values.yaml.gotmpl
. -
charts/dev/largetests/values.yaml
.
List of all configs: https://github.com/helm/charts/tree/master/stable/selenium#configuration.
Keycloak
-
For Keycloak configuration, use:
charts/dev/keycloak/values.yaml.gotmpl
file in env. repository list of keycloak configs: https://github.com/codecentric/helm-charts/tree/master/charts/keycloak#configuration. -
For default realm configuration in Vault, find
keycloak-relams/efault-relam.json
key and edit existed json.-
If you need to add more realms, add new realm secret into
charts/dev/keycloak/templates
and configure incharts/dev/keycloak/values.yaml
.
-
Metrics
Metrics chart kube-prometheus-stack.
-
For custom monitors and gradana dashboard, use the folder:
charts/dev/prometheusmonitors/templates
. -
For metrics stack configuration, use:
-
charts/prometheus-community/kube-prometheus-stack/values.yaml.gotmpl
. -
charts/prometheus-community/kube-prometheus-stack/values.yaml
.
-
Alertmanager
Configure Slack notifications
-
In Vault find
alertmanager.yaml
secret and replace${SLACK_HOOK_URL}
with your hook URL. Example:charts/prometheus-community/kube-prometheus-stack/secret-schema.yaml
.
Runbook
Vault
-
For port forward Vault type:
jx secret vault portforward
. After that you can access Vault at: https://localhost:8200. -
Vault root token can be found in secret:
vault-unseal-keys
, key:vault-root
.
Principles
Trunk Based Development (Branching)
Trunk-based development (TBD) is a source-control branching model for software development where developers merge every new feature, bug fix, or other code changes to one central branch in the version control system. This branch is called “trunk”. TBD enables continuous integration – and, by extension, continuous delivery – by creating an environment where commits to trunk naturally occur multiple times daily for each programmer. This makes it easy to satisfy the “everyone on the development team commits to trunk at least every 24 hours” requirement of continuous integration and lays the foundation for the codebase to be releasable at any time, as it is necessary for continuous delivery and continuous deployment. Let’s have a look at the trunk-based development workflow.
Typically we use Scaled Trunk-Based Development.
Best Practices
-
We use only fast-forward merges to trunk.
-
We disallow pushing to trunk.
-
We write machine-readable commit messages.
-
We merge squash multiple changes in short life branches.
-
The best practice is to rebase master against short-lived branch to keep it up to date. We do NOT merge master back to short-lived branch and then again to trunk, to avoid multiple merge comments and confusing history.
-
Feature branches must be short-lived.
-
We work with a ticketing system, and add the ticket number to each commit.
-
We keep commit messages as concise as possible.
-
We avoid hotfixes. Fixing forward is better than rolling back. If we don’t want to include another feature that has been released, we disable it with feature flags. We want to keep the flow of updates and releases as short and constant as possible, the hotfix will only complicate your repo.
-
We have to understand git commands and internals before using graphic tools like git "Kraken". They can be very helpful, especially when looking at history graphs. However, we still need to know certain basic things like the difference between revert and reset.
-
We never commit secrets of any kind to the repository -EVER.
Quality Gateway
The Quality Gateway is an organizational\infrastructure point for testing code. The purpose of a quality gateway is to trap incorrect code and bugs as early as possible to prevent getting the incorrect code from the local codebase into a shared codebase. To get through the quality gateway, the code must satisfy several Test (Test case). The tests are devised to make sure that each requirement meets business needs.
Quality Gates
-
Verification of component versions
local
. -
Code Compilation
local
. -
Small Tests
local
. -
Static analysis
local
. -
Medium Tests
local
. -
Code Compilation
remote
. -
Static analysis
remote
. -
Medium Tests
remote
. Optional if acceptable by 10 minutes rule. -
Large Tests
remote
.
Test (Test case)
The test case is a specification of the inputs, execution conditions, testing procedures, and expected results that define a single test to be executed to achieve a particular software testing objective. For example, to exercise a particular program path or to verify compliance with a specific requirement. Test cases underlie testing that is methodical rather than haphazard. A battery of test cases can be built to produce the desired coverage of the software being tested. Formally defined test cases allow the same tests to be run repeatedly against successive versions of the software, allowing for effective and consistent regression testing.
Google practices the language of the small, medium, and large tests, featuring scope over form, instead of marking between code, integration, and system testing. According to the book How Google Tests Software, we define three types of test:
-
Small Tests - covers a single unit of code in a completely faked environment.
unit
tests -
Medium Tests - covers multiple and interacting units of code in a faked environment.
integration
,capability
tests -
Large Tests - covers any number of units of code in the real integrated environment close to production environment with real and not faked resources.
E2E
,Smoke
,Sanity
,Functional
,NFR
tests
Small Tests
Small tests execute the code within a single function or module. The focus is on typical functional issues, data corruption, error conditions, and off-by-one mistakes. Small tests are of short duration, usually running in seconds or less.
Small Tests are Unit Tests in testing terminology.
They are most likely written by an SWE, less often by a SETs, and hardly ever by TEs. Small tests usually require mocks and faked environments to run. (Mocks and fakes are stubs—substitutes for actual functions—that act as placeholders for dependencies that might not exist, are too buggy to be reliable, or too difficult to emulate error conditions.) [TEs](https://github.com/vitech-team/SDLC/wiki/Glossary) rarely write small tests but might run them when they are trying to diagnose a particular failure.
Small tests try to answer the following question: "Does this code do what it is supposed to do?"
Running of small tests is usually required during test build phase in Continuous Integration pipeline.
A test that doesn’t require dependency on external resources (file system, database, network, wiremocks, another OS process) is a small one. |
Medium Tests
Medium tests are regularly automated and involve two or more interacting features. The focus is on testing the directly interaction between features, so-called nearest neighbor functions. SETs supports the development of medium tests as completed individual features early in the product cycle and SWEs is heavily involved in writing, debugging, and maintaining the actual tests. If a medium test fails or breaks, the developer takes care of it autonomously.
In a majority of cases Medium Tests reflect Integration Tests in testing terminology.
Later in the development cycle, TEs can execute medium tests either manually (if the test is difficult to automate) or with automation.
Medium tests try to answer the following question: "Does a set of near neighbor functions interoperate with each other the way they are supposed to?"
The following neighbor functions can be used for a specific function under test: another component, module, network interface, file system, database, message broker, storage, etc. In the majority of cases, medium tests rely on an external process running on the same host/VM/container. A good example of an external process is a docker service running on the same host/VM with a test-runner process, which can be utilized by testcontainers framework.
Medium Tests must be separated from Small Tests in a project structure. Running of medium tests is usually required during integration-test build phase in Continuous Integration pipeline. Test Coverage tools should have separate reports for Medium Tests.
It’s expected that medium tests shouldn’t run longer than 5-10 minutes. The majority of time is usually spent on a dependent process start, but once they are running - tests should complete fast.
Large Tests
Large tests are usually running over component(s) deployed to the environment by the same Continuous Deployment pipeline that deploys to production.
Large Tests can be reflected by the following test suites:
-
End-To-End.
-
Functional.
-
Load/Stress/Performance (NFR gates).
-
Security.
-
Smoke/Sanity.
-
Any other tests which are running over deployed components.
Large tests try to answer the following question: “Does the product operate the way a user expects (from functional and non-functional requirements perspective) and produce the desired results?”
Large Tests need more time to run than medium tests, they rely on a full PROD-like deployment alongside real (not stubbed/mocked) infrastructure services.
Running of Large Tests can be required in the following phases/places:
-
Pull Requests checks (if they are fast enough and overall PR time doesn’t go beyond ~15mins).
-
Functional Test Suite in Continuous Deployment pipeline (after-deployment step). If their run takes too long — it’s expected to have separate Smoke/Sanity test suite extracted for that purpose, and running of Functional tests can be executed by separate pipeline.
-
NFRs gate in Continuous Deployment pipeline. It’s expected that desired/existed application benchmarks have been already collected by performance tests and put as an NFR’s thresholds/gates. Metrics collected during NFR gate tests have to be trended in time.
Versions
Given a version number MAJOR.MINOR.PATCH
, increment the:
-
MAJOR - version when you make incompatible API changes.
-
MINOR - version when you add functionality in a backward compatible manner.
-
PATCH - version when you make backward compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
Example:
1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.
Version change should be driven by commit messages. See examples: Conventional Commits.
Other
Connect to the existing cluster / JX.
-
download JX CLI/client which is defined by env repository.
-
login into GC cloud.
gcloud auth login
gcloud auth application-default login
-
connect to cluster.
CLUSTER_NAME="DUMMY"
ZONE="DUMMY"
GCP_PROJECT="DUMMY"
gcloud container clusters get-credentials "${CLUSTER_NAME}" --zone "${ZONE}" --project "${GCP_PROJECT}"
-
switch to
jx
namespace.
jx ns jx
Gradle
To use the artifacts mirror locally, set following properties in .gradle.properties
or in global gradle properties ~/.gradle/.gradle.properties
:
-
artifacts.url
-
artifacts.user
-
artifacts.password
Current properties are applicable for projects generated by SDLC quickstart. |
Local Git credentials
If you need to change local git user and token, do it in the file: ~/git/credentials
.