VITech Software Development Life Cycle Manual

This document is created to represent our SDLC. Our development process is effective, modern, scalable, and lightweight. Our methods make us competitive compared with other top-performing companies, we can work fast without losing efficiency. We can deliver value to customers minimizing the failures and creating a blame-free environment. We keep our employees motivated, satisfied, and inspired by using modern and effective tools and processes. We save money by reducing the development period. We collect, share and reuse knowledge. We create an atmosphere of trust and constant self-improving.

Inspired by Accelerate

Overview

Current solution is built on top of Jenkins X.

Jenkins X aims to automate and accelerate Continuous Integration and Continuous Delivery for developers on the cloud and/or Kubernetes allowing them to focus on building awesome software instead of figuring out how to use the cloud or manually create and maintain pipelines.

Architecture

Simple view

Extended

CI/CD flow

Pull Request flow

Stack

Setup Cluster & Environment Repository

Prerequisites

After a successful login into GCP, the user should have full access to cluster resources including the ability to create IAM roles.

Setup

1. Create Infrastructure Repository.

Do not clone the created repository on your local machine, it will be done automatically by the install script.

2. Create Environment Repository.

3. Prepare install script.

install.sh
#!/usr/bin/env bash

set -e

export INFRA_REPO_NAME="demo-infra" (1)
export ENV_REPO_NAME="demo-environment" (2)

export BASE_REPO_URL="https://github.com/vitech-team"
export INFRA_GIT="$BASE_REPO_URL/$INFRA_REPO_NAME.git" (3)
export ENV_GIT="$BASE_REPO_URL/$ENV_REPO_NAME.git" (3)

export TF_VAR_jx_bot_username=XXX (4)
export TF_VAR_jx_bot_token=XXX (4)

export CLUSTER_NAME="demo-time" (5)
export GCP_PROJECT="XXX" (6)
export ZONE="europe-west1-c" (7)
export MIN_NODE_COUNT="4" (8)
export FORCE_DESTROY="false" (9)

export green="\e[32m"
export nrm="\e[39m"

if [ ! -d $INFRA_REPO_NAME ]; then
  git clone $INFRA_GIT
fi
if [ ! -d $ENV_REPO_NAME ]; then
  git clone $ENV_GIT
fi


cd "$ENV_REPO_NAME" || exit
git pull

jx gitops update

git add .

git commit -m "chore: gitops update from upstream"

git push

cd "../$INFRA_REPO_NAME" || exit

rm -f values.auto.tfvars
(10)
cat <<EOF >values.auto.tfvars
resource_labels = { "provider" : "jx" }
jx_git_url = "${ENV_GIT}"
gcp_project = "${GCP_PROJECT}"
cluster_name = "${CLUSTER_NAME}"
cluster_location = "${ZONE}"
force_destroy = "${FORCE_DESTROY}"
min_node_count = "${MIN_NODE_COUNT}"
EOF

git commit -a -m "fix: configure cluster repository and project" && git push || echo "Nothing to push"

terraform init
terraform apply

echo -e "${green}Setup kubeconfig...${nrm}"
gcloud components update
gcloud container clusters get-credentials "${CLUSTER_NAME}" --zone "${ZONE}" --project "${GCP_PROJECT}"

echo "Taling logs..."
jx admin log

echo -e "${green}Okay, now we are creating new key for service account...${nrm}"
gcloud iam service-accounts keys create keyfile.json --iam-account "${CLUSTER_NAME}-tekton@${GCP_PROJECT}.iam.gserviceaccount.com"
SECRETNAME=docker-registry-auth
kubectl create secret docker-registry $SECRETNAME \
  --docker-server=https://gcr.io \
  --docker-username=_json_key \
  --docker-email=sdlc@vitechteam.com \
  --docker-password="$(cat keyfile.json)" \
  --namespace=jx
kubectl label secret $SECRETNAME secret.jenkins-x.io/replica-source=true --namespace=jx

cd "../$ENV_REPO_NAME" || exit
git pull

echo -e "${green}Okay, now we need populate secrets which is related on other services...${nrm}"
echo -e "${green}Creating proxy to vault...${nrm}"
jx secret vault portforward &
sleep 3
jx secret verify
jx secret populate
jx secret edit -f sonar
jx secret edit -f oauth2-proxy

echo -e "${green}Killing proxy process...${nrm}"
kill %1

echo -e "For destroy open infrastructure folder and execute: ${green}terraform destroy${nrm}"

echo -e "For vault root token use: ${green}kubectl get secrets vault-unseal-keys  -n secret-infra -o jsonpath={.data.vault-root} | base64 --decode${nrm}"
1 Infrastructure repository name.
2 Environment repository name.
3 Infra. and env. repo URLs.
4 GitHub username and token. User should have settings permission to all repositories.
5 Cluster name to be created.
6 GCP project id.
7 Cluster zone: https://cloud.google.com/compute/docs/regions-zones#available.
8 Default node count.
9 If buckets and PVCs should be deleted in case of terraform destroy command.
10 Store cluster configs in file. For more configs, see: https://github.com/jx3-gitops-repositories/jx3-terraform-gke#terraform-inputs.

Put the script into the folder, where you want to clone your repositories, and execute it.

4. Populate secrets.

4.1 Create vault proxy.

First, we need to start the vault proxy

sec-vault-start.sh
jx secret vault portforward
4.2 Auto populate secrets
sec-auto-populate.sh
jx secret populate
4.3 Populate required secrets
sec-required-populate.sh
jx secret edit -f slack

jx secret edit -f snyk

jx secret edit -f sonar
Secrets can be populated via Vault UI as well: Vault.
4.4 Verify secrets

Execute jx secret verify and check, if all needed secrets are populated like: sonar, slack, etc…​

5. Create application based on SDLC quickstart.

5.1 Spring?

If you need some REST API backend service, use a template name: vitech-sdlc-backend.

quick-start-backend.sh
jx project quickstart --pack="spring-gradle"
5.2 Angular / React?

If you need a frontend application on Angular, use: vitech-sdlc-frontend.

quickstart-forntend.sh
jx project quickstart --pack="angular"

After setup, you need to edit default configs in the environments folder.

  • keycloak url: kubectrl get ingress -n keycloak.

  • change backend service name in nginx.conf.

Setup nexus npm proxy.

NPM_CONF="~/.npmrc"
USER="admin"
# possible to pull pass from k8s secret.
# secret: nexus
PASSWORD="admin123"

JSON_PATH="{.spec.rules[*].host}"
NEXUS_URL=$(kubectl get ingress -n jx nexus -o jsonpath="$JSON_PATH")
ENC_PASS="$(echo -n "${USER}:${PASSWORD}" | base64)"
echo "registry=${NEXUS_URL}" >> "$NPM_CONF"
echo "_auth=${ENC_PASS}" >> "$NPM_CONF"

Customization

Keycloak

If you need some auth server, in our case it’s Keycloak execute following commands inside environment repository folder.

  • add Keycloak chart

jx gitops helmfile add --chart=vitech-sdlc/keycloak --namespace=keycloak --repository=https://vitech-team.github.io/sdlc-charts
  • resolve main helmfile

jx gitops helmfile resolve
  • create new branch and create PR

  • wait for green pipelines

  • merge PR

Oauth access for Pipelines dashboard

Default setup is configured for GitHub, if you need change your auth provider check: https://jenkins-x.io/v3/admin/setup/ingress/oauth/

Pipelines

Pipelines catalog and pack

All shared tasks and packs are stored in: https://github.com/vitech-team/tekton-pipelines-catalog.

Packs

All tasks, packs and pipelines are managed by syncing to environment repository via Kpt.

For more information about tasks and pipelines, check Tekton docs.
For more information about pipelines on JX, see JX Pipeline Docs.

Large Tests

Currently, we have only Large Tests implementation based on WebdriverIO. We have added a few steps to release and pullrequest pipelines:

  • Check if the large test has been executed on Staging before promote it to Production environment.

  • Execute Large Tests after changes applied on Production environment.

Enable
  • Open .lighthouse/large-test/triggers.yaml and change: always_run: false, optional: false to true.

  • Open .lighthouse/jenkins-x/release.yaml and uncomment commented tasks: ` large-test-prepare-and-check` and large-test-execute.

    • Change large test image name property: LARGE_REPORTS_IMAGE.

    • Change your app URLs properties: APP_URL_STAGING, APP_URL_PRODUCTION. If you have more environments, just add additional property like: APP_URL_XXX.

Selenium

For selenium hub config, use:

  • charts/dev/largetests/values.yaml.gotmpl.

  • charts/dev/largetests/values.yaml.

Slack notifications

If you want to edit Large test execution message in slack, please open and add changes: charts/dev/secret/templates/slack-messages.yaml.

You can use following variables that can be populated/replaced: $idea, ${STATUS}, ${REPORT_URL}, ${DETAILS} and ${GIT_SHA}.

Keycloak

  • For Keycloak configuration, use: charts/dev/keycloak/values.yaml.gotmpl file in env. repository list of keycloak configs: https://github.com/codecentric/helm-charts/tree/master/charts/keycloak#configuration.

  • For default realm configuration in Vault, find keycloak-relams/efault-relam.json key and edit existed json.

    • If you need to add more realms, add new realm secret into charts/dev/keycloak/templates and configure in charts/dev/keycloak/values.yaml.

Metrics

Metrics chart kube-prometheus-stack.

  • For custom monitors and gradana dashboard, use the folder: charts/dev/prometheusmonitors/templates.

  • For metrics stack configuration, use:

    • charts/prometheus-community/kube-prometheus-stack/values.yaml.gotmpl.

    • charts/prometheus-community/kube-prometheus-stack/values.yaml.

Alertmanager

Configure Slack notifications
  • In Vault find alertmanager.yaml secret and replace ${SLACK_HOOK_URL} with your hook URL. Example: charts/prometheus-community/kube-prometheus-stack/secret-schema.yaml.

Runbook

Сluster delete
  • For the cluster, delete cd from your infra repository and execute terraform destroy.

Vault
  • For port forward Vault type: jx secret vault portforward. After that you can access Vault at: https://localhost:8200.

  • Vault root token can be found in secret: vault-unseal-keys, key: vault-root.

Keycloak
  • Keycloak url: kubectrl get ingress -n keycloak.

Principles

Trunk Based Development (Branching)

Trunk-based development (TBD) is a source-control branching model for software development where developers merge every new feature, bug fix, or other code changes to one central branch in the version control system. This branch is called “trunk”. TBD enables continuous integration – and, by extension, continuous delivery – by creating an environment where commits to trunk naturally occur multiple times daily for each programmer. This makes it easy to satisfy the “everyone on the development team commits to trunk at least every 24 hours” requirement of continuous integration and lays the foundation for the codebase to be releasable at any time, as it is necessary for continuous delivery and continuous deployment. Let’s have a look at the trunk-based development workflow.

Typically we use Scaled Trunk-Based Development.

Best Practices

  • We use only fast-forward merges to trunk.

  • We disallow pushing to trunk.

  • We write machine-readable commit messages.

  • We merge squash multiple changes in short life branches.

  • The best practice is to rebase master against short-lived branch to keep it up to date. We do NOT merge master back to short-lived branch and then again to trunk, to avoid multiple merge comments and confusing history.

  • Feature branches must be short-lived.

  • We work with a ticketing system, and add the ticket number to each commit.

  • We keep commit messages as concise as possible.

  • We avoid hotfixes. Fixing forward is better than rolling back. If we don’t want to include another feature that has been released, we disable it with feature flags. We want to keep the flow of updates and releases as short and constant as possible, the hotfix will only complicate your repo.

  • We have to understand git commands and internals before using graphic tools like git "Kraken". They can be very helpful, especially when looking at history graphs. However, we still need to know certain basic things like the difference between revert and reset.

  • We never commit secrets of any kind to the repository -EVER.

For more information, see here and here

Quality Gateway

The Quality Gateway is an organizational\infrastructure point for testing code. The purpose of a quality gateway is to trap incorrect code and bugs as early as possible to prevent getting the incorrect code from the local codebase into a shared codebase. To get through the quality gateway, the code must satisfy several Test (Test case). The tests are devised to make sure that each requirement meets business needs.

Quality Gates

  • Verification of component versions local.

  • Code Compilation local.

  • Small Tests local.

  • Static analysis local.

  • Medium Tests local.

  • Code Compilation remote.

  • Static analysis remote.

  • Medium Tests remote. Optional if acceptable by 10 minutes rule.

  • Large Tests remote.

Test (Test case)

The test case is a specification of the inputs, execution conditions, testing procedures, and expected results that define a single test to be executed to achieve a particular software testing objective. For example, to exercise a particular program path or to verify compliance with a specific requirement. Test cases underlie testing that is methodical rather than haphazard. A battery of test cases can be built to produce the desired coverage of the software being tested. Formally defined test cases allow the same tests to be run repeatedly against successive versions of the software, allowing for effective and consistent regression testing.

Google practices the language of the small, medium, and large tests, featuring scope over form, instead of marking between code, integration, and system testing. According to the book How Google Tests Software, we define three types of test:

  • Small Tests - covers a single unit of code in a completely faked environment. unit tests

  • Medium Tests - covers multiple and interacting units of code in a faked environment. integration, capability tests

  • Large Tests - covers any number of units of code in the real integrated environment close to production environment with real and not faked resources. E2E, Smoke, Sanity, Functional, NFR tests

Small Tests

Small tests execute the code within a single function or module. The focus is on typical functional issues, data corruption, error conditions, and off-by-one mistakes. Small tests are of short duration, usually running in seconds or less.

Small Tests are Unit Tests in testing terminology.

They are most likely written by an SWE, less often by a SETs, and hardly ever by TEs. Small tests usually require mocks and faked environments to run. (Mocks and fakes are stubs—substitutes for actual functions—that act as placeholders for dependencies that might not exist, are too buggy to be reliable, or too difficult to emulate error conditions.) [TEs](https://github.com/vitech-team/SDLC/wiki/Glossary) rarely write small tests but might run them when they are trying to diagnose a particular failure.

Small tests try to answer the following question: "Does this code do what it is supposed to do?"

Running of small tests is usually required during test build phase in Continuous Integration pipeline.

A test that doesn’t require dependency on external resources (file system, database, network, wiremocks, another OS process) is a small one.

Medium Tests

Medium tests are regularly automated and involve two or more interacting features. The focus is on testing the directly interaction between features, so-called nearest neighbor functions. SETs supports the development of medium tests as completed individual features early in the product cycle and SWEs is heavily involved in writing, debugging, and maintaining the actual tests. If a medium test fails or breaks, the developer takes care of it autonomously.

In a majority of cases Medium Tests reflect Integration Tests in testing terminology.

Later in the development cycle, TEs can execute medium tests either manually (if the test is difficult to automate) or with automation.

Medium tests try to answer the following question: "Does a set of near neighbor functions interoperate with each other the way they are supposed to?"

The following neighbor functions can be used for a specific function under test: another component, module, network interface, file system, database, message broker, storage, etc. In the majority of cases, medium tests rely on an external process running on the same host/VM/container. A good example of an external process is a docker service running on the same host/VM with a test-runner process, which can be utilized by testcontainers framework.

Medium Tests must be separated from Small Tests in a project structure. Running of medium tests is usually required during integration-test build phase in Continuous Integration pipeline. Test Coverage tools should have separate reports for Medium Tests.

It’s expected that medium tests shouldn’t run longer than 5-10 minutes. The majority of time is usually spent on a dependent process start, but once they are running - tests should complete fast.

Large Tests

Large tests are usually running over component(s) deployed to the environment by the same Continuous Deployment pipeline that deploys to production.

Large Tests can be reflected by the following test suites:

  • End-To-End.

  • Functional.

  • Load/Stress/Performance (NFR gates).

  • Security.

  • Smoke/Sanity.

  • Any other tests which are running over deployed components.

Large tests try to answer the following question: “Does the product operate the way a user expects (from functional and non-functional requirements perspective) and produce the desired results?”

Large Tests need more time to run than medium tests, they rely on a full PROD-like deployment alongside real (not stubbed/mocked) infrastructure services.

Running of Large Tests can be required in the following phases/places:

  • Pull Requests checks (if they are fast enough and overall PR time doesn’t go beyond ~15mins).

  • Functional Test Suite in Continuous Deployment pipeline (after-deployment step). If their run takes too long — it’s expected to have separate Smoke/Sanity test suite extracted for that purpose, and running of Functional tests can be executed by separate pipeline.

  • NFRs gate in Continuous Deployment pipeline. It’s expected that desired/existed application benchmarks have been already collected by performance tests and put as an NFR’s thresholds/gates. Metrics collected during NFR gate tests have to be trended in time.

Versions

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR - version when you make incompatible API changes.

  • MINOR - version when you add functionality in a backward compatible manner.

  • PATCH - version when you make backward compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Example:

1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.

Version change should be driven by commit messages. See examples: Conventional Commits.

Roles

  • SWE -Software Engineer.

  • SET -Software engineer in Testing is responsible for the complete design and maintenance of the test cases.

  • TEs -Test engineers.

Other

Connect to the existing cluster / JX.

gcloud auth login
gcloud auth application-default login
  • connect to cluster.

CLUSTER_NAME="DUMMY"
ZONE="DUMMY"
GCP_PROJECT="DUMMY"
gcloud container clusters get-credentials "${CLUSTER_NAME}" --zone "${ZONE}" --project "${GCP_PROJECT}"
  • switch to jx namespace.

jx ns jx

Gradle

To use the artifacts mirror locally, set following properties in .gradle.properties or in global gradle properties ~/.gradle/.gradle.properties:

  • artifacts.url

  • artifacts.user

  • artifacts.password

Current properties are applicable for projects generated by SDLC quickstart.

Local Git credentials

If you need to change local git user and token, do it in the file: ~/git/credentials.

Some tools