BankNext Case Study : ServiceMesh with Docker-K8-Istio-Sidecar
My scratch notes & experience on Docker, K8 & Istio combination
Getting started with Docker & Kubernetes can be daunting. At times, seemingly straight forward steps in the instruction list don’t seem to work as expected.
Reasons could be elusive. It could be your machine, your specific OS, your minikube version, color of your T-shirt (kidding..).
Now imagine adding the mysterious ServiceMesh/Istio to the mix.
After going through this trepidation, I jotted down the detailed steps which could potentially save someone this pain.
This end-end ServiceMesh platform can be found on my GitHub https://github.com/vijayredkar/kyc-docker-k8-istio
Goal
1- Add Istio ServiceMesh support to a set of microservices
2- Detail the steps required to make Istio work with Docker-minikube
3- Access this application over the external browser
4- Experience the immense flexibility this ServiceMesh adds to our architecture
Business Scenario
1- BankNext’s rapid digital transformation is supported by a huge volume of microservices which interact with each other
2- Know-Your-Customer is one such critical mSvc that interacts with Credit-Check-Basic-Svc to evaluate eligibility of new applicants
3- A newer & more sophisticated Credit-Check-Advanced-Svc is developed in line with the business demand for a more rigorous eligibility evaluation
4- Now the existing Know-Your-Customer mSvc needs to integrate with this new credit check svc to accomplish highest level of accuracy
Problem with Current Architecture
1- As always, business wants this integration to happen fast in view of the anticipated demand uptick in upcoming financial quarter
2- Since the newer credit check svc is unexplored yet in production, a canary deployment is desired so that the migration can happen in a controlled manner
3- Engineering team analyzes and realizes — Houston we have a problem!
4- This adaptation requires intrusive changes not only in application but also may involve gateway level routing changes
5- This implies that time consuming QA, regression and production deployment cycles will be needed
6- This implies that there is strong possibility of application downtime which will hurt the Tier 1 accreditation of this service
7- Inevitably, this type of exercise demands deliberate multi-team co-ordination and much more time
8- Moreover, BankNext has many such integrations planned in the roadmap and all these will face the same predicament as KYC
9- Business — Not happy!
Solution with New ServiceMesh Architecture Approach
1- kyc-aggregator-mgt today invokes the legacy kyc-credit-check-basic directly
2- kyc-aggregator-mgt needs flexibility via an external configuration approach to switch invocations between the advanced & basic versions of credit check
3- In future if a subsequent newer version of the credit check becomes available, then this approach should work for that too
4- Engineering team decides to incorporate the Istio ServiceMesh to meet all the above design challenges
5- Istio has the capability to deploy itself as a sidecar with our mSvcs
6- Istio has the capability to intercept and redirect any traffic coming to our mSvc
7- Istio accomplishes this via the configurations specified in the Gateway, VirtualService & DestinationRules ymls
8- Istio also has the capability to distribute the traffic load between services based off the weight specifications in the configuration
Downloads & Installations
1- Docker-
https://hub.docker.com/editions/community/docker-ce-desktop-windows
(Kinematic does not work anymore)
Docker version 20.10.7, build f0df350
2- Minikube — https://minikube.sigs.k8s.io/docs/start/
minikube version: v1.21.0
commit: 76d74191d82c47883dc7e1319ef7cebd3e00ee11
3- Istio- https://istio.io/latest/docs/setup/getting-started/#download
client version: 1.8.0
control plane version: 1.8.0
data plane version: 1.8.0 (5 proxies)
4- Oracle VirtualBox -
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
(20GB download- takes a long time)
VirtualBox-6.1.22–144080-Win
5- My Machine Specs -
Device name DESKTOP-D091LLJ
Processor Intel(R) Core(TM) i7–3632QM CPU @ 2.20GHz 2.20 GHz
Installed RAM 8.00 GB (7.88 GB usable)
Device ID F80CBB2E-4E2F-464A-B594-A428352C8B8E
Product ID 00330–80000–00000-AA124
System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display
Edition Windows 10 Pro
Version 20H2
Installed on 5/25/2021
OS build 19042.1348
Experience Windows Feature Experience Pack 120.2212.3920.0
6- Set your System PATH (for your reference — my PATH settings)
JAVA_HOME = C:\Vijay\Java\jdk-11.0.6
M2 = %M2_HOME%\bin
M2_HOME =C:\Vijay\Java\apache-maven-3.6.3
C:\Vijay\Java\apache-maven-3.6.3\bin
%JAVA_HOME%\bin
C:\Program Files\Docker\Docker\resources\bin
C:\ProgramData\DockerDesktop\version-bin
C:\Vijay\Java\Kubernetes\Minikube_2021
C:\Vijay\Java\Istio\istio-1.10.2\bin
C:\Vijay\Java\Istio\istio-1.10.2\bin\isoctl.exe
C:\Vijay\Java\Kubernetes\Minikube\kubectl.exe
C:\Vijay\Java\Kubernetes\Minikube\minikube.exe
Virtualization Enable (very imp setting)
1- To enable Virtualization in Windows 10
2- Navigate to
“Apps and Features” > Programs and Features > Turn Windows Features on or off → Control Panel\Programs\Programs and Features →Turn Windows Features on or off
3- Uncheck “Hyper-V or Windows Hypervisor platform” > OK
4- Restart Windows
5- When booting enter BIOS (usually “F2” press continuously)> Advanced > Virtualization > Enabled [update or verify that this is enabled. This may show up as other names like Vtx AMD]
6- Navigate to the saved location and launch the \Oracle\VirtualBox\VirtualBox.exe
Docker Launch
Launch Docker Desktop & ensure that it has fully started
C:\Program Files\Docker\Docker\Docker Desktop.exe
Console for running commands
Either DOS or GIT Bash in Administrator mode
I prefer using GIT Bash on Windows because it helps running the Linux type commands hassle free.
Detailed Implementation Steps
1- Clear out any previous inconsistent references (cautionary step to ensure removal of any previous run remnants)
$ minikube delete
! "minikube" profile does not exist, trying anyways.
* Removed all traces of the "minikube" cluster.$ minikube stop
E1112 13:02:54.085434 10628 daemonize_windows.go:38] error terminating scheduled stop for profile minikube: Error loading existing host. Please try running [minikube delete], then run [minikube start] again.: filestore "minikube": Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
* Profile "minikube" not found. Run "minikube profile list" to view all profiles.
To start a cluster, run: "minikube start"
2- Start Minikube afresh with the docker context
$ minikube start --driver=docker* minikube v1.21.0 on Microsoft Windows 10 Pro 10.0.19042 Build 19042
! Both driver=docker and vm-driver=virtualbox have been set.
Since vm-driver is deprecated, minikube will default to driver=docker.
If vm-driver is set in the global config, please run "minikube config unset vm-driver" to resolve this warning.
* minikube 1.24.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.24.0
* To disable this notice, run: 'minikube config set WantUpdateNotification false'
* Using the docker driver based on user configuration
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 2.3729385s
* Restarting the docker service may improve performance.
! This container is having trouble accessing https://k8s.gcr.io
* To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
* Preparing Kubernetes v1.20.7 on Docker 20.10.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Configuring RBAC rules ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: default-storageclass, storage-provisioner
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
3- Docker Login via Minikube (prompted for Docker credentials, if not already authenticated)
$ docker loginAuthenticating with existing credentials...
time="2021-11-12T13:09:16-08:00" level=error msg="(3a53acc8) a5b58191-CredentialHelperPKG C<-S No response POST /registry/credstore-updated (1.5370461s): Post \"http://ipc/registry/credstore-updated\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[[STACK]]"
time="2021-11-12T13:09:18-08:00" level=error msg="(6dc7cfbd) a5b58191-CredentialHelperPKG C<-S No response GET /ping (1.1539758s): Get \"http://ipc/ping\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[[STACK]]"
time="2021-11-12T13:09:21-08:00" level=error msg="(40232350) a5b58191-CredentialHelperPKG C<-S No response POST /registry/credstore-updated (1.0127004s): Post \"http://ipc/registry/credstore-updated\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[[STACK]]"Login Succeeded
4- Provide Docker environment context for Minikube
$ minikube docker-envexport DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:49962"
export DOCKER_CERT_PATH="C:\Users\user\.minikube\certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)
5- Istio Install
$ istioctl install --set profile=demo -y✔ Istio core installed
✔ Istiod installed
✔ Egress gateways installed
✔ Ingress gateways installed
✔ Installation complete*If any errors, re-run this cmd till you get the items checked as shown
6- Enable Istio Sidecar Deployment
$ kubectl label namespace default istio-injection=enablednamespace/default labeled
7- Istio Gateway IP
$ minikube ip! Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 3.0719554s
* Restarting the docker service may improve performance.192.168.49.2*This is your Istio Gateway IP
8- Istio Gateway Port
$ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'32716*This is your Istio Gateway Port.
9- Istio Gateway URL will be in the format
http://192.168.49.2:3271610- Adapt the kyc-aggregator-mgt svc to invoke this Istio Gateway instead of directly calling individual mSvc
Update \kyc-aggregator-mgt\src\main\resources\application.properties
istio-base-url=http://192.168.49.2:
istio-gateway-port=3271611- kyc-aggregator-mgt - build and create docker image
(rmi for cautionary cleanup step)
$ cd /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-aggregator-mgt$ docker rmi kyc-aggregator-mgt:latest
$ docker rmi -f kyc-aggregator-mgt:latest
$ docker rmi -f vijayredkar/kyc-aggregator-mgt:latest$ mvn clean install
$ docker build -t kyc-aggregator-mgt -f Dockerfile .
$ docker tag kyc-aggregator-mgt vijayredkar/kyc-aggregator-mgt:latest
$ docker push vijayredkar/kyc-aggregator-mgt
12- kyc-credit-check-basic -build and create docker image
(rmi for cautionary cleanup step)
$ cd /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-basic
$ docker rmi -f kyc-credit-check-basic:latest
$ docker rmi -f vijayredkar/kyc-credit-check-basic:latest$ mvn clean install
$ docker build -t kyc-credit-check-basic -f Dockerfile .
$ docker tag kyc-credit-check-basic vijayredkar/kyc-credit-check-basic:latest
$ docker push vijayredkar/kyc-credit-check-basic
13- kyc-credit-check-advanced -build and create docker image
(rmi for cautionary cleanup step)
$ cd /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-advanced
$ docker rmi -f kyc-credit-check-advanced:latest
$ docker rmi -f vijayredkar/kyc-credit-check-advanced:latest$ mvn clean install
$ docker build -t kyc-credit-check-advanced -f Dockerfile .
$ docker tag kyc-credit-check-advanced vijayredkar/kyc-credit-check-advanced:latest
$ docker push vijayredkar/kyc-credit-check-advanced
14- Docker Image Verifications
(ensure docker images are created & pushed — o/p for reference)
$ docker image lsREPOSITORY TAG IMAGE ID
kyc-credit-check-advanced latest f5694be8631d
vijayredkar/kyc-credit-check-advanced latest f5694be8631d
vijayredkar/kyc-credit-check-basic latest 49568e4da791
kyc-credit-check-basic latest 49568e4da791
kyc-aggregator-mgt latest 6ae3c011d1e3
vijayredkar/kyc-aggregator-mgt latest 6ae3c011d1e3
15- Clean up any previous references of Istio remnants.
(cautionary step to eliminate any unpredictable behavior)
$ kubectl delete service kyc-aggregator-mgt
$ kubectl delete deployment kyc-aggregator-mgt$ kubectl delete service kyc-credit-check-basic
$ kubectl delete deployment kyc-credit-check-basic$ kubectl delete service kyc-credit-check-advanced
$ kubectl delete deployment kyc-credit-check-advanced$ kubectl delete gateway kyc
$ kubectl delete virtualservice kyc
$ kubectl delete destinationrule kyc-aggregator-mgt$ kubectl delete gateway kyc-credit-check
$ kubectl delete virtualservice kyc-credit-check$ kubectl delete destinationrule kyc-credit-check-basic$ kubectl delete destinationrule kyc-credit-check-advanced
16- Deploy mSvcs with Istio support enabled
(routing rules, gateway, virtual services, destination rules)
$ cd /c/Vijay/Java/projects/kyc-k8-docker-istio/networking$ kubectl apply -f kyc-aggregator-mgt-k8-istio.yml
$ kubectl apply -f kyc-aggregator-mgt-istio-routing.yaml
$ kubectl apply -f kyc-aggregator-mgt-destination-rule.yaml$ kubectl apply -f kyc-credit-check-basic-k8-istio.yml
$ kubectl apply -f kyc-credit-check-basic-destination-rule.yaml$ kubectl apply -f kyc-credit-check-advanced-k8-istio.yml
$ kubectl apply -f kyc-credit-check-advanced-destination-rule.yaml$ kubectl apply -f kyc-credit-check-istio-routing.yaml
17- Verify the ServiceMesh setup
(ensure resources are created — o/p for your reference)
$ kubectl get podsNAME READY STATUS
kyc-aggregator-mgt-7f6ddc8bbd-sfjvg 2/2 Running
kyc-credit-check-advanced-799795b457-bh2p8 2/2 Running
kyc-credit-check-basic-d87477769-szd96 2/2 Running*wait until the pods show "2/2 Running"
$ kubectl get servicesNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
kyc-aggregator-mgt ClusterIP 10.105.20.233 <none> 8080/TCP
kyc-credit-check-advanced ClusterIP 10.102.235.90 <none> 8080/TCP
kyc-credit-check-basic ClusterIP 10.105.250.45 <none> 8080/TCP
$ kubectl get deploymentsNAME READY UP-TO-DATE AVAILABLE AGE
kyc-aggregator-mgt 1/1 1 1 37m
kyc-credit-check-advanced 1/1 1 1 37m
kyc-credit-check-basic 1/1 1 1 37m*ensure that deployments show "1/1 Ready"
$ kubectl get virtualservicesNAME GATEWAYS HOSTS AGE
kyc ["kyc"] ["*"] 26s
kyc-credit-check ["kyc"] ["*"] 19s
$ kubectl get gateways
NAME AGE
kyc 28s
$ kubectl get destinationrulesNAME HOST AGE
kyc-aggregator-mgt kyc-aggregator-mgt 25s
kyc-credit-check-advanced kyc-credit-check-advanced 9s
kyc-credit-check-basic kyc-credit-check-basic 18s
Testing KYC ServiceMesh
Port forwarding for browser access to kyc-aggregator-mgt inside ServiceMesh
$ export POD_NAME=$(kubectl get pods --no-headers -o custom-columns=":metadata.name" --selector app=kyc-aggregator-mgt)$ kubectl port-forward $POD_NAME 8080:8080URLs for browser access -
http://localhost:8080/kyc/v1/customerAlternately Swagger UI can be launched with -
http://localhost:8080/
Output & Benefits of KYC ServiceMesh Invocation
1- o/p shows 50–50% load distributed between credit-check-basic & credit-check-advanced
2- kyc-aggregator-mgt is completely oblivious to which which version of credit check is serving the request
3- kyc-aggregator-mgt simply invokes the URL “/credit-check/report”
4- The canary load distribution is managed entirely by our ServiceMesh’s VirtualService configuration in \networking\kyc-credit-check-istio-routing.yaml
5- In future, when credit-check-advanced is vetted well enough in production, we can simply change this weight to 100 in the VirtualService configuration
6- All the credit-check traffic will then flow to the new version, without any code change.
7- In future, if another newer version of credit check becomes available, the redirection to this version can be entirely managed in the VirtualService configuration, without any code change.
