BankNext Case Study — mTLS with Istio-ServiceMesh

Enforce mutual TLS with a single configuration. Secure all your microservices without code change

BankNext’s MS architecture helped it rapidly bring features to market but now with 300+mSvcs in production, the team is struggling with the maintenance of these numerous services. BankNext’s commitment to the best security practices requires that it enforces TLS across all it’s services. Given the scale, this is turning out to be a daunting task.

Current Architecture & Challenges

1- To enforce TLS, BankNext needs to modify each of it’s 300+ mSvcs to not allow plain text traffic processing
2- First a public-private key pair needs to be created with Java keytool
3- Multiple SSL enabled attributes need to be added in the configuration of each of the 300+ Spring Boot mSvcs
server.ssl.enabled=true
• server.ssl.key-store-type=PKCS12
• server.ssl.key-store=classpath:keystore/keystore.p12
• server.ssl.key-store-password=*****
• server.ssl.protocol=TLS
• server.ssl.enabled-protocols=TLSv1.2

4- In every calling service “http” should be manually changed to “https”
5- Redeploy all these services in production for the TLS to take effect
6- Perform time consuming QA, regression & production deployments
7- High possibility of downtimes, hurting the Tier 1 accreditations
8- Potential of severe business loss
9- Engineering team in unison — Houston we have a problem!

Expectations from the Solution

1- Configuration driven capability to switch on/off mTLS across all Msvcs
2- Granular control to switch on/off mTLS on specific mSvcs
3- Current interactions between mSvcs is insecure & uses plain text
Challenge -1 : Enforce TLS interaction between kyc-aggregator-mgt and kyc-credit-check-advanced
Challenge -2 : Permit plain text interaction between kyc-aggregator-mgt and kyc-credit-check-basic
Challenge -3 : Enforce mTLS across all mSvcs in one step

New Architecture Capabilities w/ Istio ServiceMesh

mTLS_KYC_ServiceMesh_Istio_Arch
  1. The below mTLS implementation steps are a continuation of Istio ServiceMesh setup for KYC
  2. Start minikube cluster w/ Istio support
 minikube delete
minikube stop
minikube start --driver=docker
docker login
minikube docker-env
istioctl install --set profile=demo -y

3- kyc-aggregator-mgt — build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-aggregator-mgt
docker rmi kyc-aggregator-mgt:latest
docker rmi -f kyc-aggregator-mgt:latest
docker rmi -f vijayredkar/kyc-aggregator-mgt:latest$ mvn clean install

docker build -t kyc-aggregator-mgt -f Dockerfile .
docker tag kyc-aggregator-mgt vijayredkar/kyc-aggregator-mgt:latest

docker push vijayredkar/kyc-aggregator-mgt

4- kyc-credit-check-basic -build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-basicdocker rmi -f kyc-credit-check-basic:latest
docker rmi -f vijayredkar/kyc-credit-check-basic:latest$ mvn clean install
docker build -t kyc-credit-check-basic -f Dockerfile .
docker tag kyc-credit-check-basic vijayredkar/kyc-credit-check-basic:latest
docker push vijayredkar/kyc-credit-check-basic

5- kyc-credit-check-advanced -build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-advanceddocker rmi -f kyc-credit-check-advanced:latest
docker rmi -f vijayredkar/kyc-credit-check-advanced:latest$ mvn clean install
docker build -t kyc-credit-check-advanced -f Dockerfile .
docker tag kyc-credit-check-advanced vijayredkar/kyc-credit-check-advanced:latest
docker push vijayredkar/kyc-credit-check-advanced

6- Docker Image Verifications
(ensure docker images are created & pushed — o/p for reference)

docker image lsREPOSITORY                              TAG       IMAGE ID    
kyc-credit-check-advanced latest f5694be8631d
vijayredkar/kyc-credit-check-advanced latest f5694be8631d
vijayredkar/kyc-credit-check-basic latest 49568e4da791
kyc-credit-check-basic latest 49568e4da791
kyc-aggregator-mgt latest 6ae3c011d1e3
vijayredkar/kyc-aggregator-mgt latest 6ae3c011d1e3

7.Create kyc-credit-check-basic pod in basic namespace w/ Istio enabled

kubectl create ns basickubectl label namespace basic istio-injection=enabledkubectl apply -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-credit-check-basic-k8-istio-mtls.yml  -n basic

8. Create kyc-credit-check-advanced pod in advanced namespace w/ Istio enabled

kubectl create ns advancedkubectl label namespace advanced istio-injection=enabledkubectl apply -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-credit-check-advanced-k8-istio-mtls.yml  -n advanced

9. Create kyc-aggregator-mgt pod in consumer namespace without Istio

kubectl create ns consumerkubectl create -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-aggregator-mgt-k8-istio-mtls.yml -n consumer

10a. Check pod readiness

kubectl get pods -n basic -o wide
kubectl get services -n basic -o wide
kubectl get pods -n advanced -o wide
kubectl get services -n advanced -o wide
kubectl get pods -n consumer -o wide
kubectl get services -n consumer -o wide

10b. Expected output

NAME                                     READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATESkyc-credit-check-basic-d87477769-t5qms   2/2     Running   0          9m38s   172.17.0.6   minikube   <none>           <none>NAME                     TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE     SELECTORkyc-credit-check-basic   NodePort   10.102.169.141   <none>        8080:30276/TCP   9m39s   app=kyc-credit-check-basicNAME                                         READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATESkyc-credit-check-advanced-799795b457-r4dxv   2/2     Running   0          9m30s   172.17.0.7   minikube   <none>           <none>NAME                        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE     SELECTORkyc-credit-check-advanced   NodePort   10.108.8.139   <none>        8080:30469/TCP   9m33s   app=kyc-credit-check-advanced

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kyc-aggregator-mgt-7f6ddc8bbd-mgh6f 1/1 Running 0 9m20s 172.17.0.8 minikube <none> <none>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTORkyc-aggregator-mgt NodePort 10.96.57.158 <none> 8080:30778/TCP 9m23s app=kyc-aggregator-mgt

11. Open external access to kyc-aggregator-mgt via port forwarding

export POD_NAME=$(kubectl get pods -n consumer --no-headers -o custom-columns=":metadata.name" --selector app=kyc-aggregator-mgt)kubectl port-forward $POD_NAME 8082:8080 -n consumer

Challenge 1 : Solved
Enforce Strict TLS on kyc-credit-check-advanced w/ the below Istio configuration

  • PeerAuthentication Strict mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "pa-advanced"
namespace: "advanced"
spec:
selector:
matchLabels:
app: kyc-credit-check-advanced
mtls:
mode: STRICT
EOF
  • Destinationrule Istio Mutual mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "dr-advanced"
spec:
host: "kyc-credit-check-advanced.advanced.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF
  • Test the solution
    below attempt to access kyc-credit-check-advanced w/ http fails with ResourceAccessException (exit code 56) because only https is allowed
HTTP_Blocked_Specific_Msvc_With_Strict_Mtls

Challenge 2: Solved
Permit plain text traffic on kyc-credit-check-basic w/ the below Istio configuration

  • PeerAuthentication Permissive mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "pa-basic"
namespace: "basic"
spec:
selector:
matchLabels:
app: kyc-credit-check-basic
mtls:
mode: PERMISSIVE
EOF
  • Destinationrule Disable Mutual TLS
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "dr-basic"
spec:
host: "kyc-credit-check-basic.basic.svc.cluster.local"
trafficPolicy:
tls:
mode: DISABLE
EOF
  • Test the solution
    below attempt to access kyc-credit-check-basic w/ http succeeds because both http and https is allowed
HTTP_Permitted_Specific_Msvc_With_Permissive_Mtls

Challenge 3: Solved
Enforce mutual TLS across all the mSvcs w/ the below Istio configuration

  • PeerAuthentication Strict mode
-cleanup any existing rules
kubectl get PeerAuthentications --all-namespaces
kubectl delete PeerAuthentication default -n istio-system
kubectl delete PeerAuthentication pa-basic -n basic
kubectl delete PeerAuthentication pa-advanced -n advanced
kubectl get Destinationrules --all-namespaces
kubectl delete Destinationrule defaut-n istio-system
kubectl delete Destinationrule dr-basic -n basic
kubectl delete Destinationrule dr-advanced -n advanced
-apply mTLS rulekubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "default"
namespace: "istio-system"
spec:
mtls:
mode: STRICT
EOF
  • Test the solution
    attempt to access any mSvc w/ http fails with ResourceAccessException (exit code 56) because only https is allowed
HTTP_Blocked_All_Msvc_With_Strict_Mtls

Conclusion - Objectives Accomplished

1- Mutual TLS enforced across all mSvcs without any code change
2- Granular control on mTLS enforcement
3- Completely abstracted mSvcs w/ Istio managing mTLS transparently
4- Any new mSvc to production will use mTLS without any code change
5- Github KYC mTLS w/ Istio ServiceMesh

--

--

15+ years Java professional with extensive experience in Digital Transformation, Banking, Payments, eCommerce, Application architecture and Platform development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vijay Redkar

15+ years Java professional with extensive experience in Digital Transformation, Banking, Payments, eCommerce, Application architecture and Platform development