Open in app

Sign in

Write

Sign in

BankNext Case Study — mTLS with Istio-ServiceMesh

Vijay Redkar
6 min readNov 20, 2021

Enforce mutual TLS with a single configuration. Secure all your microservices without code change

BankNext’s MS architecture helped it rapidly bring features to market but now with 300+mSvcs in production, the team is struggling with the maintenance of these numerous services. BankNext’s commitment to the best security practices requires that it enforces TLS across all it’s services. Given the scale, this is turning out to be a daunting task.

Current Architecture & Challenges

1- To enforce TLS, BankNext needs to modify each of it’s 300+ mSvcs to not allow plain text traffic processing
2- First a public-private key pair needs to be created with Java keytool
3- Multiple SSL enabled attributes need to be added in the configuration of each of the 300+ Spring Boot mSvcs
• server.ssl.enabled=true
• server.ssl.key-store-type=PKCS12
• server.ssl.key-store=classpath:keystore/keystore.p12
• server.ssl.key-store-password=*****
• server.ssl.protocol=TLS
• server.ssl.enabled-protocols=TLSv1.2
4- In every calling service “http” should be manually changed to “https”
5- Redeploy all these services in production for the TLS to take effect
6- Perform time consuming QA, regression & production deployments
7- High possibility of downtimes, hurting the Tier 1 accreditations
8- Potential of severe business loss
9- Engineering team in unison — Houston we have a problem!

Expectations from the Solution

1- Configuration driven capability to switch on/off mTLS across all Msvcs
2- Granular control to switch on/off mTLS on specific mSvcs
3- Current interactions between mSvcs is insecure & uses plain text
• Challenge -1 : Enforce TLS interaction between kyc-aggregator-mgt and kyc-credit-check-advanced
• Challenge -2 : Permit plain text interaction between kyc-aggregator-mgt and kyc-credit-check-basic
• Challenge -3 : Enforce mTLS across all mSvcs in one step

New Architecture Capabilities w/ Istio ServiceMesh

mTLS_KYC_ServiceMesh_Istio_Arch
mTLS_KYC_ServiceMesh_Istio_Arch
  1. The below mTLS implementation steps are a continuation of Istio ServiceMesh setup for KYC
  2. Start minikube cluster w/ Istio support
 minikube delete
 minikube stop
 minikube start --driver=docker
 docker login
 minikube docker-env
 istioctl install --set profile=demo -y

3- kyc-aggregator-mgt — build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-aggregator-mgt
docker rmi kyc-aggregator-mgt:latest
docker rmi -f kyc-aggregator-mgt:latest
docker rmi -f vijayredkar/kyc-aggregator-mgt:latest$ mvn clean install
docker build -t kyc-aggregator-mgt -f Dockerfile .
docker tag  kyc-aggregator-mgt vijayredkar/kyc-aggregator-mgt:latest
docker push vijayredkar/kyc-aggregator-mgt

4- kyc-credit-check-basic -build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-basicdocker rmi -f kyc-credit-check-basic:latest
docker rmi -f vijayredkar/kyc-credit-check-basic:latest$ mvn clean install
docker build -t kyc-credit-check-basic -f Dockerfile .
docker tag  kyc-credit-check-basic vijayredkar/kyc-credit-check-basic:latest
docker push vijayredkar/kyc-credit-check-basic

5- kyc-credit-check-advanced -build and create docker image
(rmi for cautionary cleanup step)

cd  /c/Vijay/Java/projects/kyc-k8-docker-istio/kyc-credit-check-advanceddocker rmi -f kyc-credit-check-advanced:latest
docker rmi -f vijayredkar/kyc-credit-check-advanced:latest$ mvn clean install
docker build -t kyc-credit-check-advanced -f Dockerfile .
docker tag  kyc-credit-check-advanced vijayredkar/kyc-credit-check-advanced:latest
docker push vijayredkar/kyc-credit-check-advanced

6- Docker Image Verifications
(ensure docker images are created & pushed — o/p for reference)

docker image lsREPOSITORY                              TAG       IMAGE ID    
kyc-credit-check-advanced               latest    f5694be8631d
vijayredkar/kyc-credit-check-advanced   latest    f5694be8631d
vijayredkar/kyc-credit-check-basic      latest    49568e4da791
kyc-credit-check-basic                  latest    49568e4da791
kyc-aggregator-mgt                      latest    6ae3c011d1e3
vijayredkar/kyc-aggregator-mgt          latest    6ae3c011d1e3

7.Create kyc-credit-check-basic pod in basic namespace w/ Istio enabled

kubectl create ns basickubectl label namespace basic istio-injection=enabledkubectl apply -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-credit-check-basic-k8-istio-mtls.yml  -n basic

8. Create kyc-credit-check-advanced pod in advanced namespace w/ Istio enabled

kubectl create ns advancedkubectl label namespace advanced istio-injection=enabledkubectl apply -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-credit-check-advanced-k8-istio-mtls.yml  -n advanced

9. Create kyc-aggregator-mgt pod in consumer namespace without Istio

kubectl create ns consumerkubectl create -f /c/Vijay/Java/projects/kyc-k8-docker-istio/networking/kyc-aggregator-mgt-k8-istio-mtls.yml -n consumer

10a. Check pod readiness

kubectl get pods -n basic -o wide
kubectl get services -n basic -o widekubectl get pods -n advanced -o wide
kubectl get services -n advanced -o widekubectl get pods -n consumer -o wide
kubectl get services -n consumer -o wide

10b. Expected output

NAME                                     READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATESkyc-credit-check-basic-d87477769-t5qms   2/2     Running   0          9m38s   172.17.0.6   minikube   <none>           <none>NAME                     TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE     SELECTORkyc-credit-check-basic   NodePort   10.102.169.141   <none>        8080:30276/TCP   9m39s   app=kyc-credit-check-basicNAME                                         READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATESkyc-credit-check-advanced-799795b457-r4dxv   2/2     Running   0          9m30s   172.17.0.7   minikube   <none>           <none>NAME                        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE     SELECTORkyc-credit-check-advanced   NodePort   10.108.8.139   <none>        8080:30469/TCP   9m33s   app=kyc-credit-check-advanced

NAME                                  READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATESkyc-aggregator-mgt-7f6ddc8bbd-mgh6f   1/1     Running   0          9m20s   172.17.0.8   minikube   <none>           <none>NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE     SELECTORkyc-aggregator-mgt   NodePort   10.96.57.158   <none>        8080:30778/TCP   9m23s   app=kyc-aggregator-mgt

11. Open external access to kyc-aggregator-mgt via port forwarding

export POD_NAME=$(kubectl get pods -n consumer --no-headers -o custom-columns=":metadata.name" --selector app=kyc-aggregator-mgt)kubectl port-forward $POD_NAME 8082:8080 -n consumer

Challenge 1 : Solved
Enforce Strict TLS on kyc-credit-check-advanced w/ the below Istio configuration

  • PeerAuthentication Strict mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "pa-advanced"
  namespace: "advanced"
spec:
  selector:
    matchLabels:
      app: kyc-credit-check-advanced
  mtls:
    mode: STRICT
EOF
  • Destinationrule Istio Mutual mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: "dr-advanced"
spec:
  host: "kyc-credit-check-advanced.advanced.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL           
EOF
  • Test the solution
    below attempt to access kyc-credit-check-advanced w/ http fails with ResourceAccessException (exit code 56) because only https is allowed
1-HTTP_Blocked_Specific_Msvc_With_Strict_Mtls
HTTP_Blocked_Specific_Msvc_With_Strict_Mtls

Challenge 2: Solved
Permit plain text traffic on kyc-credit-check-basic w/ the below Istio configuration

  • PeerAuthentication Permissive mode
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "pa-basic"
  namespace: "basic"
spec:
  selector:
    matchLabels:
      app: kyc-credit-check-basic
  mtls:
    mode: PERMISSIVE
EOF
  • Destinationrule Disable Mutual TLS
cat <<EOF | kubectl apply -n advanced -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: "dr-basic"
spec:
  host: "kyc-credit-check-basic.basic.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE          
EOF
  • Test the solution
    below attempt to access kyc-credit-check-basic w/ http succeeds because both http and https is allowed
HTTP_Permitted_Specific_Msvc_With_Permissive_Mtls
HTTP_Permitted_Specific_Msvc_With_Permissive_Mtls

Challenge 3: Solved
Enforce mutual TLS across all the mSvcs w/ the below Istio configuration

  • PeerAuthentication Strict mode
-cleanup any existing rules
kubectl get PeerAuthentications --all-namespaces
kubectl delete PeerAuthentication default -n istio-system
kubectl delete PeerAuthentication pa-basic -n basic
kubectl delete PeerAuthentication pa-advanced -n advancedkubectl get Destinationrules --all-namespaces
kubectl delete Destinationrule defaut-n istio-system
kubectl delete Destinationrule dr-basic -n basic
kubectl delete Destinationrule dr-advanced -n advanced-apply mTLS rulekubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF
  • Test the solution
    attempt to access any mSvc w/ http fails with ResourceAccessException (exit code 56) because only https is allowed
HTTP_Blocked_All_Msvc_With_Strict_Mtls

Conclusion - Objectives Accomplished

1- Mutual TLS enforced across all mSvcs without any code change
2- Granular control on mTLS enforcement
3- Completely abstracted mSvcs w/ Istio managing mTLS transparently
4- Any new mSvc to production will use mTLS without any code change
5- Github KYC mTLS w/ Istio ServiceMesh

Microservices
Istio Service Mesh
Digital Transformation
Mutual Tls
Fintech

--

--

Vijay Redkar

Written by Vijay Redkar

72 Followers

15+ years Java professional with extensive experience in Digital Transformation, Banking, Payments, eCommerce, Application architecture and Platform development

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams