BankNext Case Study - AI & LLM to solve GDPR challenge

GenAi with local LLM server & Java to tactfully solve the PII/PCI/GDPR challenge.

Problem Statement

BankNext’s Generative AI solution for GDPR/PII/PCI compliance substantially improves cost efficiencies, but it also requires invoking the external OpenAi APIs. Given its unyielding commitment to data protection, sending such data outside its network is absolutely prohibited.
At the same time, not employing the revolutionary breakthroughs of AI presages an imminent extinction as a technological leader.
BankNext needs to embrace AI but in a responsible way & also staunchly maintain its data privacy commitments. BankNext needs a practical solution & it needs it fast.

Business Scenario

1. GDPR/PII/PCI mandates that sensitive data fields should be masked.
2. If used in plain text, then additional strenuous procedures are mandated.
3. Every instance of non compliance imposes heavy financial penalties.
4. All request & responses w/ sensitive data fields need modification.
4. Every msvc w/ sensitive data needs this adaptation individually.
5. There is no centralized way to ease this massive undertaking.
6. Manpower, timelines & money required for this approach is colossal.
7. Penalties are piling. Business is Hyperventilating.
8. Houston, we have a problem!

Challenges w/ Current Architecture : Concrete Scenario

1. BankNext’s current architecture uses the traditional non AI solution.
2. Spring Cloud Gateway accomplishes this sensitive data filtering.
3. This solution is stable but involves a substantial construction effort.
4. Identifying the everchanging sensitive fields list is a manual exercise.
5. Sensitive fields list changes based on the demography.
6. Requires business teams to constantly keep the list up to date.
7. Adapting 300+ msvcs to identify such fields is the not practical.
8. GDPR can add more fields to the sensitive fields list in the future.

Capabilities w/ New Architecture : Target State

1. Automatically identify fields that are within GDPR/PII/PCI purview.
2. Future proof to accommodate newly designated sensitive fields.
3. Substantially reduce engineering effort & time.
4. Provide high degree of accuracy & allow effective review mechanisms.
5. Make contextually aware decisions based on demography.
6. Demonstrate adequate reasoning & inferencing capability.
7. Provide the highest standards in data privacy.
8. Restrict system interactions & data traffic within premise only.
9. Simple to manage.

Solution w/ Generative Ai : Final State

GenAi Local LLM w/ Vector DB Architecture

1. GitHub : Local LLM with GenAi
2. Local LLM server for inferencing engine - Ollama
3. Vector database for knowledge & context storage - Chroma
4. Pre-trained large language model - llama2
5. Java based tool for LLM interactions - LangChain4J
6. Embedding model - AllMiniLmL6V2EmbeddingModel
7. General purpose framework - SpringBoot/Java
8. Docker - my Docker runtime setup for your reference
9. Min machine config - RAM : 16GB, Storage : 50 GB, Cores : 4

System Design : Workflow

1. Input / User Prompt:
“El monto del pago con tarjeta de crédito para el Sr. Gary Thompson con cinco siete uno EIDA, 764132566 Número de Seguro Social y número de contacto 7536785621 es de USD 752.63"

2. Goal :
We want GenAi to autonomously identify any PII fields in this input text.

3. Full context sent to LLM:
“You are a helpful assistant. This instruction is to detect. Given a text, you will respond with fields that indicate person’s information. Your response should contain only these specific fields separted by commas. Be very concise in your response. The text is “El monto del pago con tarjeta de crédito para el Sr. Gary Thompson con cinco siete uno EIDA, 764132566 Número de Seguro Social y número de contacto 7536785621 es de USD 752.63”

4. Chroma :
Context is pre-loaded into the Chroma DB & semantically stored as embeddings. This helps GenAi focus on the specifics of the problem scenario.

5. Lanchain4J :
Input is passed to LangChain4J & converted into embeddings. This is required to perform semantic similarity comparisons.

6. Retrieval Augmented Generation :
LangChain4J extracts the context from Chroma.
Concatenates the context with the user prompt.
Loads the LLM pre-trained model llama2 in to memory.
Invokes the local LLM server to generate the response.

7. Ollama :
This local LLM server has inferencing capabilities.
With the combination of context + input LLM semantically understands the task to be performed.

8. GenAi Output :

Sure! Based on the provided text, here are the requested fields:
* Sr. Gary Thompson
* five siete uno EIDA (0764132566)
* 764132566 (Social Security Number)
* 7536785621 (Contact number)
Please let me know if you need any further assistance!

Application Setup

1. Startup your Docker runtime and the Virtualization env.
2. Chroma DB setup and launch

cd C:\Program Files\Docker\Docker\resources\bin
docker run -p 8000:8000 chromadb/chroma

-logs
Rebuilding hnsw to ensure architecture compatibility
Collecting chroma-hnswlib
  Downloading chroma_hnswlib-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 6.2 MB/s eta 0:00:00
...
...
Started server process [12]
Waiting for application startup.
Application startup complete.
Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

3. Launch SpringBoot with LangChain4J & Ollama

1. Get the codebase from GitHub
 cd <YOUR-PATH>
 git clone https://github.com/vijayredkar/GenAi-Local-LLM-to-accomplish-GDPR-compliance.git

2. build and launch
 cd \GenAi-Local-LLM-to-accomplish-GDPR-compliance\gen-ai-llm-gdpr
 mvn clean install
 java -jar target/gen-ai-llm-local-data-privacy.jar

3. After launch, this application
 a. connects to Chroma DB on startup
 b. loads llama2 in to memory
 c. listens to incoming user prompts
 d. converts text to embeddings
 e. interacts with Chroma and Ollama LLM server
 f. Understands the semantic context
 g. Provides the response

4. cURLs for testing
 curl --request GET
--url 'http://localhost:8888/gen-ai/v1/llm/retrieve?text=The%20amount%20of%20credit%20card%20payment%20for%20Mr.%20Gary%20Thompson%20with%205671425%20EIDA%2C%20SSN%20764132566%2C%20and%207536785621%20contact%20number%20is%20USD%20752.63'
 
 curl --request GET
--url 'http://localhost:8888/gen-ai/v1/llm/retrieve?text=El%20monto%20del%20pago%20con%20tarjeta%20de%20cr%C3%A9dito%20para%20el%20Sr.%20Gary%20Thompson%20con%205671425%20EIDA%2C%20764132566%20SSN%20y%20n%C3%BAmero%20de%20contacto%207536785621%20es%20de%20USD%20752.63'

Application Demo

Results- GenAi Output

Detects PII even in non English scenarios

Conclusion -

1. Positives:
   - BankNext solved the GDPR problem w/ Java based Generative Ai.
   - Mitigated data privacy challenge w/ Ollama local LLM server setup.
   - Finetuned response w/ Chroma Vector DB tuned embeddings.
   - Inferenced unstructured data seamlessly with acceptable accuracy.
   - Eliminated custom coding for multiple scenarios.
2. Negatives:
   - Requires massive computing GPU resources.
   - Latency is high on regular CPU based machines.
   - Variations occur in response content with almost every run.
   - Unexpected/incomplete LLM response, at times.
   - Additional manual review of the output is a must.
3. Accomplishment:
   GenAi’s versatility allowed BankNext to a solve a huge set of banking use cases that would have otherwise been very tedious.