BankNext Case Study - AI & LLM for Credit Card Fraud Detection
GenAi with local LLM server & Java to detect credit card payment fraud based on spending patterns analysis.
Problem Statement
BankNext’s fraud detection flow utilizes a licensed proprietary tool to analyze the payment transactions. Every transaction analyzed by this 3rd party tool is charged. The massive volume of payments to be processed, means that the cumulative monetary costs become severely prohibitive. A balance needs to be found to keep costs down as well as accomplish the mandatory transaction scrutiny. BankNext needs a practical solution & it needs it fast.
Business Scenario
1. Every credit card txn needs a fraud check.
2.Every txn is sent to a proprietary 3rd party verification tool & is charged.
3. The txn volume is expected to increase manifolds & so will the cost.
4. In reality only a small set of the total txns turn out to be actual fraud.
5. But still all txns are sent for verification & hence cost is high.
6. Costs are amassing. Business is feeling distressed.
7. Houston, we have a problem!
Capabilities w/ New Architecture : Target State
1.Substantially lower the cost incurred to perform fraud check.
2. Detect real fraud cases to an acceptable level of accuracy.
3. Only this subset should be sent to the core proprietary tool.
4. Substantially reduce engineering effort, time & cost.
5. Provide high degree of accuracy & allow effective review mechanisms.
6. Extensibility to accommodate new payment types & detection scenarios.
7. Provide reasoning on why a txn is marked as potentially fraud.
8. Restrict system interactions & data traffic within premise only.
9. Simple to manage.
Solution w/ Generative Ai : Final State
1. GitHub : Local-LLM-Payment-Fraud-Detect
2. Local LLM server for inferencing engine -Ollama
3. Vector DB for storage & context -ElasticSearch/Chroma/Qdrant
4. Pre-trained large language model -llama2
5. Java based tool for LLM interactions - LangChain4J
6. Embedding model -AllMiniLmL6V2EmbeddingModel
7. General purpose framework -SpringBoot/Java
8. Docker - my Docker runtime setup
9. Min machine config - RAM:16GB, Storage:50GB, Cores:4
Training the AI: recent payment history PDFs
System Design: Workflow
1. Load the Vector Db with recent customer payment patterns:
curl --request POST \
--url http://localhost:8888/gen-ai/v1/llm/contextWithGenuineTxns
2. Input / Live txn:
“Adam Thorpe has a transaction on his card showing an amount of USD 7000 paid for a wrist watch”
3. Goal :
We want GenAi to autonomously determine if this txn is potentially fraud.
4. Full context sent to LLM:
“You are a helpful assistant. You will be provided the customer’s recent genuine payment transactions. Based on the provided patterns your task is to classify the given test transaction as fraud or not.
The recent genuine transactions are
Adam Thorpe recent transaction was for an amount USD 100.00 in Hong Kong, China on 14-Jan-2024. Item purchased is Gym membership.
Adam Thorpe recent transaction was for an amount USD 700.87 in Hong Kong, China on 28-Jan-2024. Item purchased is School fees.
Adam Thorpe recent transaction was for an amount USD 15.00 in Hong Kong, China on 4-Jan-2024. Item purchased is Zara leather jacket.
Adam Thorpe recent transaction was for an amount USD 402.25 in Hong Kong, China on 14-Jan-2024. Item purchased is Xbox.
Adam Thorpe recent transaction was for an amount USD 10.00 in Hong Kong, China on 1-Jan-2024. Item purchased is Milk.
Test transaction to classify “Adam Thorpe has a transaction on his card showing an amount of USD 7000 paid for a wrist watch”
Be concise in your response. Only state if the given test transaction is fraud or not alongwith a brief reason.”
5. VectorDB:
Context is pre-loaded into the VectorDB & semantically stored as embeddings. This helps GenAi focus on the specifics of the problem scenario.
6. Lanchain4J :
Live txn from the source system is passed to LangChain4J & converted into embeddings. This is required to perform semantic similarity comparisons.
7. Retrieval Augmented Generation:
LangChain4J extracts the context from the VectorDB.
Concatenates the context with the core txn data.
Loads the LLM pre-trained model llama2 in to memory.
Invokes the local LLM server to generate the response.
8. Ollama :
This local LLM server has inferencing capabilities.
With the combination of context + input LLM semantically understands the task to be performed.
9. GenAi Output :
Based on the recent genuine payment transactions provided,
I can confirm that the test transaction "Adam Thorpe has a transaction on his
card showing an amount of USD 7000 paid for a wrist watch" is most likely fraudulent.
The reason for this conclusion is that the amount of the test transaction (USD 7000) is significantly higher than any of the amounts listed in the recent genuine transactions provided (max: USD 402.25). This discrepancy suggests that the test transaction may be a fake or manipulated transaction, rather than an authentic one.Application Setup
1. Startup your Docker runtime and the Virtualization env.
2. ElasticSearchDB setup and launch
cd C:\Program Files\Docker\Docker\resources\bin
docker run -d -p 9200:9200 -p 9300:9300 -e discovery.type=single-node -e xpack.security.enabled=false docker.elastic.co/elasticsearch/elasticsearch:8.9.0
#logs to verify launch status
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2879bd7257ae docker.elastic.co/elasticsearch/elasticsearch:8.9.0 "/bin/tini -- /usr/l…" 5 hours ago Up 5 hours 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, :::9300->9300/tcp dazzling_haibt-logs3. Launch SpringBoot with LangChain4J & Ollama
1. Get the codebase from GitHub
cd <YOUR-PATH>
git clone https://github.com/vijayredkar/GenAi-Local-LLM-Payment-Fraud-Detect.git
2. build and launch
cd <your-path>\GenAi-Local-LLM-Payment-Fraud-Detect\gen-ai-llm-fraud-detect
mvn clean install
java -jar target/gen-ai-llm-local-fraud-detect.jar
3. After launch, this application
a. connects to ElasticSearchDB on startup
b. loads llama2 in to memory
c. listens to incoming user prompts
d. converts text to embeddings
e. interacts with VectorDB and Ollama LLM server
f. Understands the semantic context
g. Provides the response#cURLs to test
1. Live txn for fraud check test
curl --request GET \
--url 'http://localhost:8888/gen-ai/v1/llm/retrieve?minScore=0.6&maxLimit=5&text=Adam%20Thorpe%20has%20a%20transaction%20on%20his%20card%20showing%20an%20amount%20of%20USD%207000%20paid%20for%20a%20wrist%20watch'
2. Live txn for fraud check test
curl --request GET \
--url 'http://localhost:8888/gen-ai/v1/llm/retrieve?minScore=0.6&maxLimit=5&text=John%20Mayo%20has%20a%20transaction%20on%20his%20card%20showing%20an%20amount%20of%20USD%20450%20made%20in%20South%20Africa'Application Demo
Results - GenAi Output
Conclusion
- Positives:
- BankNext drastically shrunk the fraud check costs by applying GenAi.
- Effectively separated out txns that are genuine from potentially fraud.
- Eliminated the costs of verification for these genuine txns.
- Successfully implemented the Ollama local LLM server setup.
- Finetuned response with Vector DB tuned embeddings.
- Inferenced unstructured data seamlessly with acceptable accuracy.
- Eliminated custom coding for multiple scenarios. - Negatives:
- Requires massive computing GPU resources.
- Latency is high on regular CPU based machines.
- Variations occur in response content with almost every run.
- Unexpected/inaccurate LLM response, at times.
- Additional manual review of the output is a must. - Accomplishment:
GenAi provides a effective mechanism to augment BankNext’s fraud detection operations & massively lower the operational costs while maintaining an acceptable level of accuracy.
