“Artificial intelligence is one of the most powerful technologies of our time, with broad applications. President Biden has been clear that in order to seize the opportunities artificial intelligence presents, we must first manage its risks.”
The White House, 2023
Large language models (LLMs) have emerged as a new application for artificial intelligence (AI), propelled by breakthrough research in an age of unprecedented information. Public attention has focused on ChatGPT for its ability to answer questions in a human-like way, showcasing helpful uses for LLMs such as text summarization, narrative generation, and language translation. Together, AI and LLM technologies have the potential to advance the way financial institutions do business — from automating routine tasks to enhancing fraud prevention — but as with any innovation, bad actors will also take advantage. Financial institutions must recognize the threat posed by the abuse of AI and LLMs, and how responsible organizations are leveraging their potential to fight back.
Cracking LLMs: How Criminals Exploit Helpful Technology
Criminals are making a concerted effort to abuse LLMs for fraud by bypassing content moderation — the safeguards that developers of responsible LLMs, such as ChatGPT, have implemented to prevent their model from engaging in discourse surrounding violence, sex, hate or self-harm. This includes using prompt engineering and jailbreaking to manipulate an LLM into generating text in a specific tone — ideal for spear phishing — or even developing LLMs that are designed for nefarious purposes.
Dark LLMs represent the pinnacle for misuse of LLM technology, where criminals develop, train, and share their own model that is purpose built for nefarious activity. While dark LLMs are largely a future concern, examples do exist today such as FraudGPT, and WormGPT which may be used to facilitate Business Email Compromise (BEC).
Abusing LLMs for Scams
When compromised, an LLM allows scammers to research unfamiliar tactics on command, operate in languages they do not comprehend, and generate messaging that is harder to detect, with appropriate context and few grammatical errors. These characteristics make LLMs ideal for scams that rely on text-based dialogue or phishing, such as authorized push payment (APP) fraud and related typologies — including BEC and romance scams. A custom dark LLM named WormGPT is circulating on criminal forums and was recently exposed and accessed. Researchers testing the tool prompted it to generate a convincing email for use in a BEC attack, written in the voice of a CEO and demanding urgent payment of an invoice.
LLMs may also be abused to code chat bots and generate genuine-feeling dating profiles for use in romance scams. As reported by Tinder, online daters prefer authenticity and immediacy in their romantic matches — making this exploit potentially lucrative for fraudsters.
With criminals able to produce persuasive messaging on demand, financial institutions should partner with a provider of financial crime management solutions that possess deep domain expertise in AI and criminal typologies, with a proven record of using AI to prevent fraud quickly and decisively.
Fighting Back: Using AI for Good
For over 20 years, Verafin has developed solutions that responsibly harness the power of AI to effectively fight financial crime. Through big data intelligence, we analyze over a billion transactions every week in our Cloud, enabling consortium analytics, cross-institutional analysis, machine learning and more — and use these innovations to help our financial institution partners shut down fraud at the earliest opportunity. This includes fraudulent wires, ACH transfers and other payment types often exploited in APP fraud, such as BEC and romance scams.
As AI and LLMs advance, so does the weight of their power. With fraudsters exploiting these tools for their gain, it is crucial to partner with a financial crime management solutions provider that has deep expertise in using AI for good. To learn more about how Verafin uses AI and big data intelligence to combat payment fraud, download our Product Brochure.