FENCE: A Financial and Multimodal Jailbreak Detection Dataset

A benchmark dataset of 1,000 multimodal queries (500 harmful / 500 benign) across 5 image types for evaluating VLM safety against jailbreak attacks in the financial domain.

[Paper]   [PDF]

---

Overview

FENCE is a benchmark dataset for evaluating the safety of VLMs against multimodal jailbreak attacks in the financial domain.

• 1,000 queries — 500 harmful and 500 benign
• 967 unique images spanning 5 types: baseimg, textimg, eng_textimg, figstep, and eng_figstep
• Multimodal inputs combining text and image
• Built-in placeholder system for anonymizing financial entities

Repository Structure

See Repository Structure for the full directory layout.

Quick Start

1. Clone & Navigate

git clone https://github.com/kakaobank/FENCE.git
cd FENCE

2. Prerequisites

Install uv (the only required tool):

macOS

brew install uv

Linux (Debian/Ubuntu)

curl -LsSf https://astral.sh/uv/install.sh | sh

3. Install Dependencies

uv sync

4. Configure Environment

cp .env.example .env

Set the following keys in .env:

Key	Description	Required
PIXABAY_API_KEY	API key for downloading Pixabay images	Yes

5. Build Dataset

Tasks are defined in pyproject.toml and run via poe (uv run poe <task>).

Full pipeline (with placeholder replacement):

uv run poe all

Quick pipeline (without placeholder replacement):

uv run poe dataset

To run each stage individually, see the full task list

Examples

Sample images from each of the 5 image types (benign queries shown).

baseimg
Korean
textimg	figstep
English
eng_textimg	eng_figstep

Documentation

Topic	Description
Dataset Reference	CSV columns, input_query vs query, and the image types
Pixabay Images	Which images to download and how to fetch them
Placeholder System	Anonymizing entity names and replacing placeholders

Citation

If you find this dataset useful, please cite our paper:

@inproceedings{kim2026fence,
    title     = {FENCE: A Financial and Multimodal Jailbreak Detection Dataset},
    author    = {Kim, Mirae and Jeong, Seonghun and Kwak, Youngjun},
    booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
    year      = {2026},
    doi       = {10.63317/4a35sc6sgwwv},
    url       = {https://lrec.elra.info/lrec2026-main-712}
}

License

Dataset annotations and non-Pixabay images are provided under the CC-BY-NC-4.0 license.

Pixabay-sourced images are subject to the Pixabay License.