Instructions to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora")

Transformers

How to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora

SGLang

How to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora with Docker Model Runner:
```
docker model run hf.co/lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora
```

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Frame2KG LoRA Adapter

Overview

This repository provides a LoRA adapter for Frame2KG, a frame-to-knowledge-graph generation task for embodied robotics and interpretable visual understanding.

The adapter fine-tunes Qwen2.5-VL to convert an image frame into a structured knowledge graph containing:

objects (nodes)
attributes
spatial relationships (edges)
bounding box locations

The goal is to enable interpretable, on-device visual reasoning for robotics and privacy-preserving AI systems.

Training summary

This adapter was trained using the Frame2KG-Trainer with:

LoRA rank: r = 8
LoRA alpha: 16
LoRA dropout: 0.05
Fine-tuned modules: Q, K, V, O + Gate / Up / Down projections
Training data: Frame2KG-YC2 synthetic dataset

Frame2KG training config can be found here.

This repository contains adapter weights only, not the base model.

Intended use

The adapter is designed for:

frame-to-knowledge-graph generation
interpretable visual reasoning
embodied robotics research
edge / on-device deployment

It is not intended for general-purpose captioning or chat use.

How to load the adapter

from transformers import AutoModelForVision2Seq, AutoProcessor
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_id = "lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora"

model = AutoModelForVision2Seq.from_pretrained(
    base_model_id,
    device_map="auto"
)

model = PeftModel.from_pretrained(model, adapter_id)
processor = AutoProcessor.from_pretrained(base_model_id)

Model selection

During training, multiple checkpoints were evaluated on a held-out validation set. This repository contains the selected checkpoint chosen based on overall node and edge F1 performance.

Selected checkpoint: checkpoint-1000, chosen based on validation_dev node and edge F1.

Additional evaluated checkpoints are available in:
lewiswatson/frame2kg-adapter-archive

Limitations

trained on synthetic data created from YouCook2
performance may degrade on unusual viewpoints or unseen domains
outputs should be validated before safety-critical use

Citation

If you use this adapter in your work, please cite the paper:

@inproceedings{watson2026frame2kg,
  title = {Frame2KG: A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation},
  author = {Watson, Lewis N. and Strathearn, Carl and Mitchell, Kenny and Yu, Yanchao},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {10912--10926},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/4ys6kofrzoc5},
  url = {https://doi.org/10.63317/4ys6kofrzoc5}

Disclaimer

This model is provided as is, without any warranties or guarantees of any kind, either express or implied. The authors make no representations regarding the accuracy, reliability, safety, suitability, or performance of the model or any outputs it produces.

The model may generate incorrect, incomplete, or misleading results and should not be relied upon for critical, safety-sensitive, legal, medical, financial, or other high-stakes decisions. Use of this model is entirely at your own risk.

The authors accept no liability for any damages, losses, or consequences arising from the use, misuse, or inability to use the model, including but not limited to direct, indirect, incidental, or consequential damages.

By using this model, you acknowledge and agree to these terms.

Licence

This repository contains LoRA adapter weights only.
Use of the base model is subject to the Qwen licence.

Downloads last month: -

Model tree for lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora

Base model

Qwen/Qwen2.5-VL-3B-Instruct

Adapter

(302)

this model

Dataset used to train lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora

Collection including lewiswatson/frame2kg-qwen2_5vl-3b-qkvo-gateupdown-lora

Frame2KG

Collection

A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation • 6 items • Updated Jun 9 • 1