100m-sft-python

Full-parameter fine-tune of Ananda100/pocketcoder100M on a custom Python coding-problem dataset, using the deepseek-ai/deepseek-coder-6.7b-base tokenizer.

• Fine-tune tokens processed: 356,253,696
• Best validation loss: 0.9313
• Held-out pass@1: 7.5%
• Block size: 512
• Vocab size: 32022

Loading

This is a custom architecture, not a native transformers model, so AutoModel.from_pretrained won't work out of the box:

import json, torch
from safetensors.torch import load_file

with open("config.json") as f:
    cfg = json.load(f)

config = GPTConfig(**cfg)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()