Preparing JSONL Training Data for Language Models

Prompt, Chosen, Rejected

In the open-source world this shape is settled. TRL reads three columns, prompt, chosen and rejected, in either a plain-string form or a conversational form where each value is itself a messages array. TRL recommends the explicit prompt over folding it into both completions.

{"prompt": [{"role": "user", "content": "Explain a rollback to a non-engineer."}], "chosen": [{"role": "assistant", "content": "A rollback is putting the previous version of the software back in place, the way you would restore a document to yesterday's draft."}], "rejected": [{"role": "assistant", "content": "We revert the deployment artifact to the prior immutable tag and re-point the ingress."}]}

Two variants are worth knowing. An unpaired record swaps the pair for a single completion plus a boolean label, which is what you get when reviewers rate answers one at a time. A stepwise record carries a completions array with a matching labels array, for grading reasoning chains step by step.

The Same Idea, a Different Envelope

OpenAI carries identical information under different keys: the prompt sits inside an input object holding the messages array, and the two candidates are preferred_output and non_preferred_output, each an array of assistant messages. As of 2026 the documented constraint is one-turn conversations, with the preferred and non-preferred messages as the last assistant message.

{"input": {"messages": [{"role": "user", "content": "Explain a rollback to a non-engineer."}]}, "preferred_output": [{"role": "assistant", "content": "A rollback is putting the previous version of the software back in place."}], "non_preferred_output": [{"role": "assistant", "content": "We revert the deployment artifact to the prior immutable tag."}]}

A Validator You Can Run Right Now

This catches the failures that actually block a job: a line that is not valid JSON, an empty or missing messages array, a role the API does not accept, a content field that is not a string, and an example that does not end on the assistant turn. It streams, so file size does not matter.

import json
import sys

VALID_ROLES = {"system", "user", "assistant", "tool"}

def validate(path):
    errors = []
    count = 0

    with open(path, encoding="utf-8") as f:
        for n, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                errors.append(f"line {n}: blank line")
                continue

            try:
                record = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"line {n}: not valid JSON ({e.msg} at col {e.colno})")
                continue

            if not isinstance(record, dict):
                errors.append(f"line {n}: top level is a {type(record).__name__}, expected an object")
                continue

            messages = record.get("messages")
            if not isinstance(messages, list) or not messages:
                errors.append(f"line {n}: messages must be a non-empty array")
                continue

            for i, msg in enumerate(messages):
                if not isinstance(msg, dict):
                    errors.append(f"line {n}: message {i} is not an object")
                    continue
                role = msg.get("role")
                if role not in VALID_ROLES:
                    errors.append(f"line {n}: message {i} has role {role!r}")
                if not isinstance(msg.get("content"), str):
                    errors.append(f"line {n}: message {i} content is not a string")

            if isinstance(messages[-1], dict) and messages[-1].get("role") != "assistant":
                errors.append(f"line {n}: last message is not the assistant turn")

            count += 1

    return count, errors

examples, problems = validate("train.jsonl")
for problem in problems[:25]:
    print(problem)
print(f"{examples} examples, {len(problems)} problems")
sys.exit(1 if problems else 0)

Uploading Once It Passes

The current flow is two calls: upload the validated file, then create the job against the returned file id. Keep the model identifier in one variable, because that is the line you will be changing every few months.

from openai import OpenAI

client = OpenAI()

training = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
validation = client.files.create(file=open("valid.jsonl", "rb"), purpose="fine-tune")

job = client.fine_tuning.jobs.create(
    training_file=training.id,
    validation_file=validation.id,
    model=BASE_MODEL,   # look this up in the provider's current tunable-model list
)
print(job.id)

Deduplicate, Split, Shuffle

A holdout of five to ten percent is a reasonable default, with a floor of a few dozen examples so the number means something. Seed the shuffle: an unseeded split is not reproducible, and you will want to rerun it.

import hashlib
import json
import random

def load(path):
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

def fingerprint(record):
    canonical = json.dumps(record, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

records = load("all.jsonl")

seen = set()
unique = []
for record in records:
    key = fingerprint(record)
    if key not in seen:
        seen.add(key)
        unique.append(record)

print(f"removed {len(records) - len(unique)} duplicates")

random.Random(20260725).shuffle(unique)     # seeded, so the split is reproducible
cut = max(20, len(unique) // 10)
valid, train = unique[:cut], unique[cut:]

def dump(path, rows):
    with open(path, "w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

dump("train.jsonl", train)
dump("valid.jsonl", valid)

Shuffling matters beyond the split. Data that arrives sorted, all the billing tickets and then all the shipping tickets, presents a curriculum you did not intend.

Budgeting Tokens

Training is billed and bounded by tokens, not by examples or megabytes. Total tokens in the training file, multiplied by the number of epochs, is the quantity that drives both cost and wall-clock time. Multiply that by your provider's current per-token training rate for a figure, and read that rate off their pricing page on the day you run.

Count with the tokenizer belonging to the model you are tuning. Character and word counts are not a substitute, and a tokenizer from another model family can be far off on code, JSON or non-English text.

from transformers import AutoTokenizer
import json

tok = AutoTokenizer.from_pretrained(BASE_CHECKPOINT)

lengths = []
with open("train.jsonl", encoding="utf-8") as f:
    for line in f:
        messages = json.loads(line)["messages"]
        text = tok.apply_chat_template(messages, tokenize=False)
        lengths.append(len(tok(text).input_ids))

lengths.sort()
total = sum(lengths)
print(f"examples          {len(lengths)}")
print(f"total tokens      {total}")
print(f"median example    {lengths[len(lengths) // 2]}")
print(f"95th percentile   {lengths[int(len(lengths) * 0.95)]}")
print(f"longest example   {lengths[-1]}")
print(f"tokens x 3 epochs {total * 3}")

Look at the tail, not just the average. Examples longer than the training context window get truncated, and truncation eats the end of the text, which is the assistant turn you wanted to learn. If the ninety-fifth percentile is close to the limit, shorten or drop those examples deliberately.

Quality Over Volume

Because loss lands only on assistant text, a few hundred carefully written responses routinely beat tens of thousands of scraped ones. Read a random sample of your own file before you launch anything. If the answers are inconsistent in tone, length or format, the tuned model will be inconsistent in exactly those ways, and more data does not fix it.

Version the file the way you version code: a dated filename, a note of what changed, and a seed that reproduces the split. When a tuned model regresses, the first question is which data produced it, and JSONL answers it with a line-level diff.