Balancing Chain-of-Thought and Answer Supervision for Small Language Models on Medical Question Answering
| Source: arXiv AI
Tags: MedQA, Med-O1, Qwen3.5-4B, Llama-3.2, chain-of-thought fine-tuning, medical QA
For small medical QA models, segment-weighted chain-of-thought loss works best on multiple choice while uniform weighting wins on free text, so fine-tuning should match the task format.
Details
Small language models of 9B parameters or less can run medical QA locally, easing privacy and latency concerns, but fine-tuning them well is hard. This paper compares two chain-of-thought fine-tuning methods: Uniform-across-Sequence, which weights the whole CoT and answer equally, and Segment-weighted, which weights CoT and answer losses separately. Across six general and medical models, Segment-weighted generally did better on MedQA and EMS-MCQA multiple choice, with Qwen3.5-4B improving up to 44.8% over zero-shot on MedQA. Medical-domain models gained less or even lost ground there. On free-text Med-O1, Uniform weighting was best for nearly all models, and the 3B Llama-3.2 approached 7B medical models. Overly long CoT generation was linked to lower scores in some models.