经过多次尝试,在kaggle 双T4训练Qwen2.5-0.5B的正确打开方式是:

经过多次尝试,在kaggle 双T4训练Qwen2.5-0.5B的正确打开方式是:

!torchrun --nproc_per_node=2 \ -m swift.cli.sft \ --model "./qwen2.5-0.5b-instruct" \ --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \ --max_length 4096 \ --num_train_epochs 3 \ --per_device_train_batch_size 4 \ --learning_rate 5e-5 \ --output_dir ./output_v2 \ --logging_steps 5 \ --save_steps 500 \ --eval_steps 500 \ --split_dataset_ratio 0.1 \ --bf16 true

详细过程

在kaggle上的最佳实践

先安装库

!pip install ms-swift[llm] -U -q !pip install torchao -U -q

下载模型

因为ms-swift自己下载模型太慢,用transformers下载

from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen2.5-0.5B-Instruct" save_dir = "./qwen2.5-0.5b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) tokenizer.save_pretrained(save_dir) model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True) model.save_pretrained(save_dir) print(f"下载完成，保存在 {save_dir}")

上传训练数据集

我就偷懒了,直接下载段言项目的代码,里面自带数据集

!git clone https://gitcode.com/skywalk163/duan/

怎么偷懒呢? 直接让程序帮我们找到数据集的位置

import os for root, dirs, files in os.walk("/kaggle"): for f in files: if f == "sft_dataset.jsonl": print(os.path.join(root, f))

这段代码会自动输出数据集的路径:

/kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl

开始训练

一般我们都是用swift sft 开训,但是在kaggle上双T4卡训练会报错,所以要用torchrun启动:

!torchrun --nproc_per_node=2 \ -m swift.cli.sft \ --model "./qwen2.5-0.5b-instruct" \ --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \ --max_length 4096 \ --num_train_epochs 3 \ --per_device_train_batch_size 4 \ --learning_rate 5e-5 \ --output_dir ./output_v2 \ --logging_steps 5 \ --save_steps 500 \ --eval_steps 500 \ --split_dataset_ratio 0.1 \ --bf16 true

现在max_length 设为4096, train_batch_size 设为4也能正常训练,不爆显存 .以前用段言自带的训练脚本,max_length 设为2048, train_batch_size 设为1 才能不爆显存!

训练完毕:

Train: 100%|██████████████████████████████████| 201/201 [14:58<00:00, 2.82s/it] {'eval_loss': '0.2401', 'eval_runtime': '8.635', 'eval_samples_per_second': '13.55', 'eval_steps_per_second': '6.833', 'eval_token_acc': '0.9391', 'epoch': '3', 'global_step/max_steps': '201/201', 'elapsed_time': '15m 7s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.514'} Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00, 7.01it/s] /usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201 {'train_runtime': '908.2', 'train_samples_per_second': '3.495', 'train_steps_per_second': '0.221', 'train_loss': '0.3376', 'epoch': '3', 'global_step/max_steps': '201/201', 'elapsed_time': '15m 8s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.518'} Train: 100%|██████████████████████████████████| 201/201 [15:08<00:00, 4.52s/it] [INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201 [INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v2-20260801-011216/checkpoint-201 [INFO:swift] images_dir: /kaggle/working/output_v2/v2-20260801-011216/images [INFO:swift] End time of running main: 2026-08-01 01:27:46.525213 [rank0]:[W801 01:27:47.297239747 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

最优模型确认：框架（这里使用的是swift）将checkpoint-201（即最后一个epoch保存的检查点）标记为最佳模型。

再用6个epoch试试!

有了当前提升准确率的感觉.

训练完成:

Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00, 7.01it/s] /usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402 {'train_runtime': '1786', 'train_samples_per_second': '3.554', 'train_steps_per_second': '0.225', 'train_loss': '0.2082', 'epoch': '6', 'global_step/max_steps': '402/402', 'elapsed_time': '29m 46s', 'remaining_time': '0s', 'memory(GiB)': '13.83', 'train_speed(s/it)': '4.443'} Train: 100%|██████████████████████████████████| 402/402 [29:46<00:00, 4.44s/it] [INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402 [INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v3-20260801-013448/checkpoint-402 [INFO:swift] images_dir: /kaggle/working/output_v2/v3-20260801-013448/images [INFO:swift] End time of running main: 2026-08-01 02:04:54.818971 [rank0]:[W801 02:04:55.541631968 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see Redirecting… (function operator())

最后测试下来,500步的效果最好

!torchrun --nproc_per_node=2 \ -m swift.cli.sft \ --model "./qwen2.5-0.5b-instruct" \ --dataset /kaggle/working/duan/tools/ai_copilot/sft_dataset.jsonl \ --max_length 4096 \ --num_train_epochs 12 \ --per_device_train_batch_size 4 \ --learning_rate 5e-5 \ --output_dir ./output_v2 \ --logging_steps 5 \ --save_steps 500 \ --eval_steps 500 \ --split_dataset_ratio 0.1 \ --bf16 true

Train: 100%|██████████████████████████████████| 804/804 [47:10<00:00, 2.91s/it] {'eval_loss': '0.2208', 'eval_runtime': '8.648', 'eval_samples_per_second': '13.53', 'eval_steps_per_second': '6.822', 'eval_token_acc': '0.952', 'epoch': '12', 'global_step/max_steps': '804/804', 'elapsed_time': '47m 19s', 'remaining_time': '0s', 'memory(GiB)': '14.06', 'train_speed(s/it)': '3.531'} Val: 100%|██████████████████████████████████████| 59/59 [00:08<00:00, 6.98it/s] /usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [INFO:swift] Saving model checkpoint to /kaggle/working/output_v2/v7-20260801-042924/checkpoint-804 {'train_runtime': '2840', 'train_samples_per_second': '4.47', 'train_steps_per_second': '0.283', 'train_loss': '0.1082', 'epoch': '12', 'global_step/max_steps': '804/804', 'elapsed_time': '47m 20s', 'remaining_time': '0s', 'memory(GiB)': '14.06', 'train_speed(s/it)': '3.532'} Train: 100%|██████████████████████████████████| 804/804 [47:20<00:00, 3.53s/it] [INFO:swift] last_model_checkpoint: /kaggle/working/output_v2/v7-20260801-042924/checkpoint-804 [INFO:swift] best_model_checkpoint: /kaggle/working/output_v2/v7-20260801-042924/checkpoint-500 [INFO:swift] images_dir: /kaggle/working/output_v2/v7-20260801-042924/images [INFO:swift] End time of running main: 2026-08-01 05:17:05.974298 [rank0]:[W801 05:17:07.697822352 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

如果爆显存,就把batch_size减小即可.

合并模型

!swift merge-lora --adapters /kaggle/working/output_v2/v7-20260801-042924/checkpoint-500

转为gguf

这步我在kaggle上需要先编译安装llama.cpp,需要较长时间,且经常有报错.所以转gguf我都是在本地让Trae帮我转的.

测试效果

全部完成！v7: 18/18 PASS (100%)，比 v3 的 17/18 有明显提升。

v3 vs v7 对比

模型信息

v7 核心改进

• 列表推导：从显式循环升级为[x 遍历 x 之 20至0：如果 x 取余 2 等于 0]语法
• 海象运算符：正确输出(设 n 为 len(data))形式
• with 语句：从直接失败变为可通过
• lambda 高阶函数：开始使用筛选/映射关键字
• 更少的冗余设 xxx 为 空声明

先到这里吧,暂时训练告一段落,该想想这个东西怎么用了.