服务器训练过程程序崩溃，显卡资源释放方式
使用服务器训练过程出现程序崩溃，但是显卡资源未能释放的问题解决方式，主要是多卡使用过程，不能影响其他人正在使用的显卡资源。
一、查看显卡使用情况
查看显卡正在使用的进程
watch nvidia-smi|NVIDIA-SMI580.126.09Driver Version:580.126.09CUDA Version:13.0|+-----------------------------------------+------------------------+----------------------+|GPU Name Persistence-M|Bus-Id Disp.A|Volatile Uncorr.ECC||Fan Temp Perf Pwr:Usage/Cap|Memory-Usage|GPU-Util Compute M.||||MIG M.||=========================================+========================+======================||0NVIDIA A100-SXM4-80GB Off|00000000:00:09.0Off|0||N/A39C P089W/400W|24375MiB/81920MiB|0%Default||||Disabled|+-----------------------------------------+------------------------+----------------------+|1NVIDIA A100-SXM4-80GB Off|00000000:00:0A.0Off|0||N/A35C P095W/400W|45173MiB/81920MiB|0%Default||||Disabled|+-----------------------------------------+------------------------+----------------------+|2NVIDIA A100-SXM4-80GB Off|00000000:00:0B.0Off|0||N/A39C P093W/400W|77961MiB/81920MiB|56%Default||||Disabled|+-----------------------------------------+------------------------+----------------------+|3NVIDIA A100-SXM4-80GB Off|00000000:00:0C.0Off|0||N/A37C P099W/400W|78351MiB/81920MiB|100%Default||||Disabled|+-----------------------------------------+------------------------+----------------------++-----------------------------------------------------------------------------------------+|Processes:||GPU GI CI PID Type Process name GPU Memory||ID ID Usage||=========================================================================================||2N/A N/A3342634C/bin/python377918MiB||3N/A N/A3342635C/bin/python378308MiB|程序用的卡0和1训练，但是程序已经崩溃，卡0和1资源没有释放。
二、查看显卡正在使用的进程
查看显卡正在使用的进程
sudo fuser-v/dev/nvidia*USER PID ACCESS COMMAND/dev/nvidia0:root3341936F...m pt_main_thread root3342634F....pt_main_thread root3342635F....pt_main_thread root3348049F...m python3 root3348112F...m python3/dev/nvidia1:root3341936F...m pt_main_thread root3342634F....pt_main_thread root3342635F....pt_main_thread root3348049F...m python3 root3348112F...m python3/dev/nvidia2:root3341936F...m pt_main_thread root3342634F...m pt_main_thread root3342635F...m pt_main_thread root3348049F....python3 root3348112F....python3/dev/nvidia3:root3341936F...m pt_main_thread root3342634F...m pt_main_thread root3342635F...m pt_main_thread root3348049F....python3 root3348112F....python3三、查看进程对应的节点信息
ps-p3348049,3348112-o pid,ppid,stat,cmdPID PPID STAT CMD33480492589037Sl/opt/bin/python3-u tools/train.py--local-rank=0projects/configs/stage.py--launcher pytorch--deterministic--work-dir./work_dirs/stage33481122589037Sl/opt/bin/python3-u tools/train.py--local-rank=0projects/configs/stage.py--launcher pytorch--deterministic--work-dir./work_dirs/stage发现这两个进行号对应的是之前所用的程序导致的崩溃。
四、kill掉对应的进程号
sudo kill-933480493348112再运行第一步，显示显卡资源正常释放
五、停止之前的容器，重新进入
docker stop e847dc3213cf docker start e847dc3213cf docker exec-it e847dc3213cf bash六、多卡通信延迟报错
export NCCL_TIMEOUT=36000000