计算化学公社

标题: CP2K运行一会就会任务中断 [打印本页]

作者
Author: 嘤嘤嘤 时间: yesterday 15:05
标题: CP2K运行一会就会任务中断
最近在192核服务器上跑两个CP2K任务，之前是96核100G运行没毛病，后来重启服务器后就开始运行几小时后任务中断，推测可能是内存不够。于是我降低核数，从96核降低到64核，内存100G，还是运行几小时后任务中断。请问这是什么原因？机器总内存是254G的。
#!/bin/bash
#SBATCH --job-name=cp2k
#SBATCH --output=test_%j.out
#SBATCH --error=test_%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --cpus-per-task=1
#SBATCH --mem=100G

这是我部分脚本参数

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 41 PID 280326 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 42 PID 280327 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 43 PID 280328 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 44 PID 280329 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 45 PID 280330 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 46 PID 280331 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 47 PID 280332 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 48 PID 280333 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 49 PID 280334 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 50 PID 280335 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 51 PID 280336 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 52 PID 280337 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 53 PID 280338 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 54 PID 280339 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 55 PID 280340 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 56 PID 280341 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 57 PID 280342 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 58 PID 280343 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 59 PID 280344 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 60 PID 280345 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 61 PID 280346 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 62 PID 280347 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 63 PID 280348 RUNNING AT cn031
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
CP2K calculation ended at: Wed Aug 12 02:24:15 PM CST 2026
CP2K exit status: 255
End time: Wed Aug 12 02:24:15 PM CST 2026
CP2K failed.
Please check:
test_431759.err

这是报错信息。
求各位老师指点，持续很长时间了，任务重启很多次了。而且我还发现，我96核任务每一步时间是21s，64核任务每步时间是31s，之前跑48核任务还是31s.总觉得哪里诡异。

作者
Author: KazusaT 时间: yesterday 15:12
可以用sacct看看是不是OOM了
运行时间不能单纯的看核数，也与体系有关系，并且当核数是n^2时一般会快一些

作者
Author: 嘤嘤嘤 时间: yesterday 15:23
更新一下，刚刚从64降低到48核，内存降低到80G后，每一步时间升高到40s了。。感觉核数不重要，内存最重要？但是之前确实48核100g内存的时候是30s的，我和工程师都发现过。。。

作者
Author: 嘤嘤嘤 时间: yesterday 15:34

KazusaT 发表于 2026-8-12 15:12
可以用sacct看看是不是OOM了
运行时间不能单纯的看核数，也与体系有关系，并且当核数是n^2时一般会快一些

这个 sacct 结果比前面更清楚了，但仍然要谨慎解释。

你现在看到的是：

431759          FAILED    1:0          0:0    100G
431759.batch    FAILED    1:0                               16.50M
431759.0    COMPLETED    0:0                         194533896K 194533896K

这里最关键的一点是：431759.0 这个 step 的 MaxRSS 和 AveRSS 都是 194533896K，而且 MaxRSSTask=0。这意味着 Slurm accounting 看到的是某个 step/task 的峰值 RSS，大约 185.5 GiB，但它并没有直接把主作业状态标成 OUT_OF_MEMORY，主作业只是 FAILED。

所以目前可以说：

内存超限非常可疑，但还不能仅凭这条 sacct 100% 定性为 OOM。

原因在于你这个 .0 step 是 MPI/Hydra 相关 step，MaxRSS 的统计口径在不同集群配置下可能不是你直觉理解的“整个 CP2K 作业总内存”。不过它确实是一个非常强的异常信号，因为你申请的是 100G，而 accounting 记录到了约 185.5 GiB。

换算一下：

194533896 KiB / 1024 / 1024 ≈ 185.5 GiB

这和你之前看到的 SIGKILL 9 是吻合的：如果 cgroup 或节点内存限制触发，进程会直接被系统 kill，而不是 CP2K 自己正常退出。

你好，这是我问AI给的结果，报错文件里没有明确的说内存超出

作者
Author: Stardust0831 时间: yesterday 17:44
既然怀疑OOM，那要不装个netdata看看？可以看到一段时间内的每个cgroup的内存使用情况。当然如果是独占节点的话也可以把free 的命令定时输出并重定向到文本文件。既然有slurm、推测你是集群环境、也可以让集群管理员看看，管理员对软硬件环境肯定是最熟的。

作者
Author: 嘤嘤嘤 时间: yesterday 19:17

Stardust0831 发表于 2026-8-12 17:44
既然怀疑OOM，那要不装个netdata看看？可以看到一段时间内的每个cgroup的内存使用情况。当然如果是独占节点 ...

管理员不回消息了。。。给他发消息想让他重新编译一下也不理我，我现在换成24*4运行了，明天看看任务会不会掉吧

作者
Author: 嘤嘤嘤 时间: yesterday 19:25

KazusaT 发表于 2026-8-12 15:12
可以用sacct看看是不是OOM了
运行时间不能单纯的看核数，也与体系有关系，并且当核数是n^2时一般会快一些

CP2K软件64核100G和24*4核120G的任务，相同体系，同一台服务器，运行时间一样，都是30-31s，这合理吗

欢迎光临计算化学公社 (http://bbs.keinsci.com/)