计算化学公社

标题: cp2k2024.3提交并行作业出现报错该怎么解决？ [打印本页]

作者
Author: 生煎小馒头 时间: 2024-12-24 15:51
标题: cp2k2024.3提交并行作业出现报错该怎么解决？
最近在集群上运行了cp2k2024.3版本的psmp作业，cp2k是参照sob老师的安装过程：http://bbs.keinsci.com/thread-21608-1-1.html                                                                                                       参考了其他帖子提交作业：http://bbs.keinsci.com/forum.php ... ht=cp2k%B2%A2%D0%D0                                                                                                          还是出现报错：                                                                                                                                                                                                                                                          [icn256:40145] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and MPI will try to terminate your MPI job as well)
[icn256:40154] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and MPI will try to terminate your MPI job as well)
[icn256:40160] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and MPI will try to terminate your MPI job as well)
[icn256:40157] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Tue Dec 24 15:36:51 CST 2024
请问这种问题该怎么解决？                                                                                                                                                                                                                                                       这是我提交作业的脚本                                                                                                                                                                                                                                                             #!/bin/bash
#SBATCH --job-name=jht_bulk
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --cpus-per-task=1
#SBATCH --partition=hcpu48
#SBATCH --exclude=icn201,icn264
# load the environment
module purge
module load cp2k/2024.3
source /data0/software/cp2k/cp2k-2024.3/tools/toolchain/install/setup
# 定义输出文件
LOGFILE="memory_usage.log"

echo "Start monitoring memory usage..." > $LOGFILE
while true; do
echo "Timestamp: $(date)" >> $LOGFILE
free -h >> $LOGFILE
echo "========================" >> $LOGFILE
sleep 60  # 每隔60秒记录一次
done &

date
#srun cp2k.psmp input.inp > output.out
mpirun --mca pml ucx --mca btl '^openib'  -np 96 cp2k.psmp input.inp > output.out
#cp2k.psmp input.inp > output.out
date

作者
Author: Uus/pMeC6H4-/キ 时间: 2024-12-24 16:30
本帖最后由 Uus/pMeC6H4-/キ于 2024-12-24 16:37 编辑

集群很可能已经装有MPI模块，用module list可以查找、module load可以加载、which mpirun可以确定版本，在自己编译CP2K时加载模块、在提交脚本里写上加载模块的指令和srun cp2k.psmp应该是最理想的。现在的问题估计是因为自己在安装工具链时新装了MPI而不能很好兼容作业调度系统。

编辑：提议把这帖和上次的帖子合并，毕竟问题和解法都差不多一样

作者
Author: 生煎小馒头 时间: 2024-12-24 16:35

Uus/pMeC6H4-/キ发表于 2024-12-24 16:30
集群很可能已经装有MPI模块，用module list可以查找、module load可以加载、which mpirun可以确定版本，在 ...

那我是需要重新用集群的openmpi来重新编译cp2k吗

作者
Author: Uus/pMeC6H4-/キ 时间: 2024-12-24 16:44

生煎小馒头发表于 2024-12-24 16:35
那我是需要重新用集群的openmpi来重新编译cp2k吗

有的话那肯定优先考虑用啊，在install_cp2k_toolchain.sh那一步用--with-openmpi=system能指定。不是所有的选项都需要照搬社长的安装教程里的，得按集群实际有什么编译器什么MPI变通嘛

作者
Author: 生煎小馒头 时间: 2024-12-24 16:49

Uus/pMeC6H4-/キ发表于 2024-12-24 16:44
有的话那肯定优先考虑用啊，在install_cp2k_toolchain.sh那一步用--with-openmpi=system能指定。不是所有 ...

好的那我试试，谢谢老师

作者
Author: Santz 时间: 2024-12-24 20:26
先试试单节点的，或者 salloc到某个计算节点直接在计算节点下运行，看能否直接运行。slurm 脚本里不建议加内存监测相关的，如果想监控内存可以用 jobperf 或 jobstats。

作者
Author: abin 时间: 2024-12-24 20:37
在未确认网络状态，网络规格情况下，
请勿在mpirun 里面使用你不明了的参数……

欢迎光临计算化学公社 (http://bbs.keinsci.com/)