IBMPlatformLSF家族安装和配置简介.V1.0

____________________________________之五兆芳芳创作IBMPlatformLSF家族装置和配置简介V1.0版马雪洁目录1集群结构11.1单纯LSF情况（命令行提交）11.2LSF+PAC情况（WEB提交）11.3LSF+PM情况（PM提交）32LSF装置和根本配置举例333444442.1.7启动/停止LSF进程（三种）5556667778899101010102.6.4设定Generallimits111111121212121313133LSF命令行集成应用示例133.1CFD++集成（spoolingfile）133.1.1CFD++装置和许可证13143.1.3添加CFD++jobstarter143.1.4添加CFDAPPprofile153.1.5CFD++命令行提交脚本实例153.2GAUSSIAN集成方法（spoolingfile）1515153.3Abaqus的脚本集成（bsub命令）153.4PlatformMPI作业16203.6IntelMPI作业202021264装置PAC285使用PAC进行应用程序集成29305.2CFD++集成后界面和后台脚本33356装置LicenseScheduler363636366.2.2映射许可证feature：3737377罕有问378使用manpage379售后技巧支持371集群结构较大的集群都会单独的登录节点，用户只能ssh到登录节点，不克不及直接ssh到集群的任何主节点和计较节点.同时配置用户在计较节点之间的ssh互信，为了并行作业的运行.登录节点也装置LSF，配置为LSF静态Client或MXJ值为0，也即不运行作业的客户端.集群的WEB节点与办公拜访局域网一个网段.如需使用浮动client，主节点网卡需要1.1单纯LSF情况（命令行提交）作业提交脚本……设计流程脚本bsubjobsDesktopDesktopDesktopDesktopLSFFloatClientLSFFloatClientLSFFloatClientLSFFloatClient访问网络SSHSSH作业提交脚本用户隔离计算资源，设计流程脚本脚本流程中的bsubjobsbsubjobs将作业散到登录节点集群计算节点。LSF主节点LSFStaticClient(可扩展到3个)XXXSSH管理网络JobsJobsJobsJobs高性能集群存储网络1.2LSF+PAC情况（WEB提交）用户通过portal提交作业：……访问网络登录节点（WEBPORTAL）Linux主节点管理网络Linux/Windows集群存储网络计算网络1.3LSF+PM情况（PM提交）……ProcessProcessProcessProcessManagerClientManagerClientManagerClientManagerClient访问网络登录节点（WEBPORTAL）LSF主节点LinuxProcessManagerServer管理网络高性能集群存储网络2LSF装置和根本配置举例2.1装置前的准备任务NISready；NFS/GPFSready；2.2LSF装置步调Useroottoinstall.GetNISandNFS/GPFSready.2.2.1取得LSF和PAC装置包lsf8.3_linux2.6glibc2.3x86_64.tar.Zlsf8.3_lsfinstall_linux_x86_64.tar.Zpac8.3_standard_linuxx64.tar.Z2.2.2解压缩lsfinstall装置脚本文件Putthepackageunder/root/lsf2.2.3首先添加集群办理员lsfadmin.LSF_TOP="/opt/lsf"(装置目录)LSF_ADMINS="lsfadmin"(先创建lsfadmin的用户名)LSF_CLUSTER_NAME="platform"(集群名称，任意指定)LSF_MASTER_LIST="s2s3"(LSF办理节点)LSF_ENTITLEMENT_FILE="/root/lsf/platform_hpc_std_entitlement.dat"(装置源许可证的地址)LSF_TARDIR="/root/lsf/"(装置源文件包的地址)2.2.4执行装置2.2.5配置开机自启动hostsetuprhostsetup2.2.6测试装置装置目录下的/conf目录Addsourceprofile.lsfto/etc/profileLSF_RSH="ssh"2.2.7启动/停止LSF进程（三种方法）[root@S2conf]#lsfstartup/lsfstop或lsadminlimstatup/limshutdownlsadminresstartup/resshutdownbadminhstartup/hshutdown或lsf_daemonsstart/stop[root@S2conf]#lsidIBMPlatformLSFExpress8.3forIBMPlatformHPC,May10CopyrightPlatformComputingInc.,anIBMCompany,1992.USGovernmentUsersRestrictedRightsUse,duplicationordisclosurerestrictedbyGSAADPScheduleContractwithIBMCorp.MyclusternameisplatformMymasternameiss2Youhavenewmailin/var/spool/mail/root[root@S2conf]#lsloadHOST_NAMEstatusr15sr1mr15mutpglsittmpswpmems2ok0.00.00.01%0.010151G20G61Gs4ok0.00.00.02%0.012183G20G62Gs6ok0.00.00.03%0.0123734M2G30Gs5ok0.00.00.05%0.0123468M2G30G2.2.8测试提交作业bsubsleep1000002.2.9使能root提交作业enableroottosubmitjob:LSF_ROOT_REX=local重启LSF进程.2.2.10修改配置文件后reconfig修改lsf.*配置文件后lsadminreconfig修改lsb.*配置文件后badminreconfig部分参数需要重启LSF主调度或其他进程：badminmbdrestart;lsadminlimrestart;lsadminresrestart;badminhrestart2.2.11日志和debugFindthelogsunderlogdirectory.LSFwillrunmainly3processesoneachnode,onmasternodewillhave2more.Master:lim,res,sbatchd,mbatchd,mbschedCompute:lim,res,sbatchdTurnondebugincommandline:Runlim2directlyonnodetocheckwhylimnotstartup.2.3配置文件说明目录/etc/init.d:目录/apps/platform/8.3/lsf/conf:lsf.conflsf配置文件lsf.cluster.cluster83集群配置文件lsf.shared同享资源定义文件./lsbatch/cluster83/configdir/lsb.*调度系统配置文件lsb.userslsf用户与用户组配置文件lsb.queueslsf队列配置文件lsb.moduleslsf模块配置文件2.4经常使用命令bsub:提交作业；bjobs:查抄作业信息；bhist:查抄作业历史；lshosts:查抄节点静态资源；bhosts，lsload:查抄节点状态和资源信息；bqueues：查抄队列配置；blimits：查抄限制limit信息；lsid:集群版本和主节点；bmod:修改bsuboption；等等.2.5基于资源的调度战略bsub–R“((type==LINUX2.4&&r1m<2.0)||(type==AIX&&r1m<1.0))”或在队列lsb.queues或lsb.application文件定义：RES_REQ=select[((type==LINUX2.4&&r1m<2.0)||(type==AIX&&r1m<1.0))]bsub–R"select[type==any&&swap>=300&&mem>500]order[swap:mem]rusage[swap=300,mem=500]"job1bsub–Rrusage[mem=500:app_lic_v2=1||mem=400:app_lic_v1.5=1]"job1bsub–R"select[type==any&&swp>=300&&mem>500]order[mem]"job12.6配置公道竞争调度战略2.6.1添加轮循调度队列Modifylsb.queues,addfollowingBeginQueueQUEUE_NAME=roundRobinPRIORITY=40FAIRSHARE=USER_SHARES[[default,1]]#USERS=userGroupADefineyourownusergroupEndQueueRunbadminreconfigtoenablethechange.Runbqueues–ltocheckthequeue’sconfigure2.6.2添加条理公道竞争战略Addfollowingqueuetoaddhierarchicalsharepolicy:BeginQueueQUEUE_NAME=hierarchicalSharePRIORITY=40USERS=userGroupBuserGroupCFAIRSHARE=USER_SHARES[[userGroupB,7][userGroupC,3]]EndQueue2.6.3多队列公道竞争战略在lsb.queues中添加下列队列，注意节点组和用户组定义.BeginQueueQUEUE_NAME=verilogDESCRIPTION=masterqueuedefinitioncrossqueuePRIORITY=50FAIRSHARE=USER_SHARES[[user1,100][default,1]]FAIRSHARE_QUEUES=normalshortHOSTS=hostGroupC#resourcecontention#RES_REQ=rusage[verilog=1]EndQueueBeginQueueQUEUE_NAME=shortDESCRIPTION=shortjobsPRIORITY=70#highestHOSTS=hostGroupCRUNLIMIT=510EndQueueBeginQueueQUEUE_NAME=normalDESCRIPTION=defaultqueuePRIORITY=40#lowestHOSTS=hostGroupCEndQueue2.6.4使能配置badminreconfig提交作业，并查抄队列的用户动态优先级变更：bqueues–rlnormal2.7配置抢占调度战略配置最根本的slots抢占：BeginQueueQUEUE_NAME=shortPRIORITY=70HOSTS=hostGroupC#potentialconflictPREEMPTION=PREEMPTIVE[normal]EndQueueBeginQueueQUEUE_NAME=normalPRIORITY=40HOSTS=hostGroupC#potentialconflictPREEMPTION=PREEMPTABLE[short]EndQueue向两个队列提交作业，查抄被preempt的作业的pending原因.2.8配置全局限制战略2.8.1限制用户运行的作业数目在lsb.users文件中添加：BeginUserUSER_NAMEMAX_JOBSJL/Puser14user221user32groupA8groupB@11Default2EndUser2.8.2限制节点运行作业数目在lsb.hosts文件中：BeginHostHOST_NAMEMXJJL/Uhost142host221host3!EndHost2.8.3限制队列作业的运行限制在lsb.queues中添加：BeginQueueQUEUE_NAME=myQueueHJOB_LIMIT=2PJOB_LIMIT=1UJOB_LIMIT=4HOSTS=hostGroupAUSERS=userGroupAEndQueue2.8.4设定Generallimits在lsb.resources文件定义全局generallimits示例：BeginLimitUSERSQUEUESHOSTSSLOTSMEMSWPuser1hostB20%user2normalhostA20EndLimitBeginLimitNAME=limit1USERS=user1PER_HOST=hostAhostCTMP=30%SWP=50%MEM=10%EndLimitBeginLimitPER_USERQUEUESHOSTSSLOTSMEMSWPTMPJOBSgroupAhgroup12user2normal200short200EndLimit2.8.5使能配置badminreconfig2.9配置提交控制脚本esub全局esub脚本在作业被提交是调用，可以被自动的或显式的调用从而控制用户作业提交的行动.编辑esub.project文件在$LSF_SERVERDIR下面（chmod为可执行）：#!/bin/shif["_$LSB_SUB_PARM_FILE"!="_"];then.$LSB_SUB_PARM_FILEif["_$LSB_SUB_PROJECT_NAME"=="_"];thenecho"Youmustspecifyaproject!">&2exit$LSB_SUB_ABORT_VALUEfifiexit0在lsf.conf中定义LSB_ESUB_METHOD=”project”2.10配置资源办理elim示例2.10.1请示home目录空闲大小编辑elim文件elim.home，放置在$LSF_SERVERDIR下面.chmod为可执行.#!/bin/shwhiletrue;dohome=`dfk/home|tail1|awk'{printf"%4.1f",$4/(1024*1024)}'`echo1home$homesleep30done2.10.2报告请示root进程数目编辑elim.root，放置在$LSF_SERVERDIR下面.chmod为可执行.#!/bin/shwhiletrue;doroot=`psef|grepvgrep|grepc^root`echo1rootprocs$rootsleep30done2.10.3报告请示应用程序许可证数目#!/bin/shlic_X=0;num=0whiletrue;do#onlywantthemastertogatherlic_Xif["$LSF_MASTER"="Y"];thenlic_X=`lmstat–a–clic_X.dat|grep...`>&2fi#onlywanttraining8,training1togathersimptonlicensesif["`hostname`"="training8"\–o"`hostname`"="training1"];thennum=`lmstat–a–csimpton_lic.dat|grep...`>&2fi#allhostsincludingmaster,willgatherthefollowingroot=`ps–efw|grep–vgrep|grep–croot`>>1&2tmp=`df–k/var/tmp|grepvar|awk'{print$4/1024}'`>&2if["$LSF_MASTER"="Y"];thenecho4lic_X$lic_Xsimpton$numrtprc$roottmp$tmpelseecho3simpton$numrtprc$roottmp$tmpfisleep60done2.10.4测试elim脚本直接运行./elim.root查抄elim输出是否正确.2.10.5添加资源定义和资源地图在lsf.shared文件中添加rootprocs定义，并在lsf.clusterresourcesMap中添加资源和节点的映射关系.使能配置：lsadminreconfig;badminreconfig2.10.6查抄资源数目lsload–l3LSF命令行集成应用示例本节例举几个应用的不合集成方法.使用spooling文件或bsub命令行都可以自由转换.3.1CFD++集成（spoolingfile）3.1.1CFD++装置和许可证装置路径：ln36204许可证办事器：ln36204启动许可证办事器：[hpcadmin@mn3650jessi]$sshln36204确认许可证办事器是否正常运行：3.1.2集成许可证办理elim添加elim办法：（elim全集群只需运行一个，因此只在头节点放置elim脚本便可）在头节点：cd$LSF_SERVERDIR添加如下文件：elim.lic:[root@mn3650jessi]#cd$LSF_SERVERDIR[root@mn3650etc]#pwd修改如下的配置文件：#!/bin/shtotallicences=`/gpfs/software/cfdpp/mbin/lmutillmstatac添加如下一行：/gpfs/software/cfdpp/mbin/Metacomp.lic|grep"UsersofCFD++_SOLV_Ser"|/bin/cutd''f7`cfd_licwhiletrueNumeric30Y(CFD++License)do在usedlicences=`/gpfs/software/cfdpp/mbin/lmutilresourcemap一段添加如下一行：lmstatacBegin/gpfs/software/cfdpp/mbin/Metacomp.licResourceMap|/bin/grep"UsersofCFD++_SOLV_Ser"|/bin/cutd''f13`RESOURCENAMELOCATIONcfd_lic=$((cfd_lic${totallicences}${usedlicences}[all]))hostid[default]echo"1cfd_lic${cfd_lic}"…/bin/sleep30[root@mn3650etc]#lsadminreconfig;badminreconfigdone3.1.3添加CFD++jobstarter如果使用[hpcadmin@mn3650spoolingfile可不必添加jessi]$cat.（/opt/lsf/jobstarter/cfd_starterPortal集成方法使用）#!/bin/sh添加MPI_RUN=/gpfs/software/cfdpp/hpmpi/bin/mpirunjobstarter可执行文件：case"$PRESSION"inSINGLE_PRESSION);;DOUBLE_PRESSION);;esacCMD="$*hostfile$LSB_DJOB_HOSTFILE$CFD_CMD"3.1.4添加CFDAPPprofile添加如下配置：BeginApplicationNAME=cfdJOB_STARTER=/opt/lsf/jobstarter/cfd_starterRES_REQ="rusage[cfd_lic=1]"EndApplicationbadmninreconfig使得此文件生效，使用bapp–lcfd查抄是否成功：[root@mn3650bin]#bapplcfdAPPLICATIONNAME:cfdNodescriptionprovided.STATISTICS:NJOBSPENDRUNSSUSPUSUSPRSV12120000PARAMETERS:JOB_STARTER:/opt/lsf/jobstarter/cfd_starterRES_REQ:"rusage[cfd_lic=1]"3.1.5CFD++命令行提交脚本实例然后bsubissubmittedtodefaultqueue.[root@server3conf]#bjobsJOBIDUSERSTATQUEUEFROM_HOSTEXEC_HOSTJOB_NAMESUBMIT_TIME210rootPENDnormalserver3*elloworldMay910:55[root@server3conf]#cat210.outSender:LSFSystemSubject:Job210:inclusterDoneJobwassubmittedfromhostbyuserincluster.Jobwasexecutedonhost(s)<4*computer007>,inqueue,asuserincluster.wasusedasthehomedirectory.wasusedastheworkingdirectory.StartedatThuMay918:49:06ResultsreportedatThuMay918:49:07Yourjoblookedlike:#LSBATCH:Userinput/opt/pmpi/opt/ibm/platform_mpi/bin/mpirunlsb_mcpu_hosts/opt/pmpi/opt/ibm/platform_mpi/help/helloworldSuccessfullycompleted.Resourceusagesummary:CPUtime:0.23sec.MaxMemory:2MBAverageMemory:2.00MBTotalRequestedMemory:DeltaMemory:(Delta:thedifferencebetweentotalrequestedmemoryandactualmaxusage.)MaxSwap:36MBMaxProcesses:1MaxThreads:1Theoutput(ifany)follows:Helloworld!I'm2of4oncomputer007Helloworld!I'm0of4oncomputer007Helloworld!I'm1of4oncomputer007Helloworld!I'm3of4oncomputer007PS:Readfile<.210.err>forstderroutputofthisjob.或更多参数$/opt/platform_mpi/bin/mpirunnp120ibvhostlist"cn2cn2cn2cn2cn2cn2cn2cn2cn2cn2"/data/hello_world如果希望MPI作业欠亨过LSF提走运行，修改MPI_USELF情况变量为n3.6Openmpi作业下载openmpi软件包./configureLIBS=ldlwithlsf=yesprefix=/usr/local/ompi/Openmpi1.3.2之上版本已经于LSFblaunch紧密集成.提交openmpi作业：3.7IntelMPI作业3.7.1Express版本不记账方法如果需要对作业记账，需要使用blaunch的集成方法.exportPATH=/gpfs/software/intel/composerxe/bin/:/gpfs/software/intel/mpi_41_0_024/include:/gpfs/software/intel/mpi_41_0_024/bin64:/gpfs/software/intel/composerxe/mkl:$PATHsource/gpfs/software/intel/composerxe/bin/compilervars.shintel64source/gpfs/software/intel/composerxe/mkl/bin/mklvars.shintel64MPI测试程序#include"mpi.h"#include#includeintmain(intargc,char**argv){intmyid,numprocs;intnamelen;charprocessor_name[MPI_MAX_PROCESSOR_NAME];MPI_Init(&argc,&argv);MPI_Comm_rank(MPI_COMM_WORLD,&myid);MPI_Comm_size(MPI_COMM_WORLD,&numprocs);MPI_Get_processor_name(processor_name,&namelen);fprintf(stderr,"HelloWorld!Process%dof%don%s\n",myid,numprocs,processor_name);MPI_Finalize();}命令执行，TCP协议命令执行，IB网络命令执行，Debug模式LSF提交脚本bsub_intelmpi_ib.sh#!/bin/sh#BSUBcwd.#BSUBR"span[ptile=4]"提交作业：bsubissubmittedtoqueue.<><>Helloworld:rank0of4runningonsaspm01Helloworld:rank1of4runningonsaspm01Helloworld:rank2of4runningoniquadcore01Helloworld:rank3of4runningoniquadcore01TIDHOST_NAMECOMMAND_LINESTATUSTERMINATION_TIME=========================================================================00000iquadcore./testDone03/16/20:00:4900001iquadcore./testDone03/16/20:00:4900002saspm01./testDone03/16/20:00:3900003saspm01./testDone03/16/20:00:39[iquadcore01]201%Youcanseethereisno"np4"after"bsubn4mpirun.lsf"3.7.3.4Debug办法提交命令后添加passDpass3–Tsdebug：bsubIaintelmpin4mpirun.lsf./testpassDpass3TSdebug4装置PAC1)查抄装置文件，如pac8.3_standard_linuxx64.tar.Z，许可证在装置包中自带，位于NFS同享目录/apps/platform/8.3/pac下.2)解压缩pac8.3_standard_linuxx64.tar.Z，修改pac3)4)exportPAC_TOP="/apps/platform/8.3/pac"exportMYSQL_JDBC_DRIVER_JAR="/usr/share/java/mysqlconnectorjava5.1.12.jar"5)装置mysql，并确认mysql办事启动正常.(yuminstallmysql*y)装置client和server端，servicemysqldstatus/start/stop(不必执行)修改/opt/lsf/conf/lsbatch/cluster1/configdir/lsb.params参加ENABLE_EVENT_STREAM=ybadminreconfig6)运行pacinstall.sh进行装置（运行之前确认source了LSF的情况变量）7)Source换情况变量：（将上面命令添加到/etc/profile文件结尾，登陆自动source情况）8)使用下面命令启动portal：#pmcadminstart#perfadminstartall9)使用下面命令查抄否正常启动：#pmcadminlist#perfadminlist10)使用下面地址拜访portal：http://hostipaddress:808011)使用办理员或用户身份登录（NIS用户）12)配置VNC办法，请参考PAC办理员文档.5使用PAC进行应用程序集成PAC集成的概念：配置和设计XML提交页面，在对应的脚本文件中处理XML文件中传递的情况变量.最终生成提交作业的逻辑（/opt/pac/gui/conf/application/published/app.cmd文件的最后）：JOB_RESULT=`/bin/shc"bsubq$SUB)QUEUES$JOB_NA