红帽Ceph存储性能和容量优化最佳实践

redhat.comfacebook.com/redhatinc@redhatnewslinkedin.com/company/red-hatQCT(QuantaCloudTechnology)offersafamilyofserversforbuildingdifferenttypesofscale-outstorageclustersbasedonRedHatCephStorage—eachoptimizedtosuitdifferentworkloadandbudgetaryneeds.Throughput-optimizedconfigu-rationsofferimpressiveperfor-mancewithbothstandardandhigh-densityservers.Costorcapacity-optimizedconfigurationsprovideindustry-leadingpriceanddensitywithinnovativeQCTserverplatformsthatareeasytodeployrapidlyatscale.ExtensiveRedHatandQCTtestinghelpstaketheriskoutofdeployingscale-outstoragesolutionsbasedonCeph.ExEcutivESummaryCephusersfrequentlyrequestsimple,optimizedclusterconfigurationsfordifferentworkloadtypes.Commonrequestsareforthroughput-optimizedandcapacity-optimizedworkloads,butIOPS-intensiveworkloadsonCepharealsoemerging.Toaddresstheneedforperformance,capacity,andsizingguidance,RedHatandQCT(QuantaCloudTechnology)haveperformedextensivetestingtocharacterizeoptimizedconfigurationsfordeployingRedHatCephStorageonarangeofQCTservers.tablEofcontEntS1Documentpurpose.................................................................................................32IntroDuctIon............................................................................................................33WorkloaD-optImIzeDscale-outstorageclusters................................43.1Characterizingstorageworkloads.............................................................................................43.2Sizingsummaryforworkload-optimizedclusters..................................................................54cephDIstrIbuteDstoragearchItectureovervIeW................................64.1IntroductiontoCeph......................................................................................................................64.2Cephaccessmethods.....................................................................................................................64.3Cephstoragepools.........................................................................................................................84.4Cephdataprotectionmethods....................................................................................................95referencearchItectureelements...............................................................105.1RedHatCephStorage...................................................................................................................105.2QCTserversforCeph.....................................................................................................................116archItecturalDesIgnconsIDeratIons........................................................126.1Qualifyingtheneedforscale-outstorage...............................................................................126.2Designingforthetargetworkload..............................................................................................126.3Choosingastorageaccessmethod............................................................................................136.4Identifyingstoragecapacity........................................................................................................136.5Selectingadataprotectionmethod..........................................................................................136.6Determiningfaultdomainrisktolerance..................................................................................13tEchnologyDEtailPErformancEanDSizingguiDE:rEDhatcEPhStoragEonQctSErvErS2redhat.comTechnologyDeTailRedHatCephStorageonQCTServers7testeDconfIguratIons.......................................................................................147.1QuantaGridD51PH-1ULHconfiguration.....................................................................................147.2QuantaGridT21P-4Uconfiguration............................................................................................157.3Softwareconfiguration..................................................................................................................168performancesummary.......................................................................................178.1CephBenchmarkTool(CBT)........................................................................................................178.2Clusterscale-outperformance....................................................................................................178.3Price/performance.........................................................................................................................188.4Comparingdifferentreplicationschemes................................................................................198.5Comparingdifferentjournallingconfigurations....................................................................208.640GigabitEthernetnetworkingforhigh-throughputworkloads......................................218.7Cost/capacityoptimization:relativecostperterabyte.......................................................229conclusIon..............................................................................................................2210appenDIxa:recommenDeDthroughput-optImIzeDconfIguratIons..........................................................................................................2311appenDIxb:recommenDeDcost/capacIty-optImIzeDconfIguratIons..........................................................................................................2412appenDIxc:performanceDetaIl....................................................................253redhat.comTechnologyDeTailRedHatCephStorageonQCTServersDocumEntPurPoSEThepurposeofthisdocumentistocharacterizeandcomparetheperformanceofRedHat®CephStorageonvariousQCT(QuantaCloudTechnology)servers.OptimalCephclusterconfigura-tionsareidentifiedforgeneralworkloadcategories.Asareferencearchitecture,thisdocumentprovidesdetailsonclusterhardware,software,andnetworkconfigurationcombinedwithperfor-manceresults.Thetestingmethodologyisalsoprovided,andisbasedonthestandardizedCephBenchmarkingTool,availableinaGitHubrepositoryundertheCephorganization.1Thestudydescribedhereinlargelyusedoff-the-shelfhardwareandsoftwarecomponents,anddidnotmakeadetailedstudyofchangingvariousconfigurationsettingswithinthekernel,Ceph,XFS®,orthenetwork.introDuctionAstheneedforstorageescalates,enterprisesofallkindsareseekingtoemulateefficienciesachievedbypubliccloudproviders—withtheirhighlysuccessfulsoftware-definedclouddatacen-termodelsbasedonstandardserversandopensourcesoftware.Atthesametime,the$35billionstoragemarketisundergoingafundamentalstructuralshift,withstoragecapacityreturningtotheserverfollowingdecadesofexternalNASandSANgrowth.2Software-definedscale-outstoragehasemergedasaviablealternative,wherestandardserversandindependentsoftwareunitetoprovidedataaccessandhighlyavailableservicesacrosstheenterprise.ThecombinationofQCTserversandRedHatStoragesoftwaresquarelyaddressestheseindustrytrends,andbotharealreadyattheheartofmanypublicclouddatacenters.QCTisreinventingdata-centerservertechnologytobooststoragecapacityanddensity,andredesigningscalablehardwareforcloudapplications.Astheworld’slargestenterprisesoftwarecompanywithanopensourcedevelopmentmodel,RedHathaspartneredwithseveralpubliccloudproviderstoprovideCephandGlusterstoragesoftwareinproductionenvironments.Together,QCTserversandRedHatCephStorageprovidesoftware-definedstoragesolutionsforbothprivateandpublicclouds,helpingtoacceleratetheshiftawayfromcostly,proprietaryexternalstoragesolutions.RedHatCephStoragesignificantlylowersthecostofstoringenterprisedataandhelpsenterprisesmanageexponentialdatagrowth.Thesoftwareisarobust,petabyte-scalestorageplatformforenterprisesdeployingpublicorprivateclouds.Asamodernstoragesystemforclouddeployments,RedHatCephStorageoffersmatureinterfacesforenterpriseblockandobjectstorage,makingitwellsuitedforarchival,richmedia,andcloudinfrastructureworkloadslikeOpenStack®.Deliveredinaunifiedself-healingandself-managingplatformwithnosinglepointoffailure,RedHatCephStoragehandlesdatamanagementsobusinessescanfocusonimprovingapplicationavailability.RunningRedHatCephStorageonQCTserversprovidesopeninteractionwithacommunity-basedsoftwaredevelopmentmodel,backedbythe24x7supportoftheworld’smostexperiencedopensourcesoftwarecompany.Useofstandardhardwarecomponentshelpsensurelowcosts,whileQCT’sinnovativedevelopmentmodelletsorganizationsiteratemorerapidlyonafamilyofserverdesignsoptimizedfordifferenttypesofCephworkloads.Unlikescale-upstoragesolutions,RedHatCephStorageonQCTserversletsorganizationsscaleouttothousandsofnodes,withtheabilitytoscalestorageperformanceandcapacityindependently,dependingontheneedsoftheapplicationandthechosenstorageserverplatform.1 https://github.com/ceph/cbt2 IDCWorldwideQuarterlyDiskStorageSystemsTracker,June5,20154redhat.comTechnologyDeTailRedHatCephStorageonQCTServersWorkloaD-oPtimizEDScalE-outStoragEcluStErSRedHatCephStorageonQCTserverscanbeeasilyoptimizedandsizedtoservespecificworkloadsthroughaflexiblechoiceofsystemsandcomponents.characterIzIngstorageWorkloaDsOneofthekeybenefitsofCephstorageistheabilitytoprovisiondifferenttypesofstoragepoolswithinthesamecluster,targetedfordifferentworkloads.Thisabilityallowsorganizationstotailorstorageinfrastructuretotheirchangingneeds.• Blockstoragepoolstypicallyusetriplereplicationfordataprotectiononthroughput-optimizedservers.• Objectstoragepoolstypicallyuseerasurecodingfordataprotectiononcapacity-optimizedservers.• AsIOPS-optimizedworkloadsemergeonCeph,high-IOPSserverpoolscanalsobeaddedtoaCephcluster.Table1providesthecriteriausedtoidentifyoptimalRedHatCephStorageclusterconfigurationsonQCT-basedstorageservers.Thesecategoriesareprovidedasgeneralguidelinesforhardwarepurchaseandconfigurationdecisions,andcanbeadjustedtosatisfyuniqueworkloadblendsofdif-ferentoperators.Astheworkloadmixvariesfromorganizationtoorganization,actualhardwareconfigurationschosenwillvary.table1.cephclusteroptImIzatIoncrIterIa.oPtimizationcritEriaProPErtiESExamPlEuSESioPS-oPtimizED• LowestcostperIOPS• HighestIOPS• Meetsminimumfaultdomainrecommen-dation(singleserverislessthanorequalto10%ofthecluster)• Typicallyblockstorage• 3xreplication(HDD)or2xreplication(SSD)• MySQLonOpenStackcloudsthroughPut-oPtimizED• Lowestcostpergivenunitofthroughput• Highestthroughput• HighestthroughputperBTU• Highestthroughputperwatt• Meetsminimumfaultdomainrecommen-dation(singleserverislessthanorequalto10%ofthecluster)• Blockorobjectstorage• 3xreplication• Video,audio,andimagerepositories• StreamingmediacaPacity-oPtimizED• LowestcostperTB• LowestBTUperTB• LowestwattperTB• Meetsminimumfaultdomainrecommen-dation(singleserverislessthanorequalto15%ofthecluster)• Typicallyobjectstorage• Erasurecodingcommonformaximizingusablecapacity• Objectarchive5redhat.comTechnologyDeTailRedHatCephStorageonQCTServerssIzIngsummaryforWorkloaD-optImIzeDclustersRedHatCephStorageisabletorunonmyriaddiversehardwareconfigurations.ThepurposeofthisreferencearchitecturedocumentistohelporganizationsevaluatekeyarchitecturalconceptswithcorrespondingtestresultsinordertoarchitectappropriatelysizedandoptimizedRedHatCephStorageclustersonQCTservers.Tothisend,RedHatandQCTarchitectsconductedextensiveCephtestingonvariousconfigurationsoftwoQCTservers.• QctQuantagridD51ph-1ulhserver.Idealforsmaller-capacityclusters,thecompact1rackunit(1U)QuantaGridD51PH-1ULHserverprovides12hot-swappablediskdrivesandfouradditionalhot-swappablesolidstatedrives(SSDs).• QctQuantaplext21p-4userver.TheQuantaPlexT21P-4Userverisconfigurableasasingle-node(upto78HDDs)ordual-nodesystem(upto35HDDspernode),maximizingstoragedensitytomeetthedemandforgrowingstoragecapacityinhyperscaledatacenters.Throughtesting,engineersidentifiedavarietyofthroughput-optimizedandcost/capacity-optimizedconfigurations,sizedtofittheneedsofdifferentclustersizes.Table2summarizesdifferentwork-load-optimizedconfigurationswithusablestoragecapacitiesrangingfrom100TBtomorethan2petabytes(PB).Theremainderofthisdocumentdescribeshowtheseconfigurationswereselectedandtested.table2.WorkloaDoptImIzeDconfIguratIonsofQctstorageservers.ExtraSmall(100tb*)Small(500tb*)mEDium(>1Pb*)largE(>2Pb*)IOPS-optimizedFuturedirectionFuturedirectionFuturedirectionNAThroughput-optimized7xQuantaGridD51PH-1ULH• 7U• 12x4TBHDDs• 3xSSDs• 2x10GbE• 3xreplication32xQuantaGridD51PH-1ULH• 32U• 12x4TBHDDs• 3xSSDs• 2x10GbE• 3xreplication11xQuantaPlexT21P-4U/Dual• 44U• 2x35x4TBHDDs• 2x2xPCIeSSDs• 2x1x40GbE• 3xreplication22xQuantaPlexT21P-4U/Dual• 88U• 2x35x4TBHDDs• 2x2xPCIeSSDs• 2x1x40GbE• 3xreplicationCost/capacity-optimizedNA8xQuantaGridD51PH-1ULH• 8U• 12x8TBHDDs• 0xSSDs• 2x10GbE• Erasurecoding(4:2)4xQuantaPlexT21P-4U/dual• 16U• 2x35x6TBHDDs• 0xSSDs• 2x2x10GbE• Erasurecoding(4:2)7xQuantaPlexT21P-4U/mono• 28U• 78x6TBHDDs• 0xSSDs• 2x10GbE• Erasurecoding(4:2)* Usablestoragecapacity6redhat.comTechnologyDeTailRedHatCephStorageonQCTServerscEPhDiStributEDStoragEarchitEcturEovErviEWStorageinfrastructureisundergoingtremendouschange,particularlyasorganizationsdeployinfra-structuretosupportbigdataandprivateclouds.Traditionalscale-uparraysarelimitedinscalabil-ity,andcomplexitycancompromisecost-effectiveness.Incontrast,scale-outstorageinfrastructurebasedonclusteredstorageservershasemergedasawaytodeploycost-effectiveandmanage-ablestorageatscale,withCephamongtheleadingsolutions.3Infact,cloudstoragecompaniesarealreadyusingCephatnearexabytescale,withexpectedcontinualgrowth.Forexample,Yahooesti-matesthattheirCeph-basedCloudObjectStorewillgrow20-25%annually.4IntroDuctIontocephACephstorageclusteraccommodateslargenumbersofCephnodesforscalability,fault-tolerance,andperformance.EachnodeisbasedoncommodityhardwareandusesintelligentCephdaemonsthatcommunicatewitheachotherto:• Storeandretrievedata• Replicatedata• Monitorandreportonclusterhealth• Redistributedatadynamically(remapandbackfill)• Ensuredataintegrity(scrubbing)• RecoverfromfailuresTotheCephclientinterfacethatreadsandwritesdata,aCephstorageclusterlookslikeasimplepoolwheredataisstored.However,thestorageclusterperformsmanycomplexoperationsinamannerthatiscompletelytransparenttotheclientinterface.CephclientsandCephObjectStorageDaemons(CephOSDDaemons)bothusetheCRUSH(controlledreplicationunderscalablehashing)algorithmforstorageandretrievalofobjects.cephaccessmethoDsAlldatainCeph,regardlessofdatatype,isstoredinpools.ThedataitselfisstoredintheformofobjectsviatheRADOSlayer(Figure1)which:• Avoidsasinglepointoffailure• Providesdataconsistencyandreliability• Enablesdatareplicationandmigration• Offersautomaticfaultdetectionandrecovery3 CephisandhasbeentheleadingstorageforOpenStackaccordingtoseveralsemi-annualOpenStackusersurveys.4 http://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at7redhat.comTechnologyDeTailRedHatCephStorageonQCTServersClientsApplicationsS3APISwiftAPIAdminAPIHost/VMLIBRADOSRADOSGWLIBRBDRADOSFigure1.TheReliableAutonomicDistributedObjectStore(RADOS)isthefoundationoftheCephstoragecluster.WritingandreadingdatainaCephstorageclusterisaccomplishedusingtheCephclientarchitec-ture.Cephclientsdifferfromcompetitiveofferingsinhowtheypresentdatastorageinterfaces.Awiderangeofaccessmethodsaresupported,including:• raDosgW.Abucket-basedobjectstoragegatewayservicewithS3compliantandOpenStackSwiftcompliantRESTfulinterfaces• lIbraDos.AmethodprovidingdirectaccesstoRADOSwithlibrariesformostprogramminglan-guages,includingC,C++,Java™,Python,Ruby,andPHP• rbD.ACephblockstoragedevicethatmountslikeaphysicalstoragedriveforusebybothphysi-calandvirtualsystems(withaLinux®kerneldriver,KVM/QEMUstoragebackend,oruser-spacelibraries)8redhat.comTechnologyDeTailRedHatCephStorageonQCTServerscephstoragepoolsForaCephclient,thestorageclusterisverysimple.WhenaCephclientreadsorwritesdata(referredtoasanI/Ocontext),itconnectstoastoragepoolintheCephcluster.Figure2illustratestheoverallCepharchitecture,withconceptsthataredescribedinthesectionsthatfollow.LIBRADOSRADOSRADOSGWLIBRBDClientinterfacelayerObjectsinPoolsCRUSHrulesetPlacementGroupsCephnodes:OSDhostsMonitors(MONs)MON1MON2MON3PoolID(HashObject)PoolID(HashObject)ObjObjObjObjObjObjObjObjObjObjObjObjObjObjObjObjOSD1OSD2OSD3OSD4OSD5OSD6PGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGPGCRUSHmapFigure2.ClientswritetoCephstoragepoolswhiletheCRUSHrulesetdetermineshowplacementgroupsaredistributedacrossobjectstoragedaemons(OSDs).• pools.ACephstorageclusterstoresdataobjectsinlogicalpartitionscalledpools.Poolscanbecreatedforparticulardatatypes,suchasforblockdevices,objectgateways,orsimplytosepa-rateusergroups.TheCephpooldictatesthenumberofobjectreplicasandthenumberofplace-mentgroups(PGs)inthepool.Cephstoragepoolscanbeeitherreplicatedorerasurecoded,asappropriatefortheapplicationandcostmodel.Additionally,poolscan“takeroot”atanyposi-tionintheCRUSHhierarchy,allowingplacementongroupsofserverswithdifferingperformancecharacteristics.• placementgroups.Cephmapsobjectstoplacementgroups.PGsareshardsorfragmentsofalogicalobjectpoolthatarecomposedofagroupofCephOSDdaemonsthatareinapeeringrela-tionship.Placementgroupsprovideameansofcreatingreplicationorerasurecodinggroupsofcoarsergranularitythanonaperobjectbasis.Alargernumberofplacementgroups(e.g.,100perOSD)leadstobetterbalancing.• crushruleset.TheCRUSHalgorithmprovidescontrolled,scalable,anddeclusteredplacementofreplicatedorerasure-codeddatawithinCephanddetermineshowtostoreandretrievedatabycomputingdatastoragelocations.CRUSHempowersCephclientstocommunicatewithOSDsdirectly,ratherthanthroughacentralizedserverorbroker.Bydeterminingamethodofstoringandretrievingdatabyalgorithm,Cephavoidsasinglepointoffailure,aperformancebottleneck,andaphysicallimittoscalability.9redhat.comTechnologyDeTailRedHatCephStorageonQCTServers• cephosDdaemons.InaCephcluster,CephOSDdaemonsstoredataandhandledatareplication,recovery,backfilling,andrebalancing.TheyalsoprovidesomemonitoringinformationtoCephmonitorsbycheckingotherCephOSDdaemonswithaheartbeatmechanism.ACephstorageclusterrequiresatleasttwoCephOSDdaemons(defaultisthree)toachieveanactiveandcleanstatewhentheclustermakestwocopiesofstoreddata.CephOSDdaemonsroughlycorrespondtoafilesystemonaphysicalharddiskdrive.• cephmonitors(mons).BeforeCephclientscanreadorwritedata,theymustcontactaCephMONtoobtainthemostrecentcopyoftheclustermap.ACephstorageclustercanoperatewithasinglemonitor,butthisintroducesasinglepointoffailure.Foraddedreliabilityandfaulttolerance,Cephsupportsaclusterofmonitors.Consensusamongvariousmonitorinstancesensuresconsis-tentknowledgeaboutthestateofthecluster.cephDataprotectIonmethoDsApplicationshavediverseneedsfordurabilityandavailability,anddifferentsensitivitiestodataloss.Asaresult,Cephprovidesdataprotectionatthestoragepoollevel.• replicatedstoragepools.Replicationmakesfullcopiesofstoredobjects,andisidealforquickrecovery.Inareplicatedstoragepool,Cephdefaultstomakingthreecopiesofanobjectwithaminimumoftwocopiesforcleanwriteoperations.IftwoofthethreeOSDsfail,thedatawillstillbepreservedbutwriteoperationswillbeinterrupted.• erasure-codedstoragepools.Erasurecodingprovidesasinglecopyofdataplusparity,anditisusefulforarchivestorageandcost-effectivedurability.Witherasurecoding,storagepoolobjectsaredividedintochunksusingthen=k+mnotation,wherekisthenumberdatachunksthatarecreated,misthenumberofcodingchunksthatwillbecreatedtoprovidedataprotection,andnisthetotalnumberofchunksplacedbyCRUSHaftertheerasurecodingprocess.TypicalCephread/writeoperationsfollowthestepsbelow:1.CephclientscontactaCephmonitortoverifythattheyhaveanup-to-dateversionoftheclustermap,andifnot,retrievethemostrecentchanges.2.Dataisconvertedintoobjectscontaining