经典：OSIsoft-PI-时序数据库核心技术

OSIsoftOverviewDevelopmentsinthePISystemFamilyofProductsGreggLeBlancPISystemProductManagerWhyScalability?MoredatacanbetrackedeverydayPhoneshaveGPSlocationdataReal-timemonitoringinthehomeElectricalusageSecuritysystemsFire/FlooddetectionITApplicationsEverymachineonanetworkneedsmonitoringExample:700,000homesx10datapoints=7,000,000pointsBenefitsITMonitoringcandetectIntrusionsImpendinghardwareproblemsDiagnosesoftwareproblemsVirustrafficExample:BecauseWiredCitycoulddetectandrecordtheamountofinboundtrafficfromtheNimdaworm,theygotarefundfromMCIWorldcomWhereisPIToday?100,000+DataStreams100’sofsimultaneousclients300+Interfaces6,500+Customers1,000’sofeventsrecorded/secIssuesWithinScalabilityAscalablesystemhasseveralaspectsUsersservedCalculationsandanalysisIntegrationissuesDatacapacityDatathroughputSecurityInfrastructureUserLoad&IntegrationUsersneedfastaccesstothedataClient/ServerusersStatelessUsersClient/ServerusersconnectdirectlytoPIStatelessUsersBroaderaudiencePIICEWebservicesandintegrationServerManager(PISDK)SafelyexposesPIdataThePITrustTableretainspointbypointsecurityConnectionmanagementbalancesuserrequestsInternetInformationServer(IIS)BrowserSecurityinICEPIWebServicePISDKSOAPMessagesPIEnterpriseServerPIPointSecurityWindowsAuthentication(AggregatedQueries)PITrustTableConnectionsinICEElvispidemoNortonpiadminAlicepialiceRickypiadminPISystemLucypidemoICENorton,RickyAliceElvis,LucyPISDKServerObjects1peruniquePIuser1TCP/IPConnectionperserverPISDKSessionManagerUnderlyingWebServiceStructurePIICEWebServicesLayerICEWebPartsICEWebPartsICEWebPartsICEWebPartsPISDKPIServerOtherHistoriansRelationalDatabasesDCSSCADALIMS…LooselyCoupledApplicationsTightlyCoupledWebServicesLayerLooselyCoupledComputingTheideaofWebservicesAllowsforcomputinginanenvironmentthatis:AsynchronousStatelessPlatformindependentGeographicallyindependentWillreplacesome“tightlycoupled”computingTightlyCoupledComputingInfrastructure-leveltiesbetweensystemsUsesAPIcallsUsuallyproprietaryNotasflexibleExamples:PIInterfacestootherinformationsystemsOLEDB/ODBCconnectorsPoint-to-Point,Middleware,andotherintegrationWebServiceIntegrationLooselycoupledapplicationsbuiltonWebServiceIntegrationSpaceTraditionalTightlyCoupledIntegrationSpaceICEKPIReportsAuditingERPQualityWebServicesPIBatchWebservicesRelationalDBWebservicesPIAuditingWebservicesERPWebservicesPIICEWebservicesWebServicesIntegrationSpaceInfrastructureOSIsoftismovingto.NETOlderapplicationswillmigrateNewapplicationswillleveragethe.NETFrameworkManagementtoolswilluse.NETNewtoolscomingforSMT(SystemsManagementTools)PartsofthePISystemwillbeaccessiblethrough.NETDistributedAnalysisServerBasedPerformanceEquationsDesktopBasedSpreadsheetsbasedonDataLinkDistributedCalculationsPIACEServerloadreducedCalculationscanbereusedSecurityConcernsIsn’tMicrosoftavulnerability?Overthelastyear26vulnerabilitiesinApache22inMicrosoft’sIISKeyissue:systemmaintenanceWhataboutNimda?250,000systemsin9hoursAtleast$2.4BindamagesInfectionstartedJuly13,2001PatchwasavailableJune18,2001(MS01-033)SecurityResponseMoreaccountability:MicrosoftNIPC(NationalInfrastructureProtectionCenter)workswithNERCandMicrosofttodevelopsecurityproceduresandstandardsKeyprecautionsstillremainGuardagainstsocialengineeringSystemmaintenanceUnknownvulnerabilitiesWiFi,wardrivingUnprotectedmodemlinesusedforsupportOSIsoftSecurityPIPointSecurityTrustTableSingledirectionPItoPItransfersAuditingAnewdatabasethatrecordschangestoPIPointsValuesModulechangesWhat’sBeenCoveredHandlinguserloadCalculationsIntegrationdemandSecurityInfrastructureMillionpointPISystemsHistoryoftheHistorianPI11983HP-10001985Vax-VMS1988PINetClient/ServerArchitecturePI21992Vax-VMS1994Alpha-OpenVMSPI31993DesignWorkHistoryoftheHistorianPI3Big4UnixHP-UXIBM-AIXDec-OSF/1Sun-SolarisWindowsNT3.51DevelopmentLanguageC++HistoryoftheHistorianPI3KeyconceptsofPI2wereusedSnapshotCompressionArchivecacheArchivenavigationPINetHistoryoftheHistorianPI3KeynewconceptsMulti-processRPCBasedInter-processcommunicationPISDKHistorizationofmanydifferentdatatypesDoublesStringsBLOBsCommoncodebaseforallplatforms64bitsupportPI3OriginalReleasePI3.0ReleasedNovember19951,000to100,000+PointsUpto2000eventspersecondsPI3.0toPI3.3Howmuchcanyoudoin7years?ConcentratedonfeaturesAlarmACETotalizerPISDKBatchDatabaseModuleDatabaseAuditNTSecurityCOMConnectorsDevelopmentInfra-structureAutomatedbuildsandtestingBugfixes(justafew)Scaling1996Typicalsystem<100KpointsDatarates<2000Events/Second2002Manysystemsapproaching150KpointsUpto10,000Events/SecondScaling1995Intel®Pentium®ProProcessor2P200MHz2002Intel®Xeon™Processor4P1.6GHzScalingThePIdataratesreflectincreaseinprocessorspeed.PointCountdoesnot.Limitationisnotprocessorspeed.LimitationsinPI3.3Memory2GBofvirtualmemoryperprocessArchivecacheInter-processcommunicationCompressedeventsfromsnapshottoarchiveSerializationofRPCsArchivesub-systemhandlesonecallatatimeMemoryPIDatabasesarememoryresidentPointdatabaseModuledatabaseSnapshotPIArchiveCachePI3.3CacheMemoryIssuesCacherecordcanbequitelarge.AddingasingleeventrequiresentirecacherecordSystemsreceivingdataformostpointswillhavealargememoryfootprintLotsofcacheactivityPushingrecordsoutofmemoryReadingrecordsintomemoryPI3.4ArchiveCacheTypicalPISystemMajorityofpointsreceivedataregularlyAddeventsNeweventsreceivedinorder,andnearcurrenttimeThePI3.4CacheAddressesMemoryIssuesSmallermemoryfootprintMoreefficienthandlingofneweventsSameefficiencyreadingdataDesignTradeoffsMorecomplicated3.3Snapshot–ArchiveInter-processCommunicationCurrentMechanismEventreceivedisprocessedbythecompressionalgorithm.Ifpassescompressioneventisaddedtoin-memoryeventqueueEventsinqueuearepackagedandsentviaanRPCtothearchiveprocess3.3SnapshotBufferingArchiveunabletoprocesscompressedeventsBackupsExtremelyhighdataratesSystemproblemsSnapshotmustwriteto“eventqueue”fileArchivemustreadfromtheSnapshoteventqueueResult2physicaldiskI/O’shappenonsamefile3.4Snapshot–ArchiveInter-processCommunicationChangedphysicalfiletomemorymappedfileFilesystemfeatureMappedbySnapshotandArchiveSnapshotwritescompressedeventsto“file”Archivereadscompressedeventsfrom“file”SynchronizationtechniquesrequiredtopreventcorruptionBufferingcasehandledbyfilesystemLimitation3:RPCSerializationPI3.3allsub-systemsaresinglethreadedRequeststosub-systemsareserializedArchiveexample—3simultaneouscallsArchiveSummaryPlotvaluesCompressedvaluesRPCSerializationSolutionMultipleThreadsThreadsaretheobviousapproachUnixandNThaveverypowerfulandeasytousethreadingmodelsConceptuallySimpleBut,easytowritebugsMulti-ThreadinginPI3.4PI3hadsomethreadingsincetheoriginalreleasePINetManagerSub-systemsuseathreadforreadingmessagesPI3.4Sub-systemLevelthreadmodelMain“housekeeping”threadReadthreadMessagepumpPoolofworkerthreadsMulti-ThreadinginPI3.4PI3.4Sub-systemLevelthreadmodelEverysub-systemismulti-threadedSub-systemmustimplementlocksforoptimalperformanceWorkerthreadpoolisconfigurableRuntimeKillthreadsSuspendthreadsChangepriorityAddworkerthreadsDeleteworkerthreadsMulti-ThreadinginPI3.4Whatdoesthisgetus?Thearchivecallfrom….Won’tkillyoursystem.Whatwill10archivecallsform….dotomysystem?Longtime1=100Longtime2=1134289piar_summary(1023453,&time1,&time2,&rval,&pctgood,ARCTOTAL)PIon64BitWindowsOneadvantage:largeraddressablememory32bit:2GB64bit:16–128GBXP-64:16GB.NetEnterpriseServer:64GB.NetDataCenter:128GBPIon64BitWindowsDisadvantages:SlowerVerysensitivetoproperlyoptimizedcodeServersonly64bitWindowsisnotintendedtoreplace32bitdesktops.Onlymakessensetoportserverapplications64bitdevelopmentwillbedoneon32bitmachinesPIon64BitWindowsPI3isdesignedfor64bitSupportsDecOSF/1OriginalreleaseofPIsupportedDec’s64bitUnixandstilldoes—HPTru64Unix.PI3.4willbe64bitreadyMinordifferencesbetweenWindowsandUnix64bit3.4willaddresstheseminorissues3.4willcompilecleanlySummaryPointCountlimitedbyarchivecachememory3.4archivecacheissignificantlysmallerandmoreefficientMemorymappedfilesforconfigurationdataPointDatabaseTestedmillionpointsystemontypical32bitserverclassmachine2P1GHzPentium4/4GBMemorySeveralmillionpointspossible64bit,perhaps50Millionpoints?SummarySnapshot–Archiveinter-processcommunicationSignificantlyfastertheRPCmechanismMemorymappedfilesharedbybothprocessesBackedbydisktoinsurenodatalossRPCSerializationEliminatedbythreadingProcessseveralcallssimultaneouslyQuestions?个人观点供参考，欢迎讨论个人观点供参考，欢迎讨论