基于特征串的网页去重算法

基于特征串的网页去重算法 2008???i28?3SCIENCE&TECHNOLOGYINFORMATION 1.?.?? ???P?????????c?????3????,?????????????????F?L????,???L ?O???????????v?P???L??????,?u??2006?????????????????????? ?????8??????,???L?????k???&25%-40%,?L???C?????????O???? ??,?O?1?C???o???????d?I???f,?7?F?????????????u?????]?????? ????k0???Z?????t???O?,???f?????????c?????&?????????v?v???&?? ?:?]?.???T,???????3????k0?????-???e???????v?a???]????k0 2.???????L???????t???c?????? ???????????e???[???L???????t?????????????R,?o?????v???? ???8?????????O?,???f???????????,?????????v??????,?????e???? ?o???&:??????????????k2???????OURL????k2?????,???????????? ?????=????????????k0 2.1?????????????????c???????????i???C?t?????;????,?- ?W??GB2312????6763?h?W?v???????Q?R??????,???i?????????? ?l???F?????????????R??k0?-???????n?????i???????R?????u?????? ???????c?;?c???O????k0[1] ?5???c????????,?A??????k0?Q???c???e?? ?[????,?????????t???????e,????????,?????L?y???QO(n2 ),?????? ????;?????e??,???????????????????????????c?;???Lk0[2][3] 2.2???????OURL???????????????c?S???b?:?]?.???????? ?o??????,?????????????????vk0???I???????L?O?????:?]?.?????: ?]??????????URL,???&???O??URL???????l?????Q?c???O???? ??,????????k0?5???c????????,?A??????,???????????I???O????k0?Q ???c?h?s?v?????????i????????????,?L?????l???????O???3?????L ???????????tk0 2.3????????HTML?E?????,?????????????,???????????Q Levenshtein????,?c?E?????h???;?=?????F?????h???;?=???P?????? ???Y?,??????????????k0?????n?h???;?=???,?????????????L?y???Q O(n2 )k0[4] ???n?h??????Levenshtein?????Y?????h???f??,?l???Q?????c ???L??k0?????????3????,?L???c???&?s?v?T?????????i????????,?? ?????O?[?????????????L?R,???A???L?O???????%?????????L????, ????????????????????????????????k0???,,??????????,?????????, ???-????????HTML?E?????3????k0 2.4?????????=???????Q???????i?????=??????k2????????,?? ?v???H?????U??:???v7:???Z?????I???E??7U?I???i?E?F????7:????7U, ??7:????7U?????????????;?,???????????=,?Z?G?&??7:????7U?v???i?? ?????????????=???????????F?????=k0?l???????i?H?????;?d?1?L?? ????????????,?P???I???E???????c?????;??,???Q???27:????7U?M?I ???????e,?-?W???v???????;,???a???k???????=???????????i???? ??????????k0 ???Q?????=???????u???????G?????????]????????????,?????? ??????k0?L???c?????=???????u???L???????????????????3??????; ???&?s?v???????i????????,?-???3?????L?O,?-???????????i?-?H ?F???O????;???Q???3?????=???E?????????&???L?-?v???????i???? ??,?,???????v?????t?r,?&???v???b??????????????????,???????- ?????????=?????3?%??k0 3.?????????=?????????????? ?v?????????????i???????=???G,?l???????n?????????i???f?? ?????????,?Q?????S?????????????=?????f????????k0 3.17:????7U?I???;???&???????=???????c???v7:???Z?????I?? ?E??7U?I???i?E?F????7:????7U,??7:????7U?????????????;?,???????? ???=,?Z?G?&??7:????7U?v???i???????????????=???????????F???? ?=k0?l???????i?H?????;?d?1?L??????????????,?P???I???E?????? ?c?????;??,???Q???27:????7U?M?I???????e,?-?W???v???????;,?? 7:k07Uk27:?7U??7:??7U??k0 ???c7:????7U?I???;???&?????-????????????????????????k0?&?? ??7:????7U?I???;?-,???????????i???????=????,???O???????????? ?=???????L?"?????7????k0??3-1?c?????v?&???L?O??7:????7U?I???; ????????????????????: ??3-17:????7U?I???;????????????[3] ???????????H??,?&???L?O??7:????7U?I???;?????????????????? ???c?????e??k0???l??7:????7U?I???;???&???7?F?????????????????? ???e???k17.4%k0 3.2?????=?]?.???6???????7???7?L?t???L???????]?.?????? ?????????????????&?????e??????k0???L?????????n?h?????=?????? ?????e,?????e?H???T??????????????????;???????????G???h???? ???=?????????????=???????h??,?????e?H???T????????????????k0 ?Q?????I?s?v???????i??????k2?????????????????L???????i?? ???t,???????h???????i?????????&?h???????????Q???7???6?????? ??k0???&?h?????????C??:???????i?????p??(gn),???????i?????? (cn,??120?h???;?Q???h????),???????i??7:????7U??(kl)k0???8???? ??????,???????i???????i?v70??24k????????,??120?????????I ?????????3200?h?I???Q??,kl???i?v1??100????,gn???i?v1??50 ????,???a???F??200??100??50=1000000?h?????&?????6????k0 3.3????7:????7U?I???;???????e????????????????7:????7U?I?? ?;???&?????f,???????????????e??7:????7U?I???;?l?c????????7:.7Uk0 ?Z?O7:.7U?v?????????????????????-,?????????Q????,?????????Q?Y ????,?E???I???;k0????????????7:.7U?????????????=???????????,, ?????????h????????????????,???n?h???i???f?????i???&?Q?-???? ????,?-???????????L???????t????????k0 ????????:???????h7:.7U?????n??,???????????G???c??????,?l ???Q?c?Y????,?L??k0 ??????????????,?c?Y???;??,?O?????????&???E?h,?W?v?????? ?&6000?-?hk0?P??,?????v???h7:????7U?I???;?n???????????h???;?? ?Q?????=????,???????????=?????0?e???????????????=?????I?-?? ??,?????c???????????n?h???????i,???????????=???????&?????? ???????????=k0 ????????:?????????????i?????v???h7:????7U?I???;?n???-???? ?h???;??????,?????k???????;?????9??,???R???v???h7:????7U?I?? ?;?n???????&?h?????????????????=k0 4.?????? ????,?????????v?c?v?:?]?.???T???]?.?Z?????????L?????]?? ??,???????????y???L?c????k0?????O?4????????????(?H???i3??) ?????????=?????????????? ???????????? (1.???Y?e???????G?????Q???????Y????750021;2.???Y???????????Y????750002) kF????kR?v?a?v???????????????]??,???W?-???k?e?????????O??????????,?L?????u? ?????????,?O?1???v?a?;?????-?L??k0?i?????? ?&?????????L???????t???c???5?Q????????????,???1???????????????????=???? ?????3????????k0 kF???g??kR???L???????t;?????=;?????? "????"?I???;???????????? k093.5%100% ,87.0%100% ??84.8%100% k0,80.4%100% k0??76.1%100% )?IT???c????)? 40 411 2008???i28?3SCIENCE&TECHNOLOGYINFORMATION )? (?T???i411??)????????,???????v?-?Q???N?????Y,?????Q???3?( ?????=????,?????Q???3?(?????????=?]?.k0?n?h?????[??????????, ????????????????k0???,,???????I?????????=?]?.?????????I???? ?h?Q???????F,?I??????????????????k0 ??????????????????????????????,?????s?v?T???????i???-?? ??????????????,?b??????????????k0????,???????????i?????p??, 7:????7U??,???????,,???????I???????i???S???p?????????????????Q ????????????,?????c???????i??????????????k0???Z,???-?????i?? ???Y???P?????????6???-????????????????????????????k0???c???P ??????????,???????Z???????????????????????????????????yk0 kF???8???gkR [1]DellZhang,WeeSunLee.QuestionClassificationusingSupportVector Machines. [2]?l????,?F??,?????M.WEB???i?,?????????????I??????.?????Q???H?i 30???i13?3.2004.7. [3]?9????,??????,????.?????????=???e???[???????????3?????????1??.???? ?????????i17???i2?3. [4]GiuseppeAntonioDiLuccaO,MassirnilianoDiPenta,AnnaRitaFasolinoO. AnApproachtoIdentifyDuplicatedWebPages. [?(???,???7????]?? ???3?????????l???ek0?v???h?????D?Q?T???????Y,???3?????????l ???Y,?D?Q?????????R???&?v??,?D?Q???????l????k0?v???h?????D?Q ?T,???????l??(?????f?e???c5%),???????D?????e???I???v?????? ?3??,?D?Q????????????????k0????????????,???????D?Q?l???????? ?????????a,???????&?o?????[??????(????)?????k?U??????(?? ??),???O?D???????H???c?D???3??,????????????????0.1(10%)k0?? ?D?Q??????,???????o???l?????????????7?k?????T,???????????? ?????ek0?R???????????????????Q?????3???e?Y(???3????)???D?Q?? ?????????E?Ek0 ??7 6.?G?a???Z?D?Q????? ?-???T?????I??,?????H????,?????D?Q???????????c???????l ?c,???????D?Q???D???G?D?Q???????P?;?3???P???????????k?????? ???u??k0?????k???a,???????????l?c???D?Q???Q?????????????3?? ???u??,???l?c???D?Q??????1???????????T???????????I???3?? (?e??75%?????????3??)k0?Z?O,?v???-???????v?????c?L??????k0?v ???-?????H,?????????3???&???e?????,?8??,?D?Q?????&???e?3?? ?????Z??k0?v?????????H,?D?Q?v???????????-???????????v?Y?? 20%??????(?????e??????????,????90%????????????????)k0?R?? ?&???D?Q??,???c?&?????????e?3???????&????,?L?????l???D?Q?e ???I???????????Y???3???H?O?&???Y???D?Q,???????I???????????? ?????2k0?P??,?v????????????????,?l?????R?????D?Q???D??,???? ?????D?????3???P?????D?????u??k0 ?v???d?D?Q??,?&5?h?????D?????6?????7:???????6,???1?o ??,???????6,??????????k0???&?h?c?Q?????6,?????h?D?Q?????c?L ?????W??,???i?????o???6?????Y???Ik0?????o???c???????D?????, ?Q?????????????l,?G?3?????e?Y????k0?????c?l???????????????? ?????????d?G???????????????O?????????????lk0???????????????? ?????%,?D?Q?????????????????H,?D?Q???3?????k75%?k90%??, ????????????????k0???P?3????????,???????v????,???F?F???o???k ??k0???3????????,?????????????o???k??,???????e???I???????? ?lk0?-???v?Y?3?????????D??,?R???l???3???Y???-,???O???Y????, ????????????????,k0???I???R?????D?Q???o?D??,???l?????????3?? ???????L??,?R?????????????D?Q??????k0?O??,?-?????R???-,???? ?????R???D?Q???D?3?b??,???????????l????????k0 ???????????d?D?Q???????????c???W??????,?O?????????????? ???????c?O??????,?O??????,???O?????-?????????????????????k ????????k0???D?Q?????D????????,?R???l???????D?Q???????????? ????k0?????????????????-,?????????????l???-,?-?????Z?????? ??,?R???l???????k???Z?D?Q??????k0 7.?????????D?Q?????D??? ?R?????8?????d?D???f,?????T?c?&???f,???????Q???h?f???O ?D???????d?D?P???????D??????,???l?c?????O???O?D???????d?Dk0 ???d?D???c???;????,?D???v???3???e?f????????????k0???D???c?? ???????;?????????&???f?e??????70%?????f?D??(?????f?????? ?u)k0 ??8 ?????R???v???h???3?????????I?D??,???????h???3???l?&???? ?I?D???Q0,?&???f?l?-???R,????9,?D???l???H?9?????Ik0 ??9 ???h???3???????D?????-,?D?????&???f?l????k0????,?????-?? ???Z??(?????????????????????-)???Z?????D?????-?R?????Z???? ?D?Q???D??k0?v?D?Q???????-???????3?????e?Y,?Z?G?????????Z??, ???Z???l???Z???????-???D??,???????D?Q???D???G?3???P?????u ??,??????????????,???l???????D?Q????k0???P?3???????,???Z???l ?????????D?????????I,?o?c???????D?Q???D???G?3?????u??,???D ?Q?o?c???????????????Hk0?R?????W???????????????????????????l ???k????????,?3?????,?L??,???W?????????3?????2,?-???????? ?2k0???????3???v???????Y???Z?D?Q???D????,?????????????l?c?? ?2?D?????6,???R?????c??k0 kF???8???gkR [1]???c??.?D?????????????Z????,?T?????-?e???o?,,?Q?????-?????r,2005?? 9?N. [2]?7???3.?D?Q???I?G???????P,?6?????D?e???????r,2006??2?N1??. [3]??????.???????????I?????G????,?Q?????-?????r,1996-12-1. [?(???,??:?7????] ?? )? )? )??Q???G?D??)? 3 : 40