|
|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了,ebay、amazon等一些站,采集先看robots.txt,从robots.txt中找到sitemap的索引,剩下的就想办法搞吧& h- e: m. \. u2 u
http://www.ebay.com/robots.txt # sitemaps - SRPs
+ g2 U( q' ]% A; ESitemap: http://www.ebay.com/lst/SRP_US_index.xml
) o# T% T2 _5 [5 ASitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml
& M# i, G4 Z {5 I5 P9 ?' Z8 z; C3 N9 e
# Guides sitemaps
) [$ ?2 b4 y- Z' x( b. H# oSitemap: http://www.ebay.com/lst/GUIDES-0-index.xml4 m- S2 v( M% i7 A: Q$ K i
$ i. a0 A3 S* m2 Z+ M
# SSRP sitemaps& N, g7 N/ r! g) `$ @
Sitemap: http://www.ebay.com/lst/SSRP-0-index.xml
7 E2 a1 J1 Z* i' j+ _8 y
R% W0 k* `! _$ I3 }. S( h#Stores Sitemaps& }" {0 y# q8 `
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml$ H2 }7 ~* f1 s( p. b# ~
) {8 U2 F2 Y# J) e; d( D
#BHP Sitemaps
% _# s* L, @$ f% u8 a+ b/ xSitemap: http://www.ebay.com/lst/BHP-0-index.xml
+ L# _' G, w+ p
- a" n' b4 o- Y#Collections7 {( p7 g5 h. {" E, {
Sitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml4 J% @% y+ _0 p! z" P- E$ @, B6 c
% B5 }) l& l7 T& E6 k8 D- p0 S#VI
* x* \, X8 E, J& z8 j5 I! \6 rSitemap: http://www.ebay.com/lst/VI-0-index.xml& V$ Y1 _9 z7 M, s
9 V5 r2 ~' T9 y o! F) s
#PRP+ z5 _. p2 R9 j3 x* d8 {
Sitemap: http://www.ebay.com/lst/PRP-0-index.xml " g/ ]" ~4 V N3 a2 V9 {( D* M
7 a5 d+ Y$ } G# A" Q |
|