|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了,ebay、amazon等一些站,采集先看robots.txt,从robots.txt中找到sitemap的索引,剩下的就想办法搞吧1 G# _+ u# q1 w! y
http://www.ebay.com/robots.txt # sitemaps - SRPs
) e* M- W0 P: ?2 b* {Sitemap: http://www.ebay.com/lst/SRP_US_index.xml
7 d- | ^0 f0 FSitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml
4 C, Q: d$ l- ^/ {8 Q! d2 V2 C* b( `5 V1 n) G7 [
# Guides sitemaps# u: a6 u" I# p0 j) A' @8 p! |1 w
Sitemap: http://www.ebay.com/lst/GUIDES-0-index.xml( K7 N& Z6 }! W9 m
) J1 h' }' W+ [/ U# SSRP sitemaps
( [$ n2 P2 `& l: {( }8 z I: NSitemap: http://www.ebay.com/lst/SSRP-0-index.xml3 I! U9 e# N- w
) k+ x, f& r: @! w- l
#Stores Sitemaps& a/ x/ b' t6 v& N; M- N9 t" t
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml* j) q& N/ f0 W" g: O
- T/ u4 E5 S6 L#BHP Sitemaps
. R. Q$ N2 ?+ b6 R" t+ qSitemap: http://www.ebay.com/lst/BHP-0-index.xml! ^6 `$ i& q0 ?5 B+ X* @6 C+ H9 @/ M
; o3 f7 R$ E& S9 g: I
#Collections- A# \+ I) W& J1 p# U! j2 y) f2 _
Sitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml
0 O1 P6 H' X. W \2 U
- Y0 X9 O2 `; l" T#VI
( E9 l d5 ?) }/ {: u: x' xSitemap: http://www.ebay.com/lst/VI-0-index.xml
/ O2 N& X( X$ d
% x% K7 y- G9 @6 n4 }, Q' g; K1 ?#PRP
9 F& H2 Q( [# ?6 {5 aSitemap: http://www.ebay.com/lst/PRP-0-index.xml
2 e" }- ^4 L- ]# o' R
' D. t! D1 r/ }8 M0 F" U, F! p |
|