|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了,ebay、amazon等一些站,采集先看robots.txt,从robots.txt中找到sitemap的索引,剩下的就想办法搞吧
- g2 f; x7 ^: k) uhttp://www.ebay.com/robots.txt # sitemaps - SRPs
9 K* A% ]# X( ^, N# rSitemap: http://www.ebay.com/lst/SRP_US_index.xml
* G: O0 x) F& u: S# h8 a9 PSitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml% k4 F3 I( F" ?, Y; D' P U
" W$ g3 q! n. A. t9 ~5 v4 @
# Guides sitemaps- D9 K3 r7 Z3 Y' @. `+ I( q, ^
Sitemap: http://www.ebay.com/lst/GUIDES-0-index.xml
0 P7 R+ Z0 K0 I# S4 [+ X( O8 g% k$ [6 _# o
# SSRP sitemaps9 n4 A8 k1 j3 l& p; e
Sitemap: http://www.ebay.com/lst/SSRP-0-index.xml
" {4 N' ?" b b& N [
. |% e0 k, \( X ]+ K% t1 d- r' M* Z#Stores Sitemaps6 D" o* r* N1 ]& J/ _
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml
& R& [/ Z& u# s' a0 M; N* J8 e. Q
#BHP Sitemaps
7 n+ H$ x2 [' {: zSitemap: http://www.ebay.com/lst/BHP-0-index.xml2 X3 B9 B* B3 K
+ N g/ P+ n& n- z _0 M2 m. R: {; s#Collections# @ Q; c4 C9 T& O9 c
Sitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml
- c2 J, o& p2 Y d) o. y
% @( F' U- M/ ^( [7 I% `#VI
" D: y4 k9 E& c8 @& ` ZSitemap: http://www.ebay.com/lst/VI-0-index.xml3 t* X2 P" h6 k/ c& ]9 K
4 B3 L0 N( ]. h#PRP
: V& y K; s& `Sitemap: http://www.ebay.com/lst/PRP-0-index.xml ) b2 p6 w) i6 d: L9 d
" ~/ ]2 _& P+ F2 s |
|