|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了,ebay、amazon等一些站,采集先看robots.txt,从robots.txt中找到sitemap的索引,剩下的就想办法搞吧$ E/ e8 I" J3 L, ?9 v t' x
http://www.ebay.com/robots.txt # sitemaps - SRPs
3 l% ]- s5 Q4 h% ASitemap: http://www.ebay.com/lst/SRP_US_index.xml' {# \8 k2 R' j( y
Sitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml
5 k* P7 H; {( [$ s R7 ]
; e6 ^& ~' \* g& a: V# Guides sitemaps, H6 Y* s) L' Z/ w+ D8 X3 A
Sitemap: http://www.ebay.com/lst/GUIDES-0-index.xml
7 N+ v |4 r( v: A8 w- L8 ?
; Q2 J1 E. q( B h! Y- O4 W# SSRP sitemaps
$ n6 j; r; h- t) U- mSitemap: http://www.ebay.com/lst/SSRP-0-index.xml
6 m. F ^2 Z4 v0 {$ C" z p3 C+ z# U
#Stores Sitemaps( {) J7 f9 T% ^; h
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml" P6 H" y/ O2 k! W) X+ n d
6 |. i8 X, i7 y, T R0 K#BHP Sitemaps
5 p/ Z9 \; t `$ {$ h. ySitemap: http://www.ebay.com/lst/BHP-0-index.xml1 y8 S) Y" ~5 F( ]5 n$ p
& I j/ s8 W/ b; f. C; z( @#Collections+ R8 T8 u7 n4 V; x- [# }
Sitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml; O& X- G, a4 G% n) Q: r- x
, ]7 A3 o2 J/ k8 h5 g5 A, { B#VI4 H, G& c: m$ y4 H% G' V. q
Sitemap: http://www.ebay.com/lst/VI-0-index.xml+ i6 C6 U4 g) ?! Z$ o( A
& m, y$ g) J, ]3 r7 t d9 _#PRP, J8 w. X* C: G9 m! X
Sitemap: http://www.ebay.com/lst/PRP-0-index.xml / Z; G% X7 H9 |' l7 J+ ?9 f
8 v( G) J3 H; r/ q) {1 p |
|