豆瓣读书Python爬虫项目优化版
接着上篇文章（Python爬虫项目实战-使用常用库爬取豆瓣读书 Top 250），我们把它升级成一个“麻雀虽小，五脏俱全”的教学级工程：
基于上个豆瓣读书爬虫项目，同时集成四大进阶方向，并且保持结构清晰、可运行、可扩展。
⚠️ 再次提醒：仅供学习与课程演示，请勿高频请求豆瓣。
项目名称
DoubanBookSpider-Pro
一个集Scrapy 工程化 + 分布式去重 + 反爬策略 + 异步高性能 于一体的豆瓣读书爬虫
一、项目结构（重点）
douban_book_spider_pro/ ├── scrapy.cfg ├── requirements.txt ├── Dockerfile ├── docker-compose.yml └── douban_book_spider_pro/ ├── __init__.py ├── settings.py ├── pipelines.py ├── middlewares.py ├── items.py ├── db.py └── spiders/ ├── __init__.py ├── top250_spider.py # Scrapy 主爬虫 ├── async_spider.py # aiohttp 异步爬虫 ├── distributed_spider.py # Redis 分布式爬虫 └── anti_spider_demo.py # 反爬策略演示二、依赖清单（requirements.txt）
scrapy>=2.11 redis>=5.0 fake-useragent requests beautifulsoup4 lxml pyquery selenium playwright aiohttp aioredispip install -r requirements.txt playwright install三、Item 定义（items.py）
import scrapy class BookItem(scrapy.Item): title = scrapy.Field() author = scrapy.Field() publisher = scrapy.Field() rating = scrapy.Field() detail_url = scrapy.Field()四、数据库封装（db.py）
import redis REDIS_HOST = "localhost" REDIS_PORT = 6379 def get_redis(): return redis.Redis(host=REDIS_HOST, port=REDIS_PORT, decode_responses=True)五、方向一：Scrapy 工程化（top250_spider.py）
import scrapy from douban_book_spider_pro.items import BookItem class Top250Spider(scrapy.Spider): name = "top250" allowed_domains = ["book.douban.com"] start_urls = ["https://book.douban.com/top250"] def parse(self, response): for item in response.css(".item"): book = BookItem() book["title"] = item.css(".title a::attr(title)").get() book["author"] = item.css(".author::text").get(default="").strip() book["rating"] = item.css(".rating_nums::text").get() book["detail_url"] = item.css(".title a::attr(href)").get() yield book next_page = response.css(".next a::attr(href)").get() if next_page: yield response.follow(next_page, self.parse)✅ 体现：
Spider 规范化
Item 封装
Pipeline 可扩展
自动翻页
六、方向二：分布式去重（distributed_spider.py）
from scrapy_redis.spiders import RedisSpider from douban_book_spider_pro.items import BookItem class DistributedSpider(RedisSpider): name = "distributed" redis_key = "douban:start_urls" def parse(self, response): for item in response.css(".item"): book = BookItem() book["title"] = item.css(".title a::attr(title)").get() book["rating"] = item.css(".rating_nums::text").get() yield booksettings.py 关键配置
SCHEDULER = "scrapy_redis.scheduler.Scheduler" DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" REDIS_HOST = "localhost" REDIS_PORT = 6379启动多个爬虫实例即可实现横向扩展。
七、方向三：反爬策略（middlewares.py）
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware from fake_useragent import UserAgent import random class RandomUserAgentMiddleware(UserAgentMiddleware): def __init__(self, *args, **kwargs): self.ua = UserAgent() def process_request(self, request, spider): request.headers["User-Agent"] = self.ua.random class RandomDelayMiddleware: def process_request(self, request, spider): import time time.sleep(random.uniform(0.5, 1.5))settings.py 启用
DOWNLOADER_MIDDLEWARES = { 'douban_book_spider_pro.middlewares.RandomUserAgentMiddleware': 400, 'douban_book_spider_pro.middlewares.RandomDelayMiddleware': 500, }✅ 包含：
UA 随机化
请求间隔
可扩展代理池（省略示例）
八、方向四：异步高性能（async_spider.py）
import aiohttp import asyncio from bs4 import BeautifulSoup from douban_book_spider_pro.items import BookItem URL = "https://book.douban.com/top250" async def fetch(session, url): async with session.get(url) as resp: return await resp.text() async def parse_html(html): soup = BeautifulSoup(html, "lxml") for item in soup.select(".item"): book = BookItem() book["title"] = item.select_one(".title a")["title"] book["rating"] = item.select_one(".rating_nums").text print(book) async def main(): async with aiohttp.ClientSession() as session: html = await fetch(session, URL) await parse_html(html) if __name__ == "__main__": asyncio.run(main())✅ 特点：
非阻塞 IO
高并发
适合大规模抓取
九、Docker 化部署（Dockerfile）
FROM python:3.10-slim WORKDIR /app COPY . . RUN pip install -r requirements.txt && playwright install --with-deps CMD ["scrapy", "crawl", "top250"]# docker-compose.yml version: "3" services: spider: build: . depends_on: - redis redis: image: redis:7十、整体数据流总结
aiohttp / Scrapy ↓ UA + 延迟 + 代理 ↓ Redis 去重 ↓ BookItem ↓ Pipeline ↓ JSON / DB通过这个项目，你了解了：
✅ Scrapy 工程化架构
✅ Redis 分布式爬虫
✅ 常见反爬策略
✅ 异步高性能爬虫
✅ Docker 化部署思路
下一步你可以做什么？
✅ 把Playwright / Selenium 无缝接入 Scrapy Downloader Middleware
✅ 把数据写入MySQL / MongoDB / Elasticsearch
✅ 加一个前端可视化（Flask + ECharts）
✅ 讲清楚Scrapy vs aiohttp 性能对比与选型