当前位置：首页 > news >正文

爬取某省2019年的疫情数据

news 2026/3/26 17:56:54

1.明确目标

爬取某省2019年的疫情数据：
网站：www.ncovdata.spbeen.com

2.项目创建：

要求说明

使用scrapy命令新建爬虫项目，名称为ncovdata,打开终端，在终端创建项目

scrapy startproject ncovdata

3.创建spider文件

要求说明

使用scrapy命令新建爬虫文件，名称为ncov:

scrapy genspider-t basic ncov www.ncovdata.spbeen.com

4.启动爬虫

使用scrapy命令在项目内执行爬虫文件
创建Scrapy项目中的启动脚本，方便后续调试
在终端执行：

scrapy crawl ncov

在顶级目录下创建一个脚本run.py:

fromscrapy.cmdlineimportexecute#execute(["","",""])execute("scrapy crawl ncov".split())

5.分析Scrapy的输出日志

日志结构说明

顶部：版本信息，项目结构信息
中部：日志输出，例如下载器组建的请求记录
底部：爬虫过程的统计记录，包含请求和失败的次数

#顶部#Scrapy版本2025-12-2215:27:14[scrapy.utils.log]INFO:Scrapy2.12.0started(bot:ncovdata)#依赖的库2025-12-2215:27:14[scrapy.utils.log]INFO:Versions:lxml5.4.0.0,libxml22.11.9,cssselect1.3.0,parsel1.10.0,w3lib2.3.1,Twisted24.11.0,Python3.10.3(tags/v3.10.3:a342a49,Mar162022,13:07:40)[MSC v.192964bit(AMD64)],pyOpenSSL25.0.0(OpenSSL3.4.111Feb2025),cryptography44.0.2,Platform Windows-10-10.0.19045-SP02025-12-2215:27:14[scrapy.addons]INFO:Enabled addons:[]#异步请求库2025-12-2215:27:14[asyncio]DEBUG:Using selector:SelectSelector#队列信息，调度器内容2025-12-2215:27:14[scrapy.utils.log]DEBUG:Using reactor:twisted.internet.asyncioreactor.AsyncioSelectorReactor2025-12-2215:27:14[scrapy.utils.log]DEBUG:Using asyncio event loop:asyncio.windows_events._WindowsSelectorEventLoop2025-12-2215:27:14[scrapy.utils.log]DEBUG:Using reactor:twisted.internet.asyncioreactor.AsyncioSelectorReactor2025-12-2215:27:14[scrapy.utils.log]DEBUG:Using asyncio event loop:asyncio.windows_events._WindowsSelectorEventLoop#cd4613b32575bd40这个是telnet的密码2025-12-2215:27:14[scrapy.extensions.telnet]INFO:Telnet Password:cd4613b32575bd40#启动的拓展插件2025-12-2215:27:15[scrapy.middleware]INFO:Enabled extensions:['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.logstats.LogStats']#项目设置信息2025-12-2215:27:15[scrapy.crawler]INFO:Overridden settings:{'BOT_NAME':'ncovdata','FEED_EXPORT_ENCODING':'utf-8','NEWSPIDER_MODULE':'ncovdata.spiders','SPIDER_MODULES':['ncovdata.spiders'],'TWISTED_REACTOR':'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}# 下载器中间件2025-12-2215:27:15[scrapy.middleware]INFO:Enabled downloader middlewares:['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware','scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']#代表爬虫中间件2025-12-2215:27:15[scrapy.middleware]INFO:Enabled spider middlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']2025-12-2215:27:15[scrapy.middleware]INFO:Enabled item pipelines:#代表管道[]#spider opened到Closing spider(finshed)为中间过程2025-12-2215:27:15[scrapy.core.engine]INFO:Spider opened#打开2025-12-2215:27:15[scrapy.extensions.logstats]INFO:Crawled0pages(at0pages/min),scraped0items(at0items/min)#这行的作用是定期输出一条日志，INFO后的信息为一分钟爬取了多少页，一分钟几页2025-12-2215:27:15[scrapy.extensions.telnet]INFO:Telnet console listening on127.0.0.1:6023#它代表telnet的一个服务终端，用于拓展插件用的2025-12-2215:27:15[scrapy.core.engine]DEBUG:Crawled(200)<GET http://ncovdata.spbeen.com/>(referer:None)#请求网址#finished代表停止，表示爬虫结束2025-12-2215:27:15[scrapy.core.engine]INFO:Closing spider(finished)#关闭#底部# 这个大括号为状态收集器收集的整体信息2025-12-2215:27:15[scrapy.statscollectors]INFO:Dumping Scrapy stats:{'downloader/request_bytes':220,'downloader/request_count':1,'downloader/request_method_count/GET':1,'downloader/response_bytes':57244,'downloader/response_count':1,'downloader/response_status_count/200':1,'elapsed_time_seconds':0.432231,'finish_reason':'finished','finish_time':datetime.datetime(2025,12,22,7,27,15,641880,tzinfo=datetime.timezone.utc),'items_per_minute':None,'log_count/DEBUG':6,'log_count/INFO':10,'response_received_count':1,'responses_per_minute':None,'scheduler/dequeued':1,'scheduler/dequeued/memory':1,'scheduler/enqueued':1,'scheduler/enqueued/memory':1,'start_time':datetime.datetime(2025,12,22,7,27,15,209649,tzinfo=datetime.timezone.utc)}# 日志都是日期+时间+信息的结构，中括号中的代表组件，INFO为日志的类型，分号后的为具体信息2025-12-2215:27:15[scrapy.core.engine]INFO:Spider closed(finished)

默认情况下，下载器中间件，和爬虫中间件都不需要改，重点是管道，从管道开始，到爬虫中部内容会非常多，如果运行失败，底部收集器的内容总结不会出现

6.编写并运行爬虫测试

任务说明

请求接口并拿到接口数据，多多益善
解析接口数据，转换成python的字典对象
从目标json内容中提取出所需数据

数据要求说明

获取某个省10天连续数据
单条数据：{“江西省”：{“xx”:“xx”,“xx”:“xx”}}
重新构建start_urls函数

importscrapyimportdatetimeclassNcovSpider(scrapy.Spider):name="ncov"allowed_domains=["ncovdata.spbeen.com"]#start_urls = ["http://ncovdata.spbeen.com/"]base_url="http://ncovdata.spbeen.com/apis/get_china_provinces/?query={}"defstart_requests(self):start_date_str='2020-01-26'start_date=datetime.datetime.strptime(start_date_str,'%Y-%m-%d')foriinrange(0,15):current_date=start_date+datetime.timedelta(days=i)current_url=self.base_url.format(current_date.strftime('%Y-%m-%d'))print(i,current_url)yieldscrapy.Request(url=current_url,callback=self.parse)defparse(self,response):pass

7.获取数据并用item保存

importscrapyimportdatetimeimportjsonclassNcovSpider(scrapy.Spider):name="ncov"allowed_domains=["ncovdata.spbeen.com"]# start_urls = [# "http://ncovdata.spbeen.com/apis/get_china_provinces/?query_date=2020-01-31",# ]base_url="http://ncovdata.spbeen.com/apis/get_china_provinces/?query_date={}"defstart_requests(self):start_date_str='2020-01-26'start_date=datetime.datetime.strptime(start_date_str,"%Y-%m-%d")foriinrange(0,15):current_date=start_date+datetime.timedelta(days=i)current_url=self.base_url.format(current_date.strftime("%Y-%m-%d"))# return scrapy.Request(url=current_url)yieldscrapy.Request(url=current_url)defparse(self,response):province="江西"data_list=response.json().get('data',[])forprovince_dictindata_list:# k, v = province_dict.items()ifprovinceinstr(province_dict):item=province_dict item['source']=response.textbreakelse:item={}item['source']=''# print(item)returnitem

8.管道文件存储数据

任务说明

配置settings.py代码，启用管道文件
使用管道文件接收item,并输出
将item的数据进行存储到本地的txt文件中
1.在settings文件中打开管道

进入pipelines.py文件输出item查看内容

输出结果：

新建一个data文件夹，将数据写入文件夹

fromitemadapterimportItemAdapterimportdatetimeimportjsonclassNcovdataPipeline:current_date_str=datetime.datetime.now().strftime("%Y%m%d%H%M%S")defprocess_item(self,item,spider):withopen('data/{}.txt'.format(self.current_date_str),mode='a',encoding='utf8')asfile:file.write(json.dumps(item,ensure_ascii=False))file.write('\n')returnitem