🚩简单轻量化文件爬虫
背景
当我们需要在一个网址下载多个文件如pdf、图片时，如果数量少，我们就简单挨个下载就行了，但是如果超出几十个，再挨个下载，就会出现下不全、下载慢的问题，这个工具就是用来自动化下载的一个工具，当然网上有很多教程，我也是将其中的部分代码整合了一下，形成适合自己使用的工具。
直接上代码
import re
import os
import requestsdef fileCrawler(your_website_url, save_folder):if not os.path.exists(save_folder):os.makedirs(save_folder)headers = {"User-Agent":"Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Mobile Safari/537.36","Cookie":"BAIDUID=229A18B7534A5CEA671381D45FCDC530:FG=1; BIDUPSID=229A18B7534A5CEA671381D45FCDC530; PSTM=1592693385; BDRCVFR[dG2JNJb_ajR]=mk3SLVN4HKm; userFrom=null; BDRCVFR[-pGxjrCMryR]=mk3SLVN4HKm; H_WISE_SIDS=149389_148867_148211_149537_146732_138426_150175_147527_145599_148186_147715_149253_150045_149280_145607_148660_146055_110085; delPer=0; BDORZ=AE84CDB3A529C0F8A2B9DCDD1D18B695; ysm=10315; IMG_WH=626_611; __bsi=8556698095607456048_00_14_R_R_17_0303_c02f_Y", }response = requests.get(your_website_url, headers)print(response)#返回结果为418，200为请求成功，418则是对方发现咱们是爬虫了content = response.content.decode('utf-8')# 如果是图片需要分析图片的url，自己写正则link_pattern = r'https://unsplash-assets.imgix.net/modules/.*?.jpg|https://images.unsplash.com/placeholder-avatars/.*?.jpg|https://unsplash.com/favicon-.*?.png|https://unsplash.com/mstile-.*?.png|https://images.unsplash.com/opengraph/.*?.png'detail_url = re.findall(link_pattern,content,re.DOTALL)detail_url_remove_dup = list(set(detail_url))for one_url in detail_url_remove_dup:try:img_res = requests.get(one_url)img_res.raise_for_status()img_filename = os.path.join(save_folder, one_url.split('/')[-1])with open(img_filename, 'wb') as f:f.write(img_res.content)print(f"downloading imgs: {img_filename}")except requests.RequestException as e:print(f"downloading imgs {one_url} occured error: {e}") # 这里有个bug，如果url是一个很长的字符串，这里会全部打印下来，引起很不舒服的感觉，建议直接打印个报错就行print("done!")return 0if __name__ == "__main__":your_website_url = "https://unsplash.com/"save_folder = 'downloaded_images'fileCrawler(your_website_url, save_folder)
关键点
该工具的关键点在于分析下载文件的url，要写一下正则，这是最重要也是最麻烦的地方