對于B站，目前網上的爬蟲大多都是使用通過解析服務器的響應來爬取想要的內容，下面我們通過使用XPath來爬取B站上一些想要的信息

此次任務我們需要對B站搜索到的關鍵字，并爬取搜索的視頻時間、播放量、彈幕量等信息

分析B站html框架

打開B站后，搜索關鍵字并按下F12進入開發者模式，就能看到頁面的html代碼，需要在這些代碼中找到需要爬取的信息。

點擊右上角的箭頭圖片，再點擊想要爬取內容的信息，就會自動跳轉到對應的html代碼上。

獲取內容

找到想要爬取的信息就得獲取信息的XPath表達式，這兒可以通過如下圖方法快速得到表達式。

這樣就可以得到該位置的XPath表達式了。

由于第一頁XPath表達式與后面頁的XPath表達式有些許的不同，需要通過對鏈接的驗證來使用不同的表達式

完整代碼

import requests
from lxml import etree
import time
import random
import csv
import pandas as pdheaders = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36'}result = pd.DataFrame()urls = ['https://search.bilibili.com/all?vt=69174939&keyword=%E5%A4%A7%E6%95%B0%E6%8D%AE&from_source=webtop_search&spm_id_from=333.1007&search_source=2','https://search.bilibili.com/all?keyword=%E7%89%A9%E8%81%94%E7%BD%91%E5%B7%A5%E7%A8%8B&from_source=webtop_search&spm_id_from=333.1007&search_source=2','https://search.bilibili.com/all?vt=69174939&keyword=%E7%94%B5%E5%AD%90%E7%A7%91%E5%AD%A6%E4%B8%8E%E6%8A%80%E6%9C%AF&from_source=webtop_search&spm_id_from=333.1007&search_source=2','https://search.bilibili.com/all?vt=69174939&keyword=%E8%99%9A%E6%8B%9F%E7%8E%B0%E5%AE%9E&from_source=webtop_search&spm_id_from=333.1007&search_source=2','https://search.bilibili.com/all?vt=691740939&keyword=%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD&from_source=webtop_search&spm_id_from=333.1007&search_source=2',
]url_key = [len(i) + 6 for i in urls]
for index, url in enumerate(urls):for page in range(1, 10):html = requests.get(url, headers=headers)print(url)bs = etree.HTML(html.text)if url[-8:-1] == 'source=':items = bs.xpath('//*[@id="i_cecream"]/div/div[2]/div[2]/div/div/div/div[3]/div')else:items = bs.xpath('//*[@id="i_cecream"]/div/div[2]/div[2]/div/div/div[1]')for i in range(1, 43):try:time = items[0].xpath(f'div[{i}]/div/div[2]/div/div/p/a/span[2]')[0].textexcept:time = Nonetry:up_author = items[0].xpath(f'div[{i}]/div/div[2]/div/div/p/a/span[1]')[0].textexcept:up_author = Nonetry:title = items[0].xpath(f'div[{i}]/div/div[2]/div/div/a/h3/@title')[0]except:title = Nonetry:href = items[0].xpath(f'div[{i}]/div/div[2]/div/div/a/@href')[0]except:href = Nonetry:Playback_volume = items[0].xpath(f'div[{i}]/div/div[2]/a/div/div[2]/div/div/span[1]/span')[0].textexcept:Playback_volume = Nonetry:Barrage_volume = items[0].xpath(f'div[{i}]/div/div[2]/a/div/div[2]/div/div/span[2]/span')[0].textexcept:Barrage_volume = Nonetry:Video_duration = items[0].xpath(f'div[{i}]/div/div[2]/a/div/div[2]/div/span')[0].textexcept:Video_duration = Noneprint(time, title, up_author, href, Playback_volume, Barrage_volume, Video_duration)df = pd.DataFrame({'time': [time], 'title': [title], 'up_author': [up_author], 'href': [href],'Playback_volume': [Playback_volume], 'Barrage_volume': [Barrage_volume],'Video_duration': [Video_duration]})result = pd.concat([result, df])if url[-8:-1] == 'source=':url = url + '&page=2&o=36'else:new_page = int(url[url_key[index]]) + 1url = url[:url_key[index]] + f'{new_page}&o={(new_page - 1) * 36}'
result.to_excel("F:/B站數據.xlsx", index=False)