Python-爬蟲-requests

簡介

#介紹：使用requests可以模擬瀏覽器的請求，比起之前用到的urllib，requests模塊的api更加便捷（本質就是封裝了urllib3）#注意：requests庫發送請求將網頁內容下載下來以后，并不會執行js代碼，這需要我們自己分析目標站點然后發起新的request請求#安裝：pip3 install requests#各種請求方式：常用的就是requests.get()和requests.post()
>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r = requests.post('http://httpbin.org/post', data = {'key':'value'})
>>> r = requests.put('http://httpbin.org/put', data = {'key':'value'})
>>> r = requests.delete('http://httpbin.org/delete')
>>> r = requests.head('http://httpbin.org/get')
>>> r = requests.options('http://httpbin.org/get')

1、GET請求

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

# 1、無參數實例 import?requests ret?=?requests.get('https://github.com/timeline.json') print?ret.url print?ret.text # 2、有參數實例 import?requests payload?=?{'key1':?'value1',?'key2':?'value2'} ret?=?requests.get("http://httpbin.org/get", params=payload) print?ret.url print?ret.text

向 https://github.com/timeline.json 發送一個GET請求，將請求和響應相關均封裝在 ret 對象中。

2、POST請求

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

# 1、基本POST實例 import?requests payload?=?{'key1':?'value1',?'key2':?'value2'} ret?=?requests.post("http://httpbin.org/post", data=payload) print?ret.text # 2、發送請求頭和數據實例 import?requests import?json url?=?'https://api.github.com/some/endpoint' payload?=?{'some':?'data'} headers?=?{'content-type':?'application/json'} ret?=?requests.post(url, data=json.dumps(payload), headers=headers) print?ret.text print?ret.cookies

向https://api.github.com/some/endpoint發送一個POST請求，將請求和相應相關的內容封裝在 ret 對象中。

3、其他請求

1 2 3 4 5 6 7 8 9 10

requests.get(url, params=None,?**kwargs) requests.post(url, data=None, json=None,?**kwargs) requests.put(url, data=None,?**kwargs) requests.head(url,?**kwargs) requests.delete(url,?**kwargs) requests.patch(url, data=None,?**kwargs) requests.options(url,?**kwargs) # 以上方法均是在此方法的基礎上構建 requests.request(method, url,?**kwargs)

requests模塊已經將常用的Http請求方法為用戶封裝完成，用戶直接調用其提供的相應方法即可

Practice

import requests
import re
'''
請求方式:get、post、put…
參數：params、headers、proxies、cookies、data
'''
rsp=requests.get("https://www.hellobi.com/")
ck=requests.utils.dict_from_cookiejar(rsp.cookies)
title=re.compile("<title>(.*?)</title>",re.S).findall(rsp.text)
hd={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0"}
px={"http":"http://127.0.0.1:8888","https":"http://127.0.0.1:8888",}
rsp=requests.get("https://www.hellobi.com/",proxies=px,headers=hd,cookies=ck)
key={"wd":"韋瑋",}
rsp=requests.get("http://www.baidu.com/s",headers=hd,cookies=ck,params=key)
title=re.compile("<title>(.*?)</title>",re.S).findall(rsp.text)postdata={"name":"測試賬號","pass":"測試密碼"}
rsp=requests.post("http://www.iqianyue.com/mypost/",data=postdata)

　　

?