python3: bs4 a des problèmes avec certains sites Web
Oct 04 2020
J'apprends python et bs4.
Suite à quelques suggestions et à de nombreux sites Web, j'ai écrit ce script:
import requests as rq
from bs4 import BeautifulSoup
header = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0'}
def get_price(site):
html = rq.get(site, headers=header).text
soup = BeautifulSoup(html, 'html.parser')
try:
price = soup.find(id="priceblock_ourprice").get_text()
print(site)
print(price)
except:
print(site)
print("failed")
sites = ["https://www.amazon.in/Apple-iPhone-11-64GB-Green/dp/B07XVKBY68/ref=sr_1_7?keywords=iphone+11&qid=1573668357&sr=8-7",
"https://www.amazon.it/Apple-iPhone-64GB-Verde-Ricondizionato/dp/B082DN72G3/ref=sr_1_19?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-19",
"https://www.amazon.it/Apple-iPhone-11-128GB-Verde/dp/B07XS5MSW4/ref=sr_1_1_sspa?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExNlhGMElFNUhJMTBJJmVuY3J5cHRlZElkPUEwMTI2OTMxMVpXWEtHQ1o5S0ZENCZlbmNyeXB0ZWRBZElkPUEwOTMyMTczMVdMMzlQOTRPTUE3SCZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=" ]
for site in sites:
get_price(site)
print("\n")
Je le lance et je reçois ceci:
https://www.amazon.in/Apple-iPhone-11-64GB-Green/dp/B07XVKBY68/ref=sr_1_7?keywords=iphone+11&qid=1573668357&sr=8-7
₹ 64,499.00
https://www.amazon.it/Apple-iPhone-64GB-Verde-Ricondizionato/dp/B082DN72G3/ref=sr_1_19?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-19
failed
https://www.amazon.it/Apple-iPhone-11-128GB-Verde/dp/B07XS5MSW4/ref=sr_1_1_sspa?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExNlhGMElFNUhJMTBJJmVuY3J5cHRlZElkPUEwMTI2OTMxMVpXWEtHQ1o5S0ZENCZlbmNyeXB0ZWRBZElkPUEwOTMyMTczMVdMMzlQOTRPTUE3SCZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=
749,00 €
Je n'arrive pas à comprendre pourquoi le deuxième site ne devrait pas fonctionner
La chaîne priceblock_ourpriceest présente:
$ wget -q -O - 'https://www.amazon.it/Apple-iPhone-64GB-Verde-Ricondizionato/dp/B082DN72G3/ref=sr_1_19?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-19' 2>&1 | grep \"priceblock_ourprice\"
<span id="priceblock_ourprice" class="a-size-medium a-color-price priceBlockBuyingPriceString">629,00 €</span>
Réponses
3 AndrejKesely Oct 04 2020 at 05:32
Le problème est qu'Amazon sert du HTML qui html.parserne peut pas analyser correctement. La solution est d'utiliser lxmlou d' html5libanalyser:
import requests as rq
from bs4 import BeautifulSoup
header = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0'}
def get_price(site):
html = rq.get(site, headers=header).text
soup = BeautifulSoup(html, 'lxml') # <--- use 'lxml' or 'html5lib' parser
try:
price = soup.find(id="priceblock_ourprice").get_text()
print(site)
print(price)
except:
print(site)
print("failed")
sites = ["https://www.amazon.in/Apple-iPhone-11-64GB-Green/dp/B07XVKBY68/ref=sr_1_7?keywords=iphone+11&qid=1573668357&sr=8-7",
"https://www.amazon.it/Apple-iPhone-64GB-Verde-Ricondizionato/dp/B082DN72G3/ref=sr_1_19?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-19",
"https://www.amazon.it/Apple-iPhone-11-128GB-Verde/dp/B07XS5MSW4/ref=sr_1_1_sspa?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExNlhGMElFNUhJMTBJJmVuY3J5cHRlZElkPUEwMTI2OTMxMVpXWEtHQ1o5S0ZENCZlbmNyeXB0ZWRBZElkPUEwOTMyMTczMVdMMzlQOTRPTUE3SCZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=" ]
for site in sites:
get_price(site)
print("\n")
Impressions:
https://www.amazon.in/Apple-iPhone-11-64GB-Green/dp/B07XVKBY68/ref=sr_1_7?keywords=iphone+11&qid=1573668357&sr=8-7
₹ 64,499.00
https://www.amazon.it/Apple-iPhone-64GB-Verde-Ricondizionato/dp/B082DN72G3/ref=sr_1_19?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-19
744,89 €
https://www.amazon.it/Apple-iPhone-11-128GB-Verde/dp/B07XS5MSW4/ref=sr_1_1_sspa?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=iphone+11&qid=1601755114&sr=8-1-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExNlhGMElFNUhJMTBJJmVuY3J5cHRlZElkPUEwMTI2OTMxMVpXWEtHQ1o5S0ZENCZlbmNyeXB0ZWRBZElkPUEwOTMyMTczMVdMMzlQOTRPTUE3SCZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=
749,00 €