Web Python raclant plusieurs pages

Question

Je racle tous les mots du site Web Merriam-Webster .

Je veux gratter toutes les pages à partir de az et toutes les pages qu'elles contiennent et les enregistrer dans un fichier texte. Le problème que j'ai est que je n'obtiens que le premier résultat de la table au lieu de tout. Je sais que c'est une grande quantité de texte (environ 500k) mais je le fais pour m'éduquer.

CODE:

import requests
from bs4 import BeautifulSoup as bs

URL = 'https://www.merriam-webster.com/browse/dictionary/a/'

page = 1
# for page in range(1, 75):

req = requests.get(URL + str(page))
soup = bs(req.text, 'html.parser')
containers = soup.find('div', attrs={'class', 'entries'})
table = containers.find_all('ul')

for entries in table:
    links = entries.find_all('a')
    name = links[0].text
    print(name)

Maintenant, ce que je veux, c'est obtenir toutes les entrées de cette table, mais à la place je n'obtiens que la première entrée.

Je suis un peu coincé ici, donc toute aide serait appréciée. Merci

https://www.merriam-webster.com/browse/medical/a-z
https://www.merriam-webster.com/browse/legal/a-z
https://www.merriam-webster.com/browse/dictionary/a-z
https://www.merriam-webster.com/browse/thesaurus/a-z

Answer 1

Pour obtenir toutes les entrées, vous pouvez utiliser cet exemple:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/a/'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for a in soup.select('.entries a'):
    print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

Impressions:

(a) heaven on earth            https://www.merriam-webster.com/dictionary/%28a%29%20heaven%20on%20earth
(a) method in/to one's madness https://www.merriam-webster.com/dictionary/%28a%29%20method%20in%2Fto%20one%27s%20madness
(a) penny for your thoughts    https://www.merriam-webster.com/dictionary/%28a%29%20penny%20for%20your%20thoughts
(a) quarter after              https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20after
(a) quarter of                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20of
(a) quarter past               https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20past
(a) quarter to                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20to
(all) by one's lonesome        https://www.merriam-webster.com/dictionary/%28all%29%20by%20one%27s%20lonesome
(all) choked up                https://www.merriam-webster.com/dictionary/%28all%29%20choked%20up
(all) for the best             https://www.merriam-webster.com/dictionary/%28all%29%20for%20the%20best
(all) in good time             https://www.merriam-webster.com/dictionary/%28all%29%20in%20good%20time

...and so on.

---

Pour gratter plusieurs pages:

url = 'https://www.merriam-webster.com/browse/dictionary/a/{}'

for page in range(1, 76):
    soup = BeautifulSoup(requests.get(url.format(page)).content, 'html.parser')
    for a in soup.select('.entries a'):
        print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

---

EDIT: Pour obtenir toutes les pages de A à Z:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'

for char in range(ord('a'), ord('z')+1):
    page = 1
    while True:
        soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
        for a in soup.select('.entries a'):
            print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

        last_page = soup.select_one('[aria-label="Last"]')['data-page']
        if last_page == '':
            break

        page += 1

---

EDIT 2: Pour enregistrer dans un fichier:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'


with open('data.txt', 'w') as f_out:
    for char in range(ord('a'), ord('z')+1):
        page = 1
        while True:
            soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
            for a in soup.select('.entries a'):
                print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

                print('{}\t{}'.format(a.text, 'https://www.merriam-webster.com' + a['href']), file=f_out)

            last_page = soup.select_one('[aria-label="Last"]')['data-page']
            if last_page == '':
                break

            page += 1

Answer 2

Je pense que vous avez besoin d'une autre boucle:

for entries in table:
    links = entries.find_all('a')
    for name in links:
        print(name.text)