Python web scraping varias páginas

Question

Estoy raspando todas las palabras del sitio web Merriam-Webster .

Quiero raspar todas las páginas a partir de az y todas las páginas dentro de ellas y guardarlas en un archivo de texto. El problema que tengo es que solo obtengo el primer resultado de la tabla en lugar de todos. Sé que esta es una gran cantidad de texto (alrededor de 500k) pero lo hago para educarme.

CÓDIGO:

import requests
from bs4 import BeautifulSoup as bs

URL = 'https://www.merriam-webster.com/browse/dictionary/a/'

page = 1
# for page in range(1, 75):

req = requests.get(URL + str(page))
soup = bs(req.text, 'html.parser')
containers = soup.find('div', attrs={'class', 'entries'})
table = containers.find_all('ul')

for entries in table:
    links = entries.find_all('a')
    name = links[0].text
    print(name)

Ahora lo que quiero es obtener todas las entradas de esta tabla, pero en cambio solo obtengo la primera entrada.

Estoy un poco atrapado aquí, así que cualquier ayuda sería apreciada. Gracias

https://www.merriam-webster.com/browse/medical/a-z
https://www.merriam-webster.com/browse/legal/a-z
https://www.merriam-webster.com/browse/dictionary/a-z
https://www.merriam-webster.com/browse/thesaurus/a-z

Answer 1

Para obtener todas las entradas, puede usar este ejemplo:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/a/'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for a in soup.select('.entries a'):
    print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

Huellas dactilares:

(a) heaven on earth            https://www.merriam-webster.com/dictionary/%28a%29%20heaven%20on%20earth
(a) method in/to one's madness https://www.merriam-webster.com/dictionary/%28a%29%20method%20in%2Fto%20one%27s%20madness
(a) penny for your thoughts    https://www.merriam-webster.com/dictionary/%28a%29%20penny%20for%20your%20thoughts
(a) quarter after              https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20after
(a) quarter of                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20of
(a) quarter past               https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20past
(a) quarter to                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20to
(all) by one's lonesome        https://www.merriam-webster.com/dictionary/%28all%29%20by%20one%27s%20lonesome
(all) choked up                https://www.merriam-webster.com/dictionary/%28all%29%20choked%20up
(all) for the best             https://www.merriam-webster.com/dictionary/%28all%29%20for%20the%20best
(all) in good time             https://www.merriam-webster.com/dictionary/%28all%29%20in%20good%20time

...and so on.

---

Para raspar varias páginas:

url = 'https://www.merriam-webster.com/browse/dictionary/a/{}'

for page in range(1, 76):
    soup = BeautifulSoup(requests.get(url.format(page)).content, 'html.parser')
    for a in soup.select('.entries a'):
        print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

---

EDITAR: Para obtener todas las páginas de la A a la Z:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'

for char in range(ord('a'), ord('z')+1):
    page = 1
    while True:
        soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
        for a in soup.select('.entries a'):
            print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

        last_page = soup.select_one('[aria-label="Last"]')['data-page']
        if last_page == '':
            break

        page += 1

---

EDITAR 2: Para guardar en archivo:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'


with open('data.txt', 'w') as f_out:
    for char in range(ord('a'), ord('z')+1):
        page = 1
        while True:
            soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
            for a in soup.select('.entries a'):
                print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

                print('{}\t{}'.format(a.text, 'https://www.merriam-webster.com' + a['href']), file=f_out)

            last_page = soup.select_one('[aria-label="Last"]')['data-page']
            if last_page == '':
                break

            page += 1

Answer 2

Creo que necesitas otro bucle:

for entries in table:
    links = entries.find_all('a')
    for name in links:
        print(name.text)