Python web birden çok sayfayı kazıma

Question

Merriam-Webster web sitesindeki tüm kelimeleri topluyorum .

Z'den başlayarak tüm sayfaları ve içlerindeki tüm sayfaları kazıyıp bir metin dosyasına kaydetmek istiyorum. Yaşadığım sorun, tablonun hepsinden ziyade yalnızca ilk sonucunu almam. Bunun büyük miktarda metin olduğunu biliyorum (yaklaşık 500k) ama bunu kendimi eğitmek için yapıyorum.

KOD:

import requests
from bs4 import BeautifulSoup as bs

URL = 'https://www.merriam-webster.com/browse/dictionary/a/'

page = 1
# for page in range(1, 75):

req = requests.get(URL + str(page))
soup = bs(req.text, 'html.parser')
containers = soup.find('div', attrs={'class', 'entries'})
table = containers.find_all('ul')

for entries in table:
    links = entries.find_all('a')
    name = links[0].text
    print(name)

Şimdi istediğim, bu tablodaki tüm girişleri almak, bunun yerine sadece ilk girişi alıyorum.

Burada sıkışıp kaldım, bu yüzden herhangi bir yardım takdir edilecektir. Teşekkürler

https://www.merriam-webster.com/browse/medical/a-z
https://www.merriam-webster.com/browse/legal/a-z
https://www.merriam-webster.com/browse/dictionary/a-z
https://www.merriam-webster.com/browse/thesaurus/a-z

Answer 1

Tüm girişleri almak için şu örneği kullanabilirsiniz:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/a/'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for a in soup.select('.entries a'):
    print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

Baskılar:

(a) heaven on earth            https://www.merriam-webster.com/dictionary/%28a%29%20heaven%20on%20earth
(a) method in/to one's madness https://www.merriam-webster.com/dictionary/%28a%29%20method%20in%2Fto%20one%27s%20madness
(a) penny for your thoughts    https://www.merriam-webster.com/dictionary/%28a%29%20penny%20for%20your%20thoughts
(a) quarter after              https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20after
(a) quarter of                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20of
(a) quarter past               https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20past
(a) quarter to                 https://www.merriam-webster.com/dictionary/%28a%29%20quarter%20to
(all) by one's lonesome        https://www.merriam-webster.com/dictionary/%28all%29%20by%20one%27s%20lonesome
(all) choked up                https://www.merriam-webster.com/dictionary/%28all%29%20choked%20up
(all) for the best             https://www.merriam-webster.com/dictionary/%28all%29%20for%20the%20best
(all) in good time             https://www.merriam-webster.com/dictionary/%28all%29%20in%20good%20time

...and so on.

---

Birden çok sayfayı kazımak için:

url = 'https://www.merriam-webster.com/browse/dictionary/a/{}'

for page in range(1, 76):
    soup = BeautifulSoup(requests.get(url.format(page)).content, 'html.parser')
    for a in soup.select('.entries a'):
        print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

---

DÜZENLE: A'dan Z'ye tüm sayfaları almak için:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'

for char in range(ord('a'), ord('z')+1):
    page = 1
    while True:
        soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
        for a in soup.select('.entries a'):
            print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

        last_page = soup.select_one('[aria-label="Last"]')['data-page']
        if last_page == '':
            break

        page += 1

---

DÜZENLEME 2: Dosyaya kaydetmek için:

import requests
from bs4 import BeautifulSoup


url = 'https://www.merriam-webster.com/browse/dictionary/{}/{}'


with open('data.txt', 'w') as f_out:
    for char in range(ord('a'), ord('z')+1):
        page = 1
        while True:
            soup = BeautifulSoup(requests.get(url.format(chr(char), page)).content, 'html.parser')
            for a in soup.select('.entries a'):
                print('{:<30} {}'.format(a.text, 'https://www.merriam-webster.com' + a['href']))

                print('{}\t{}'.format(a.text, 'https://www.merriam-webster.com' + a['href']), file=f_out)

            last_page = soup.select_one('[aria-label="Last"]')['data-page']
            if last_page == '':
                break

            page += 1

Answer 2

Sanırım başka bir döngüye ihtiyacın var:

for entries in table:
    links = entries.find_all('a')
    for name in links:
        print(name.text)