Python - Selenium - tabella webscrape con testo in html utilizzando WebDriverWait

Question

Cerco di scrivere sul web tutti i nomi delle società con 500 o più dipendenti del seguente sito web:

https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1

Ho scritto un codice per raschiare i nomi delle società del primo sito e lo script farà quindi clic sul pulsante "Sito successivo" e scriverà nuovamente i nomi. I nomi verranno salvati in un elenco e ciò accadrà finché l'elenco non avrà un certo numero di nomi al suo interno. Quindi trasferirà l'elenco in un dataframe e lo esporterà in un xslfile. Purtroppo al momento non lo fa. Ecco il codice

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 20:

    company_name = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='zebraTable zebraTable--companies']//following::tr[2]/td[@class='zebraTable__td zebraTable__td--companyName']/a"))).get_attribute("innerHTML")
    
    for p in range(len(company_name)):
        company_list.append(company_name)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
    print(company_list)

    df = pd.DataFrame(company_list,columns =['Unternehmensname']) 

    df.to_excel("output.xlsx")  
            
    time.sleep(5)

E il mio output è simile a questo:

['\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ']

Penso che sia perché .get_attribute () ottiene solo un attributo, ma non so come ottenere tutti gli attributi a questo punto.

inb4 Grazie

Answer 1

Sì, utilizzando .get_attribute()puoi ottenere solo un attributo alla volta. Per ottenere tutti gli attributi puoi sotto il codice javascript:

driver.execute_script('var items = {}; for (index = 0; index < arguments[0].attributes.length; ++index) { items[arguments[0].attributes[index].name] = arguments[0].attributes[index].value }; return items;', ele)

Ecco ele è il tuo elemento web.

Per stampare tutto il nome dell'azienda puoi utilizzare l'approccio seguente:

company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
for cn in company_names:
    print(cn.text)

Nota: stamperà tutti i nomi delle società sulla prima pagina. Se si desidera ottenere nomi da tutta la pagina, è necessario fare clic sull'icona della pagina successiva su ciascuna pagina e fare clic sul codice sopra in un ciclo.

Answer 2

Puoi utilizzare il find_elements_by_css_selectormetodo per trovare più elementi web (tutti i nomi di società visualizzati.

Non scriverò tutto, ma l'inizio del ciclo while dovrebbe assomigliare a questo:

companies = driver.find_elements_by_css_selector(".zebraTable__td--companyName")

Quindi dovresti scorrere l' companieselenco e ottenere gli attributi per ogni membro dell'elenco.

Answer 3

Ha funzionato con il seguente codice, che genera un elenco Excel con i primi 100 nomi di società:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 100:
    
    company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
    
    for cn in company_names:
        company_list.append(cn.text)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
   
    df = pd.DataFrame(company_list, columns =['Unternehmensname'])
    
    df.to_excel("output.xlsx")  
            
    time.sleep(5)

grazie mille ragazzi