Python-Selenium-WebDriverWait를 사용하여 HTML 텍스트가있는 웹 스크랩 테이블

Question

다음 웹 사이트의 직원이 500 명 이상인 회사 이름을 모두 웹 스크랩하려고합니다.

https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1

첫 번째 사이트의 회사 이름을 긁어내는 코드를 작성하면 스크립트가 "다음 사이트 버튼"을 클릭하고 이름을 다시 긁어냅니다. 이름은 목록에 저장되며 목록에 특정 수의 이름이 포함될 때까지 발생합니다. 그런 다음 목록을 데이터 프레임으로 전송하고 xsl 파일로 내 보냅니다. 불행히도 지금은 이것을하지 않습니다. 다음은 코드입니다.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 20:

    company_name = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='zebraTable zebraTable--companies']//following::tr[2]/td[@class='zebraTable__td zebraTable__td--companyName']/a"))).get_attribute("innerHTML")
    
    for p in range(len(company_name)):
        company_list.append(company_name)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
    print(company_list)

    df = pd.DataFrame(company_list,columns =['Unternehmensname']) 

    df.to_excel("output.xlsx")  
            
    time.sleep(5)

그리고 내 출력은 다음과 같습니다.

['\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ']

.get_attribute ()는 하나의 속성 만 가져 오기 때문에 그럴 것이라고 생각하지만이 시점에서 모든 속성을 얻는 방법을 모릅니다.

inb4 감사합니다

Answer 1

예를 사용 .get_attribute()하면 한 번에 하나의 속성 만 얻을 수 있습니다. 모든 속성을 얻으려면 javascript 코드 아래에서 할 수 있습니다.

driver.execute_script('var items = {}; for (index = 0; index < arguments[0].attributes.length; ++index) { items[arguments[0].attributes[index].name] = arguments[0].attributes[index].value }; return items;', ele)

여기 ele는 웹 요소입니다.

모든 회사 이름을 인쇄하려면 아래 접근 방식을 사용할 수 있습니다.

company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
for cn in company_names:
    print(cn.text)

참고 : 첫 페이지에 모든 회사 이름이 인쇄됩니다. 모든 페이지에서 이름을 얻으려면 각 페이지에서 다음 페이지 아이콘을 클릭하고 루프에서 위의 코드를 클릭해야합니다.

Answer 2

find_elements_by_css_selector메소드를 사용 하여 여러 웹 요소 (표시된 모든 회사 이름)를 찾을 수 있습니다 .

모든 것을 작성하지는 않겠지 만 while 루프의 시작은 다음과 같아야합니다.

companies = driver.find_elements_by_css_selector(".zebraTable__td--companyName")

그런 다음 companies목록을 반복 하고 각 목록 구성원에 대한 속성을 가져와야합니다.

Answer 3

다음 코드를 사용하여 처음 100 개의 회사 이름으로 Excel 목록을 생성했습니다.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 100:
    
    company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
    
    for cn in company_names:
        company_list.append(cn.text)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
   
    df = pd.DataFrame(company_list, columns =['Unternehmensname'])
    
    df.to_excel("output.xlsx")  
            
    time.sleep(5)

고마워요