Google News in R

Question

Sto cercando di ottenere informazioni da Google News. Questo è il mio codice:

library(rvest)
library(tidyverse)


news <- function(term) {

  html_dat <- read_html(paste0("https://news.google.com/search?q=",term,"&hl=es-419&gl=US&ceid=US%3Aes-419")) 

  dat <- data.frame(Link = html_dat %>%
                      html_nodes('.VDXfz') %>% 
                      html_attr('href')) %>% 
    mutate(Link = gsub("./articles/","https://news.google.com/articles/",Link))

  news_dat <- data.frame(
    Title = html_dat %>%
      html_nodes('.DY5T1d') %>% 
      html_text(),
    Link = dat$Link,
    Description =  html_dat %>%
      html_nodes('.Rai5ob') %>% 
      html_text()
  )

  return(news_dat)
}

noticias<-news("coronavirus")

Con questo codice, recupero titolo, link e descrizione. OK. Ma ho bisogno di ottenere 2 campi in più: data e media. Ad esempio, se ieri è stata pubblicata una notizia su un vaccino per il coronavirus, la data sarà quella. Se i media sono il New York Times, questo campo sarà quello. Ma non trovo questi nodi nell'HTML. Qualche idea per correggere il mio codice aggiungendo questi due campi?

Grazie in anticipo.

Answer 1

Forse prova questo

news <- function(term) {
  url <- paste0("https://news.google.com/search?q=", term, "&hl=es-419&gl=US&ceid=US:es-419")
  nodeset <- read_html(url) %>% html_nodes("article")
  tibble::tibble(
    Title = nodeset %>% html_nodes("h3") %>% html_text(), 
    Link = nodeset %>% html_nodes("h3 > a") %>% html_attr("href") %>% xml2::url_absolute(url), 
    Description = nodeset %>% html_nodes("div.Da10Tb.Rai5ob > span") %>% html_text(), 
    Source = nodeset %>% html_nodes("div.QmrVtf.RD0gLb.kybdz > div > a") %>% html_text(), 
    Time = nodeset %>% html_nodes("div.QmrVtf.RD0gLb.kybdz > div > time") %>% html_attr("datetime")
  )
}

Produzione

> news("coronavirus")

# A tibble: 100 x 5
   Title                                Link                                           Description                                        Source   Time     
   <chr>                                <chr>                                          <chr>                                              <chr>    <chr>    
 1 India reporta 41.100 casos nuevos d~ https://news.google.com/articles/CBMikwFodHRw~ "NUEVA DELHI (AP) — India reportó el domingo 41.1~ La Voz ~ 2020-11-~
 2 El ecuatoriano Diego Palacios, de L~ https://news.google.com/articles/CBMigwFodHRw~ "El defensa del LAFC, Diego Palacios, se encuentr~ ESPN De~ 2020-11-~
 3 Coronavirus: Austria endurece medid~ https://news.google.com/articles/CAIiEL2L0sxq~ "El canciller Sebastian Kurz pidió a la población~ DW (Esp~ 2020-11-~
 4 ++Coronavirus hoy: Gobierno alemán ~ https://news.google.com/articles/CAIiEKCZppoU~ "\"Todos los países que levantaron sus restriccio~ DW (Esp~ 2020-11-~
 5 ++Coronavirus hoy++ México supera e~ https://news.google.com/articles/CAIiEK8ndryG~ "El COVID-19 se consolidó como la cuarta causa de~ DW (Esp~ 2020-11-~
 6 Coronavirus en Estados Unidos: 5 ci~ https://news.google.com/articles/CAIiEFFHgJgZ~ "La incertidumbre política y la emergencia sanita~ BBC New~ 2020-11-~
 7 México supera el millón de casos de~ https://news.google.com/articles/CBMiRWh0dHBz~ "México sobrepasó el millón de casos confirmados ~ Reuters~ 2020-11-~
 8 Massachusetts reporta 2.800 casos d~ https://news.google.com/articles/CBMiXmh0dHA6~ "Los casos registrados en la más reciente jornada~ El Tiem~ 2020-11-~
 9 ¿Qué hará NYC para resistir una seg~ https://news.google.com/articles/CBMifWh0dHBz~ "Reaccionan políticos locales a la orden de cerra~ NY1 Not~ 2020-11-~
10 + Coronavirus hoy: Italia suma 544 ~ https://news.google.com/articles/CAIiEJ4KB7k2~ "Argentina registró este sábado (14.11.2020) 8.46~ DW (Esp~ 2020-11-~
# ... with 90 more rows

---

Aggiornare

Non ho mai pensato ai casi come segue:

1. Articoli annidati.

1. Manca l'attributo data-ora.

Ho aggiornato il codice per tenere conto di tutti questi casi, ma il codice diventa molto meno efficiente. Ad ogni modo, prova questo:

news <- function(term) {
  url <- paste0("https://news.google.com/search?q=", term, "&hl=es-419&gl=US&ceid=US:es-419")
  nodeset <- read_html(url) %>% html_nodes("article")
  dplyr::bind_rows(lapply(nodeset, function(x) tibble::tibble(
    Title = x %>% html_node(".ipQwMb.ekueJc.RD0gLb") %>% html_text(), 
    Link = x %>% html_node(".ipQwMb.ekueJc.RD0gLb > a") %>% html_attr("href") %>% xml2::url_absolute(url), 
    Description = x %>% html_node("div.Da10Tb.Rai5ob > span") %>% html_text(), 
    Source = x %>% html_node("div.QmrVtf.RD0gLb.kybdz > div > a") %>% html_text(), 
    Time = x %>% html_node("div.QmrVtf.RD0gLb.kybdz > div > time") %>% html_attr("datetime")
  )))
}