Chapter 3 Creating Corpus

Linguistic data are important to linguists. Data usually tell us something we don’t know, or something we are not sure of.

While there are many existing text data collections (cf. Structured Corpus and XML), chances are that sometimes you still need to collect your own data for a particular research question.

But please note that when you are creating your own corpus for specific research questions, always pay attention to the three important criteria: representativeness, authenticity, and size.

In this chapter, we will look at a quick way to extract linguistic data from web pages (i.e., web crawling), which are by now undoubtedly the largest sources of textual data.

Following the spirit of tidy , we will mainly do our tasks with the libraries of tidyverse and rvests.

If you are new to tidyverse R, please check its official webpage for learning resources.

## Uncomment the following line for installation
# install.packages(c("tidyverse", "rvest"))
library(tidyverse)
library(rvest)

3.1 HTML Structure

The HyperText Markup Language, or HTML is the standard markup language for documents designed to be displayed in a web browser.

3.1.1 HTML Syntax

To illustrate the structure of the HTML, please download the sample html file from: demo_data/data-sample-html.html and first open it with your browser.

<!DOCTYPE html>
<html>
  <head>
  <title>My First HTML </title>

  </head>
  
  <body>
    <h1> Introduction </h1>
    <p> Have you ever read the source code of a html page? This is how to get back to the course page: <a href="https://alvinntnu.github.io/NTNU_ENC2036_LECTURES/", target="_blank">ENC2036</a>. </p>
    <h1> Contents of the Page </h1>
    <p> Anything you can say about the page.....</p>
  </body>
</html>

An HTML document includes several important elements (cf. Figure 3.1):

  • DTD: document type definition which informs the browser about the version of the HTML standard that the document adheres to (e.g., <!DOCTYPE HTML>)
  • element: the combination of start tag, content, and end tag (e.g, <title>My First HTML</title>)
  • tag: named braces that enclose the content and define its structural function (e.g., title, body, p)
  • attribute: specific properties of the tag, which are often placed in the start end of the element (e.g., <a href= "index.html"> Homepage </a>). They are expressed as name = "value" pairs.
Syntax of An HTML Tag Element

Figure 3.1: Syntax of An HTML Tag Element

An HTML document starts with the root element <html>, which splits into two branches, <head> and <body>.

  • Most of the webpage textual contents would go into the <body> part.
  • Most of the web-related codes and metadata (e.g., javascripts, CSS) are often included in the <head> part.

All elements need to be strictly nested within each other in a well-formed and valid HTML file, as shown in Figure 3.2.

Tree Structure of An HTML Document

Figure 3.2: Tree Structure of An HTML Document

3.1.2 Tags and Attributes

HTML has plenty of legal tags and attributes. On W3CSchools, there is a complete list of HTML tags and attributes for your reference. Common tags may include:

  • Anchor tag <a>
  • Metadata tag <meta>
  • External tag <link>
  • Emphasizing tags <b>, <i>, <strong>
  • Paragraph tags <p>
  • Heading tags <h1>, <h2>, <h3>
  • Listing content tags <ul>, <ol>
  • Block tags <div>, <span>
  • Form-related tag <form>, <input>
  • Foreign script tag <script>
  • Table tag <table>, <th>, <tr>, <td>

You probably don’t have to know all the detail about the HTML tags and elements. However, in order to scrape the textual data from the Internet, you need to know at least from which parts of HTML elements you need your textual data from on the web pages.

Usually, before you scrape the data from the webpage, bear the following questions in mind:

  • From which HTML elements/tags would you like to extract the data for corpus construction?
  • Do you need the textual content of the HTML element? (e.g., the textual content of <p> ... </p>)
  • Do you need a specific attribute of the HTML element? (e.g., the hyperlink attribute of <a href="https://www.example.com">Visit Example Website</a>)

3.1.3 CSS

Cascading Style Sheet (CSS) is a language for describing the layout of HTML and other markup documents (e.g., XML).

HTML + CSS is by now the standard way to create and design web pages. The idea is that CSS specifies the formats/styles of the HTML elements. The following is an example of the CSS:

div.warnings {
  color: pink;
  font-family: "Arial"
  font-size: 120%
}

h1 {
  padding-top: 20px
  padding-bottom: 20px
}

You probably would wonder how to link a set of CSS style definitions to an HTML document. There are in general three ways: inline, internal and external. You can learn more about this in W3School.com.

Here I will show you an example of the internal method. Below is a CSS style definition for <h1>.

h1 {
  color: red;
  margin-bottom: 2em;
}

We can embed this within a <style>...</style> element. Then you put the entire <style> element under <head> of the HTML file you would like to style.

<style>
h1 {
  color: red;
  margin-bottom: 1.5em;
}
</style>

After you include the <style> in the HTML file, refresh the web page to see if the CSS style works.

3.1.4 HTML + CSS ( + JavaScript)

  • Try it:

    • HTML: the language for building web pages
    • CSS: the language for styling web pages
    • JavaScript: the language for programming web pages

3.2 Web Crawling

In the following demonstration, the text data scraped from the PTT forum is presented as it is without adjustment. Therefore, the language on PTT may strike some readers as profane, vulgar or even offensive.

When scraping data from PTT, it is important to manage the frequency of your requests. If the website detects an unusually high volume of automated traffic from a single source, it may temporarily or permanently block your IP address to prevent server strain.

To avoid being flagged as a bot, you should implement small delays between your requests and avoid running large-scale scraping tasks too rapidly. Ensuring your script mimics natural browsing behavior will help maintain your access to the site for future data collection.

library(tidyverse)
library(rvest)

3.2.1 Index Pages

In this tutorial, let’s assume that we like to scrape texts from PTT Forum. In particular, we will demonstrate how to scrape texts from the Gossiping board of PTT.

ptt.url <- "https://www.ptt.cc/bbs/Gossiping"
  • First, we create an session(). (This process is similar to opening a web browser, which can be used to navigate to a specific web page.)
## initialize the session
gossiping.session <- session(
  ptt.url,
  config = httr::add_headers(
    Cookie = "over18=1",
    `User-Agent` = "Mozilla/5.0"
  )
)

gossiping.session$response$url
[1] "https://www.ptt.cc/bbs/Gossiping/index.html"

If you use your browser to view PTT Gossiping page, you would see that you need to go through the age verification before you can enter the content page. So, our first job is to make sure that the session is able to pass through this age verification.

Check the current url of your gossiping.session:

gossiping.session$response$url
[1] "https://www.ptt.cc/bbs/Gossiping/index.html"

If the session’s current url is NOT https://www.ptt.cc/bbs/Gossiping/index.html, we may need to get pass the age verification page before you can access the bulletin posts.

We can extract the age verification form from the current page (form is also a defined HTML element)

gossiping.form <- gossiping.session %>%
  html_node("form") %>%
  html_form()

Then we automatically submit an yes to the age verification form in the earlier created session() and create another session.

gossiping.session <- session_submit(
  x = gossiping.session,
  form = gossiping.form,
  submit = "yes"
)

Now our html session, i.e., gossiping.session, should be on the front page of the Gossiping board.

Most browsers come with the functionality to inspect the page source code (i.e., HTML). This is very useful for web crawling. Before we scrape data from the webpage, we often need to inspect the structure of the web page first.

Most importantly, we need to know (a) which HTML elements, or (b) which particular attributes/values of the HTML elements we are interested in .

  • Next we need to find the most recent index page of the board
## Decide the number of index pages ----
page.latest <- gossiping.session %>%
  html_nodes("a") %>% ## extract all <a> elements
  html_attr("href") %>%  ## extract the attributes `href`
  str_subset("index[0-9]{2,}\\.html") %>% # find the `href` with the index number
  str_extract("[0-9]+") %>% ## extract the number
  as.numeric()

page.latest
[1] 39083
  • On the most recent index page, we need to extract the hyperlinks to the articles
## Retreive links

## Create the web address for the most recent index page
link <- str_c(ptt.url, "/index", page.latest, ".html")

## Visit the page and extract article links
links.article <- gossiping.session %>%
  session_jump_to(link) %>% ## move the session to the most recent index web page
  html_nodes("a") %>% ## extract article HTML elements <a>
  html_attr("href") %>% ## extract article <a> `href` attributes
  str_subset("[A-z]\\.[0-9]+\\.[A-z]\\.[A-z0-9]+\\.html") %>% ## extract article hyperlinks
  str_c("https://www.ptt.cc",.) ## add domain names for article links

## inspect article links of the page
links.article
 [1] "https://www.ptt.cc/bbs/Gossiping/M.1773183431.A.F21.html"
 [2] "https://www.ptt.cc/bbs/Gossiping/M.1773183635.A.CE6.html"
 [3] "https://www.ptt.cc/bbs/Gossiping/M.1773184047.A.83F.html"
 [4] "https://www.ptt.cc/bbs/Gossiping/M.1773184181.A.A60.html"
 [5] "https://www.ptt.cc/bbs/Gossiping/M.1773184187.A.76B.html"
 [6] "https://www.ptt.cc/bbs/Gossiping/M.1773184262.A.0D4.html"
 [7] "https://www.ptt.cc/bbs/Gossiping/M.1773184524.A.035.html"
 [8] "https://www.ptt.cc/bbs/Gossiping/M.1773184661.A.C3E.html"
 [9] "https://www.ptt.cc/bbs/Gossiping/M.1773184787.A.905.html"
[10] "https://www.ptt.cc/bbs/Gossiping/M.1773184940.A.EA3.html"
[11] "https://www.ptt.cc/bbs/Gossiping/M.1773185096.A.25B.html"
[12] "https://www.ptt.cc/bbs/Gossiping/M.1773185165.A.D75.html"
[13] "https://www.ptt.cc/bbs/Gossiping/M.1773185310.A.AB4.html"
[14] "https://www.ptt.cc/bbs/Gossiping/M.1773185378.A.AC9.html"
[15] "https://www.ptt.cc/bbs/Gossiping/M.1773185474.A.FC8.html"
[16] "https://www.ptt.cc/bbs/Gossiping/M.1773185746.A.DC5.html"
[17] "https://www.ptt.cc/bbs/Gossiping/M.1773185777.A.C61.html"
[18] "https://www.ptt.cc/bbs/Gossiping/M.1773185831.A.31A.html"
[19] "https://www.ptt.cc/bbs/Gossiping/M.1773185892.A.C6D.html"
[20] "https://www.ptt.cc/bbs/Gossiping/M.1773185944.A.DF1.html"

3.2.2 Extract Post Texts

  • Next step is to scrape texts from each article hyperlink. Let’s consider one link first.
## check first article link
article.url <- links.article[1]

## move current session to the first article link
temp.html <- gossiping.session %>% 
  session_jump_to(article.url)
  • Now the temp.html is a session on the article page. Because we are interested in the metadata and the contents of each article, now the question is: where are they in the HTML? We need to go back to the source page of the article HTML again:
HTML of an Article Page

Figure 3.3: HTML of an Article Page

  • Inspecting the article’s HTML reveals a clear structure for the data:

    • All relevant information is housed within the <div id="main-content"> container.
    • Inside this section, article metadata – such as the author or date – is stored specifically within <span> tags using the article-meta-value class (e.g., <span class="article-meta-value"> ... </span>)
    • The main body of the text is located directly within the <div id="main-content"> block, alongside these metadata elements.
  • Now we are ready to extract the metadata of the article.
## Extract article metadata
article.header <- temp.html %>%
  html_nodes("span.article-meta-value") %>% ## get <span> of a particular class
  html_text()
article.header
[1] "eric15234 (布卡)"                      
[2] "Gossiping"                             
[3] "[問卦] 人家要佈水雷 川普怎麼不出來說話"
[4] "Wed Mar 11 06:57:09 2026"

The metadata of each PTT article in fact includes four pieces of information: author, board name, title, post time. The above code retrieves directly the values of these metadata.

We can retrieve the tags of these metadata values as well:

temp.html %>%
  html_nodes("span.article-meta-tag") %>% ## get <span> of a particular class
  html_text()
[1] "作者" "看板" "標題" "時間"
  • From the article.header, we are able to extract the author, title, and time stamp of the article.
article.author <- article.header[1] %>% str_extract("^[A-z0-9_]+") ## athuor
article.title <- article.header[3] ## title
article.datetime <- article.header[4] ## time stamp

article.author
[1] "eric15234"
article.title
[1] "[問卦] 人家要佈水雷 川普怎麼不出來說話"
article.datetime
[1] "Wed Mar 11 06:57:09 2026"
  • Now we extract the main contents of the article
article.content <- temp.html %>%
        html_nodes( ## article body
          xpath = '//div[@id="main-content"]/node()[not(self::div|self::span[@class="f2"])]'
        ) %>%
        html_text(trim = TRUE) %>% ## extract texts
        str_c(collapse = "") ## combine all lines into one
article.content
[1] "這種股市大利空消息 川普怎麼不出來表態一下\n\n放什麼水雷 現在都什麼年代了 1950嗎？\n\n蘇維埃嗎？\n\n聽起來這麼可笑的事情 難道沒有防範的方法嗎？\n\n而且水雷是要人工下去設置還是有船可以把他拋下去\n\n一個海峽可以讓全球股市上上下下我也是醉了\n\n重點是Washington DC現在才晚上7點\n\n川普是在跟情婦吃燭光晚餐是不是\n\n\n--"

XPath (or XML Path Language) is a query language which is useful for addressing and extracting particular elements from XML/HTML documents. XPath allows you to exploit more features of the hierarchical tree that an HTML file represents in locating the relevant HTML elements. For more information, please see Munzert et al. (2014), Chapter 4.

The XPath '//div[@id="main-content"]/node()[not(self::div|self::span[@class="f2"])]' acts as a filter to extract the main text of an article while stripping away unwanted metadata or structural tags.

  • First, //div[@id="main-content"] locates the primary container holding the article.
  • The /node() command then selects every item directly inside that container, including raw text and HTML tags.
  • Finally, the filter [not(self::div|self::span[@class="f2"])] instructs the html_nodes() to ignore any <div> elements or <span> elements labeled with the class "f2".

In short, the XPath identifies the nodes under <div id = "main-content">, but excludes their sister nodes that are <div> or <span class="f2">.

These children elements <div> or <span class="f2"> of the <div id = "main-content"> include the push comments (推文) of the article, which are not the main content of the article.

  • Now we can combine all information related to the article into a data frame
article.table <- tibble(
      datetime = article.datetime,
      title = article.title,
      author = article.author,
      content = article.content,
      url = article.url
    )

article.table

3.2.3 Extract Post Comments

  • Next we extract the push comments at the end of the article
article.push <- temp.html %>% 
        html_nodes(xpath = "//div[@class = 'push']")

article.push
{xml_nodeset (6)}
[1] <div class="push">\n<span class="hl push-tag">推 </span><span class="f3 hl ...
[2] <div class="push">\n<span class="hl push-tag">推 </span><span class="f3 hl ...
[3] <div class="push">\n<span class="f1 hl push-tag">噓 </span><span class="f3 ...
[4] <div class="push">\n<span class="hl push-tag">推 </span><span class="f3 hl ...
[5] <div class="push">\n<span class="f1 hl push-tag">→ </span><span class="f3 ...
[6] <div class="push">\n<span class="f1 hl push-tag">→ </span><span class="f3 ...

  • We then extract relevant information from each push nodes article.push.

    • push types
    • push authors
    • push contents
## push tags
push.table.tag <- article.push %>% 
  html_nodes("span.push-tag") %>% 
  html_text(trim = TRUE) ## push types (like or dislike)
push.table.tag
[1] "推" "推" "噓" "推" "→"  "→" 
## push authors
push.table.author <- article.push %>% 
  html_nodes("span.push-userid") %>% 
  html_text(trim = TRUE) ## author
push.table.author
[1] "antigidu"   "q34355997"  "zombiechen" "ZeroArcher" "greensaru" 
[6] "achinyu"   
## push contents
push.table.content <- article.push %>% 
  html_nodes("span.push-content") %>%
  html_text(trim = TRUE)

push.table.content
[1] ": 川寶：怕三小  你們給我衝"      ": 川普有發文 敢布水雷就轟炸伊朗"
[3] ": 船被打爆了要用什麼佈雷？"      ": 自己去查好嗎？"               
[5] ": 不佈水雷，也轟炸伊朗啊，笑死"  ": 蛤..牠話能信？"               
## push time
push.table.datetime <- article.push %>% 
  html_nodes("span.push-ipdatetime") %>%
  html_text(trim = TRUE) ## push time stamp
push.table.datetime
[1] "42.73.147.68 03/11 06:58"   "118.232.105.32 03/11 06:59"
[3] "73.202.241.47 03/11 06:59"  "42.74.163.188 03/11 07:13" 
[5] "101.10.104.56 03/11 07:41"  "101.12.233.180 03/11 07:50"
  • Finally, we combine all into one Push data frame.
## Integrate comments into a DF
push.table <- tibble(
      tag = push.table.tag,
      author = push.table.author,
      content = push.table.content,
      datetime = push.table.datetime,
      url = article.url)

push.table

3.3 Functional Programming

It should now be clear that there are several routines that we need to do again and again if we want to collect text data in large amounts:

  • For each index page, we need to extract all the article hyperlinks of the page.
  • For each article hyperlink, we need to extract the article content, metadata, and the push comments.

So, it would be great if we can wrap these two routines into two functions.

3.3.2 extract_article_push_tables()

  • extract_article_push_tables(): This function takes an article link link as the argument and extracts the metadata, textual contents, and pushes of the article. It returns a list of two elements—article and push data frames.
extract_article_push_tables <- function(link){
  article.url <- link
  temp.html <- gossiping.session %>% session_jump_to(article.url) ## link to the www
  ## article header
  article.header <- temp.html %>%
    html_nodes("span.article-meta-value") %>% ## meta info regarding the article
    html_text()
  
  ## article meta
  article.author <- article.header[1] %>% str_extract("^[A-z0-9_]+") ## athuor
  article.title <- article.header[3] ## title
  article.datetime <- article.header[4] ## time stamp
  
  ## article content
  article.content <- temp.html %>%
    html_nodes( ## article body
      xpath = '//div[@id="main-content"]/node()[not(self::div|self::span[@class="f2"])]'
    ) %>%
    html_text(trim = TRUE) %>%
    str_c(collapse = "")
  
  ## Merge article table 
  article.table <- tibble(
        datetime = article.datetime,
        title = article.title,
        author = article.author,
        content = article.content,
        url = article.url
      )
      
  ## push nodes
  article.push <- temp.html %>% 
    html_nodes(xpath = "//div[@class = 'push']") ## extracting pushes
    ## NOTE: If CSS is used, div.push does a lazy match (extracting div.push.... also)
  
  ## push tags    
  push.table.tag <- article.push %>% 
    html_nodes("span.push-tag") %>% 
    html_text(trim = TRUE) ## push types (like or dislike)
  
  ## push author id
  push.table.author <- article.push %>% 
    html_nodes("span.push-userid") %>% 
    html_text(trim = TRUE) ## author
  
  ## push content    
  push.table.content <- article.push %>% 
    html_nodes("span.push-content") %>%
    html_text(trim = TRUE)
  
  ## push datetime      
  push.table.datetime <- article.push %>% 
    html_nodes("span.push-ipdatetime") %>%
    html_text(trim = TRUE) # push time stamp
  
  ## merge push table
  push.table <- tibble(
          tag = push.table.tag,
          author = push.table.author,
          content = push.table.content,
          datetime = push.table.datetime,
          url = article.url
        )
  
  ## return
  
  return(list(article.table = article.table, 
              push.table = push.table))
}## endfunc

For example, we can get the article and push tables from the first article link:

extract_article_push_tables(cur_art_links[1])
$article.table
# A tibble: 1 × 5
  datetime                 title                            author content url  
  <chr>                    <chr>                            <chr>  <chr>   <chr>
1 Wed Mar 11 06:57:09 2026 [問卦] 人家要佈水雷 川普怎麼不出來說話…… eric1… "這種股市大… http…

$push.table
# A tibble: 6 × 5
  tag   author     content                         datetime                url  
  <chr> <chr>      <chr>                           <chr>                   <chr>
1 推    antigidu   : 川寶：怕三小  你們給我衝      42.73.147.68 03/11 06:… http…
2 推    q34355997  : 川普有發文 敢布水雷就轟炸伊朗 118.232.105.32 03/11 0… http…
3 噓    zombiechen : 船被打爆了要用什麼佈雷？      73.202.241.47 03/11 06… http…
4 推    ZeroArcher : 自己去查好嗎？                42.74.163.188 03/11 07… http…
5 →     greensaru  : 不佈水雷，也轟炸伊朗啊，笑死  101.10.104.56 03/11 07… http…
6 →     achinyu    : 蛤..牠話能信？                101.12.233.180 03/11 0… http…

3.3.3 Streamline the Codes

Now we can simplify our codes quite a bit:

## Get index page
cur_index_page <- str_c(ptt.url, "/index", page.latest, ".html")

## Scrape all article.tables and push.tables from each article hyperlink
cur_index_page %>%  ## from this index page
  extract_art_links(session = gossiping.session) %>% ## extract all article hyperlinks
  map(extract_article_push_tables) -> ptt_data ## for each art hyperlin, extract data
## number of articles on this index page
length(ptt_data)
[1] 20
## Check the first contents of 1st hyperlink
ptt_data[[1]]$article.table
ptt_data[[1]]$push.table
  • Finally, the last thing we can do is to combine all article tables from each index page into one; and all push tables into one for later analysis.
## Merge all article.tables into one
article.table.all <- ptt_data %>% 
  map(function(x) x$article.table) %>% ## subset each article's text content df
  bind_rows

## Merge all push.tables into one
push.table.all <- ptt_data %>%
  map(function(x) x$push.table) %>%  ## subset each article's push df
  bind_rows

article.table.all
push.table.all

There is still one problem with the Push data frame. Right now it is still not very clear how we can match the pushes to the articles from which they were extracted. The only shared index is the url. It would be better if all the articles in the data frame have their own unique indices and in the Push data frame each push comment corresponds to a particular article index.

The following graph summarizes our work flowchart for PTT Gossipping Scraping:

3.4 Save Corpus

You can easily save your scraped texts in a CSV format.

## Save ------
write_csv(article.table, path = "PTT_GOSSIPPING_ARTICLE.csv")
write_csv(push.table, path = "PTT_GOSSIPPING_PUSH.csv")

3.5 Additional Resources

Collecting texts and digitizing them into machine-readable files is only the initial step for corpus construction. There are many other things that need to be considered to ensure the effectiveness and the sustainability of the corpus data. In particular, I would like to point you to a very useful resource, Developing Linguistic Corpora: A Guide to Good Practice, compiled by Martin Wynne. Other important issues in corpus creation include:

  1. Adding linguistic annotations to the corpus data (cf. Leech’s Chapter 2)
  2. Metadata representation of the documents (cf. Burnard’s Chapter 4)
  3. Spoken corpora (cf. Thompson’s Chapter 5)
  4. Technical parts for corpus creation (cf. Sinclair’s Appendix)

3.6 Final Remarks

  1. Please pay attention to the ethical aspects involved in the process of web crawling (esp. with personal private matters).
  2. If the website has their own API built specifically for one to gather data, use it instead of scraping.
  3. Always read the terms and conditions provided by the website regarding data gathering.
  4. Always be gentle with the data scraping (e.g., off-peak hours, spacing out the requests)
  5. Value the data you gather and treat the data with respect.

Exercise 3.1 Can you modify the R codes so that the script can automatically scrape more than one index page?

Exercise 3.2 Please utilize the code from Exercise 3.1 and collect all texts on PTT/Gossipings from 3 index pages. Please have the articles saved in PTT_GOSSIPING_ARTICLE.csv and the pushes saved in PTT_GOSSIPING_PUSH.csv under your working directory.

Also, at the end of your code, please also output in the Console the corpus size, including both the articles and the pushes. Please provide the total number of characters of all your PTT text data collected (Note: You DO NOT have to do the word segmentation yet. Please use the characters as the base unit for corpus size.)

Hint: nchar()

Your script may look something like:

## I define my own `scrapePTT()` function:
## ptt_url: specify the board to scrape texts from
## num_index_page: specify the number of index pages to be scraped
## return: list(article, push)

PTT_data <-scrapePTT(ptt_url = "https://www.ptt.cc/bbs/Gossiping", num_index_page = 3)

PTT_data$article %>% head
PTT_data$push %>% head
## corpus size

PTT_data$article$content %>% nchar %>% sum
[1] 16992

Exercise 3.3 Please choose a website (other than PTT) you are interested in and demonstrate how you can use R to retrieve textual data from the site. The final scraped text collection could be from only one static web page. The purpose of this exercise is to show that you know how to parse the HTML structure of the web page and retrieve the data you need from the website.

References

Munzert, S., Rubba, C., Meißner, P., & Nyhuis, D. (2014). Automated data collection with R: A practical guide to web scraping and text mining. John Wiley & Sons.