In this course you will learn how to perform Natural Language Parsing with Regular Expressions to discover insights in text data!

Natural Language Parsing with Regular Expressions

Another popular type of chunking is _VP-chunking_, or _**verb phrase chunking**_. A _verb phrase_ is a phrase that contains a verb and its complements, objects, or modifiers.

Verb phrases can take a variety of structures, and here you will consider two. The first structure begins with a verb `VB` of any tense, followed by a noun phrase, and ends with an optional adverb `RB` of any form. The second structure switches the order of the verb and the noun phrase, but also ends with an optional adverb.

Both structures are considered because verb phrases of each form are essentially the same in meaning. For example, consider the part-of-speech tagged verb phrases given below:

* `(('said', 'VBD'), ('the', 'DT'), ('cowardly', 'JJ'), ('lion', 'NN'))`
* `('the', 'DT'), ('cowardly', 'JJ'), ('lion', 'NN')), (('said', 'VBD'), `

The chunk grammar to find the first form of verb phrase is given below:

```py
chunk_grammar = "VP: {<VB.*><DT>?<JJ>*<NN><RB.?>?}"
```
* `VP` is the user-defined name of the chunk you are searching for. In this case `VP` stands for verb phrase
* `<VB.*>` matches any verb using the `.` as a wildcard and the `*` quantifier to match `0` or more occurrences of any character. This ensures matching verbs of any tense (ex. `VB` for present tense, `VBD` for past tense, or `VBN` for past participle)
* `<DT>?<JJ>*<NN>` matches any noun phrase
* `<RB.?>` matches any adverb using the `.` as a wildcard and the _optional quantifier_ to match `0` or `1` occurrence of any character. This ensures matching any form of adverb (regular `RB`, comparative `RBR`, or superlative `RBS`)
* `?` is an _optional quantifier_, matching either `0` or `1` adverbs

The chunk grammar for the second form of verb phrase is given below:

```py
chunk_grammar = "VP: {<DT>?<JJ>*<NN><VB.*><RB.?>?}"
```

Just like with NP-chunks, you can find all the VP-chunks in a text and perform a frequency analysis to identify important, recurring verb phrases. These verb phrases can give insight into what kind of action different characters take or how the actions that characters take are described by the author.

Once again, this is the part of the analysis where you get to be creative and use your own knowledge about the text you are working with to find interesting insights!

Chunking Verb Phrases

And there you go! Now you have the toolkit to dig into any piece of text data and perform natural language parsing with regular expressions. What insights will you gain, or what bias may you uncover? Let's review what you have learned:
* The `re` module's `.compile()` and `.match()` methods allow you to enter any regex pattern and look for a single match at the beginning of a piece of text
* The `re` module's `.search()` method lets you find a single match to a regex pattern anywhere in a string, while the `.findall()` method finds all the matches of a regex pattern in a string
* _Part-of-speech tagging_ identifies and labels the part of speech of words in a sentence, and can be performed in `nltk` using the `pos_tag()` function
* _Chunking_ groups together patterns of words by their part-of-speech tag. Chunking can be performed in `nltk` by defining a piece of chunk grammar using regular expression syntax and calling a `RegexpParser`'s `.parse()` method on a word tokenized sentence
* _NP-chunking_ chunks together an optional determiner `DT`, any number of adjectives `JJ`, and a noun `NN` to form a noun phrase. The frequency of different NP-chunks can identify important topics in a text or demonstrate how an author describes different subjects
* _VP-chunking_ chunks together a verb `VB`, a noun phrase, and an optional adverb `RB` to form a verb phrase. The frequency of different VP-chunks can give insight into what kind of action different subjects take or how the actions that different subjects take are described by an author, potentially indicating bias
* _Chunk filtering_ provides an alternative means of chunking by specifying what parts of speech you do not want in a chunk and removing them

Review

Before you dive into more complex syntax parsing, you'll begin with basic regular expressions in Python using the <a href="https://docs.python.org/3/library/re.html" target="_blank" rel="noopener noreferrer">_`re` module_</a> as a regex refresher.

The first method you will explore is _**`.compile()`**_. This method takes a regular expression pattern as an argument and compiles the pattern into a regular expression object, which you can later use to find matching text. The regular expression object below will exactly match `4` upper or lower case characters.

```py
regular_expression_object = re.compile("[A-Za-z]{4}")
```

Regular expression objects have a _**`.match()`**_ method that takes a string of text as an argument and looks for a _single_ match to the regular expression that _starts at the beginning_ of the string. To see if your regular expression matches the string `"Toto"` you can do the following:

```py
result = regular_expression_object.match("Toto")
```

If `.match()` finds a match that starts at the beginning of the string, it will return a match object. The match object lets you know what piece of text the regular expression matched, and at what index the match begins and ends. If there is no match, `.match()` will return `None`.

With the match object stored in `result`, you can access the matched text by calling `result.group(0)`. If you use a regex containing capture groups, you can access these groups by calling `.group()` with the appropriately numbered capture group as an argument.

Instead of compiling the regular expression first and then looking for a match in separate lines of code, you can simplify your match to one line:

```py
result = re.match("[A-Za-z]{4}","Toto")
```

With this syntax, `re`'s `.match()` method takes a regular expression pattern as the first argument and a string as the second argument.


Compiling and Matching

While it is useful to match and search for patterns of individual characters in a text, you can often find more meaning by analyzing text on a word-by-word basis, focusing on the part of speech of each word in a sentence. This process of identifying and labeling the part of speech of words is known as _**part-of-speech tagging**_!

It may have been a while since you've been in English class, so let's review <a href="https://content.codecademy.com/courses/nlp-regex-parsing/nlp_regex_parsing_part_of_speech_table.pdf" rel="noopener noreferrer">the nine parts of speech</a> with an example:

`Wow! Ramona and her class are happily studying the new textbook she has on NLP.`

* **Noun:** the name of a person (`Ramona`,`class`), place, thing (`textbook`), or idea (`NLP`)
* **Pronoun:** a word used in place of a noun (`her`,`she`)
* **Determiner:** a word that introduces, or "determines", a noun (`the`)
* **Verb:** expresses action (`studying`) or being (`are`,`has`)
* **Adjective:** modifies or describes a noun or pronoun (`new`)
* **Adverb:** modifies or describes a verb, an adjective, or another adverb (`happily`)
* **Preposition:** a word placed before a noun or pronoun to form a phrase modifying another word in the sentence (`on`)
* **Conjunction:** a word that joins words, phrases, or clauses (`and`)
* **Interjection:** a word used to express emotion (`Wow`)

You can automate the part-of-speech tagging process with `nltk`'s `pos_tag()` function! The function takes one argument, a list of words in the order they appear in a sentence, and returns a list of tuples, where the first entry in the tuple is a word and the second is the part-of-speech tag.

Given the sentence split into a list of words below:

```py
word_sentence = ['do', 'you', 'suppose', 'oz', 'could', 'give', 'me', 'a', 'heart', '?']
```

you can tag the parts of speech as follows:

```py
part_of_speech_tagged_sentence = pos_tag(word_sentence)
```

The call to `pos_tag()` will return the following:

```py
[('do', 'VB'), ('you', 'PRP'), ('suppose', 'VB'), ('oz', 'NNS'), ('could', 'MD'), ('give', 'VB'), ('me', 'PRP'), ('a', 'DT'), ('heart', 'NN'), ('?', '.')]
```

Abbreviations are given instead of the full part of speech name. Some common abbreviations include: `NN` for nouns, `VB` for verbs, `RB` for adverbs, `JJ` for adjectives, and `DT` for determiners. A <a href="https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html" target="_blank" rel="noopener noreferrer">complete list of part-of-speech tags and their abbreviations can be found here</a>.


Part-of-Speech Tagging

While you are able to chunk any sequence of parts of speech that you like, there are certain types of chunking that are linguistically helpful for determining meaning and bias in a piece of text. One such type of chunking is _NP-chunking_, or _**noun phrase chunking**_. A _noun phrase_ is a phrase that contains a noun and operates, as a unit, as a noun.

A popular form of noun phrase begins with a _determiner_ `DT`, which specifies the noun being referenced, followed by any number of _adjectives_ `JJ`, which describe the noun, and ends with a noun `NN`.

Consider the part-of-speech tagged sentence below:

```py
[('we', 'PRP'), ('are', 'VBP'), ('so', 'RB'), ('grateful', 'JJ'), ('to', 'TO'), ('you', 'PRP'), ('for', 'IN'), ('having', 'VBG'), ('killed', 'VBN'), ('the', 'DT'), ('wicked', 'JJ'), ('witch', 'NN'), ('of', 'IN'), ('the', 'DT'), ('east', 'NN'), (',', ','), ('and', 'CC'), ('for', 'IN'), ('setting', 'VBG'), ('our', 'PRP$'), ('people', 'NNS'), ('free', 'VBP'), ('from', 'IN'), ('bondage', 'NN'), ('.', '.')]
```

Can you spot the three noun phrases of the form described above? They are:
* `(('the', 'DT'), ('wicked', 'JJ'), ('witch', 'NN'))`
* `(('the', 'DT'), ('east', 'NN'))`
* `(('bondage', 'NN'))`

With the help of a regular expression defined chunk grammar, you can easily find all the _non-overlapping_ noun phrases in a piece of text! Just like in normal regular expressions, you can use quantifiers to indicate how many of each part of speech you want to match.

The chunk grammar for a noun phrase can be written as follows:

```py
chunk_grammar = "NP: {<DT>?<JJ>*<NN>}"
```
* `NP` is the user-defined name of the chunk you are searching for. In this case `NP` stands for noun phrase
* `<DT>` matches any determiner
* `?` is an _optional quantifier_, matching either `0` or `1` determiners
* `<JJ>` matches any adjective
* `*` is the _Kleene star_ quantifier, matching `0` or more occurrences of an adjective
* `<NN>` matches any noun, singular or plural

By finding all the NP-chunks in a text, you can perform a frequency analysis and identify important, recurring noun phrases. You can also use these NP-chunks as pseudo-topics and tag articles and documents by their highest count NP-chunks! Or perhaps your analysis has you looking at the adjective choices an author makes for different nouns. 

It is ultimately up to you, with your knowledge of the text you are working with, to interpret the meaning and use-case of the NP-chunks and their frequency of occurrence.


Chunking Noun Phrases

Another option you have to find chunks in your text is _chunk filtering_. _**Chunk filtering**_ lets you define what parts of speech you _do not want_ in a chunk and remove them.

A popular method for performing chunk filtering is to chunk an entire sentence together and then indicate which parts of speech are to be filtered out.  If the filtered parts of speech are in the middle of a chunk, it will split the chunk into two separate chunks! The chunk grammar you can use to perform chunk filtering is given below:

```py
chunk_grammar = """NP: {<.*>+}
                       }<VB.?|IN>+{"""
```
* `NP` is the user-defined name of the chunk you are searching for. In this case `NP` stands for noun phrase
* The brackets `{}` indicate what parts of speech you are chunking. `<.*>+` matches every part of speech in the sentence
* The inverted brackets `}{` indicate which parts of speech you want to filter from the chunk. `<VB.?|IN>+` will filter out any verbs or prepositions

Chunk filtering provides an alternate way for you to search through a text and find the chunks of information useful for your analysis!


Chunk Filtering

Discovering new code words in declassified CIA documents may seem like a mission for a foreign intelligence service, and detecting gender biases in the Harry Potter novels a task for a literature professor. Yet by utilizing _**natural language parsing with regular expressions**_, the power to perform such analyses is in your own hands!

While you may not put much explicit thought into the structure of your sentences as you write, the syntax choices you make are critical in ensuring your writing has meaning. Analyzing such sentence structure as well as word choice can not only provide insights into the connotation of a piece text, but can also <a href="https://medium.com/agatha-codes/a-bossy-sort-of-voice-3c3a18de3093" target="_blank" rel="noopener noreferrer">highlight the biases of its author</a> or <a href="https://twitter.com/hexadecim8/status/1068215227274137605" target="_blank" rel="noopener noreferrer">uncover additional insights that even a deep, rigorous reading of the text might not reveal</a>.

By using <a href="https://docs.python.org/3/library/re.html" target="_blank" rel="noopener noreferrer">_Python's regular expression module`re`_</a> and the <a href="https://www.nltk.org/" target="_blank" rel="noopener noreferrer">_Natural Language Toolkit_</a>, known as NLTK, you can find keywords of interest, discover where and how often they are used, and discern the parts-of-speech patterns in which they appear to understand the sometimes hidden meaning in a piece of writing. Let's get started!


Introduction

You can make your regular expression matches even more dynamic with the help of the `.search()` method. Unlike `.match()` which will only find matches at the start of a string, _**`.search()`**_ will look left to right through an entire piece of text and return a match object for the first match to the regular expression given. If no match is found, `.search()` will return `None`. For example, to search for a sequence of `8` word characters in the string `Are you a Munchkin?`:

```py
result = re.search("\w{8}","Are you a Munchkin?")
```

Using `.search()` on the string above will find a match of `"Munchkin"`, while using `.match()` on the same string would return `None`!

So far you have used methods that only return one piece of matching text. What if you want to find all the occurrences of a word or keyword in a piece of text to determine a frequency count? Step in the `.findall()` method!

Given a regular expression as its first argument and a string as its second argument, _**`.findall()`**_ will return a list of all _non-overlapping_ matches of the regular expression in the string. Consider the below piece of text:

```py
text = "Everything is green here, while in the country of the Munchkins blue was the favorite color. But the people do not seem to be as friendly as the Munchkins, and I'm afraid we shall be unable to find a place to pass the night."
```

To find all _non-overlapping_ sequences of `8` word characters in the sentence you can do the following:

```py
list_of_matches = re.findall("\w{8}",text)
```

`.findall()` will thus return the list `['Everythi', 'Munchkin', 'favorite', 'friendly', 'Munchkin']`.


Searching and Finding

You have made it to the juicy stuff! Given your part-of-speech tagged text, you can now use regular expressions to find patterns in sentence structure that give insight into the meaning of a text. This technique of grouping words by their part-of-speech tag is called _**chunking**_.

With chunking in `nltk`, you can define a pattern of parts-of-speech tags using a modified notation of regular expressions. You can then find non-overlapping matches, or _chunks_ of words, in the part-of-speech tagged sentences of a text.

The regular expression you build to find chunks is called _chunk grammar_. A piece of chunk grammar can be written as follows:

```py
chunk_grammar = "AN: {<JJ><NN>}"
```
* `AN` is a user-defined name for the kind of chunk you are searching for. You can use whatever name makes sense given your chunk grammar. In this case `AN` stands for adjective-noun
* A pair of curly braces `{}` surround the actual chunk grammar
* `<JJ>` operates similarly to a regex character class, matching any adjective
* `<NN>` matches any noun, singular or plural

The chunk grammar above will thus match any adjective that is followed by a noun.

To use the chunk grammar defined, you must create a `nltk` `RegexpParser` object and give it a piece of chunk grammar as an argument.

```py
chunk_parser = RegexpParser(chunk_grammar)
```

You can then use the `RegexpParser` object's `.parse()` method, which takes a list of part-of-speech tagged words as an argument, and identifies where such chunks occur in the sentence!

Consider the part-of-speech tagged sentence below:

```py
pos_tagged_sentence = [('where', 'WRB'), ('is', 'VBZ'), ('the', 'DT'), ('emerald', 'JJ'), ('city', 'NN'), ('?', '.')]
```

You can chunk the sentence to find any adjectives followed by a noun with the following:

```py
chunked = chunk_parser.parse(pos_tagged_sentence)
```