Use Tensorflow's Keras API to build a seq2seq deep learning network of LSTM cells for machine translation.

Generating Text with Deep Learning

It's time for some deep learning!

Deep learning models in Keras are built in layers, where each layer is a step in the model.

Our encoder requires two layer types from Keras: 
- An input layer, which defines a matrix to hold all the one-hot vectors that we'll feed to the model.
- An LSTM layer, with some output dimensionality.

We can import these layers as well as the model we need like so:
```py
from keras.layers import Input, LSTM
from keras.models import Model
```

Next, we set up the input layer, which requires some number of dimensions that we're providing. In this case, we know that we're passing in all the encoder tokens, but we don't necessarily know our batch size (how many ~~chocolate chip cookies~~ sentences we're feeding the model at a time). Fortunately, we can say `None` because the code is written to handle varying batch sizes, so we don’t need to specify that dimension.
```py
# the shape specifies the input matrix sizes
encoder_inputs = Input(shape=(None, num_encoder_tokens))
```

For the LSTM layer, we need to select the dimensionality (the size of the LSTM's hidden states, which helps determine how closely the model molds itself to the training data — something we can play around with) and whether to return the state (in this case we do):
```py
encoder_lstm = LSTM(100, return_state=True)
# we're using a dimensionality of 100
# so any LSTM output matrix will have 
# shape [batch_size, 100]
```

Remember, the only thing we want from the encoder is its final states. We can get these by linking our LSTM layer with our input layer:
```py
encoder_outputs, state_hidden, state_cell = encoder_lstm(encoder_inputs)
```

`encoder_outputs` isn't really important for us, so we can just discard it. However, the states, we'll save in a list:
```py
encoder_states = [state_hidden, state_cell]
```

There is a lot to take in here, but there's no need to memorize any of this — you got this.💪

Encoder Training Setup

Congrats! You've successfully built a machine translation program using deep learning with Tensorflow's Keras API.

While the translation output may not have been quite what you expected, this is just the beginning. There are many ways you can improve this program on your own device by using a larger data set, increasing the size of the model, and adding more epochs for training. 

You could also convert the one-hot vectors into word embeddings during training to improve the model. Using embeddings of words rather than one-hot vectors would help the model capture that semantically similar words might have semantically similar embeddings (helping the LSTM generalize).

You've learned quite a bit, even beyond the Keras implementation:

- seq2seq models are deep learning models that use recurrent neural networks like LSTMs to generate output.
- In machine translation, seq2seq networks encompass two main parts:
  - The encoder accepts language as input and outputs state vectors.
  - The decoder accepts the encoder's final state and outputs possible translations.
- Teacher forcing is a method we can use to train seq2seq decoders.
- We need to mark the beginning and end of target sentences so that the decoder knows what to expect at the beginning and end of sentences.
- One-hot vectors are a way to represent a given word in a set of words wherein we use 1 to indicate the current word and 0 to indicate every other word.
- Timesteps help us keep track of where we are in a sentence.
- We can adjust batch size, which determines how many sentences we give a model at a time.
- We can also tweak dimensionality and number of epochs, which can improve results with careful tuning.
- The Softmax function converts the output of the LSTMs into a probability distribution over words in our vocabulary.


Generating Text with Deep Learning Review

At this point we need to fill out the `1`s in each vector. We can loop over each English-Spanish pair in our training sample using the features dictionaries to add a `1` for the token in question. For example, the dog sentence (`["the", "dog", "licked", "me"]`) would be split into the following matrix of vectors:
```py
[
  [1, 0, 0, 0], # timestep 0 => "the"
  [0, 1, 0, 0], # timestep 1 => "dog"
  [0, 0, 1, 0], # timestep 2 => "licked"
  [0, 0, 0, 1], # timestep 3 => "me"
]
```
You'll notice the vectors have timesteps — we use these to track where in a given document (sentence) we are. 

To build out a three-dimensional NumPy matrix of one-hot vectors, we can assign a value of 1 for a given word at a given timestep in a given line:
```py
matrix_name[line, timestep, features_dict[token]] = 1.
```

Keras will fit — or train — the seq2seq model using these matrices of one-hot vectors:
- the encoder input data
- the decoder input data
- the decoder target data

Hang on a second, why build two matrices of decoder data? Aren't we just encoding and decoding?

The reason has to do with a technique known as _teacher forcing_ that most seq2seq models employ during training. Here's the idea: we have a Spanish input token from the previous timestep to help train the model for the current timestep's target token.

![All tokens of the target sentence preceding the current token are passed to the decoder at each timestep.](https://content.codecademy.com/programs/chatbots/seq2seq/teacher-forcing.gif)

Training Setup (part 2)

At long last, it's translation time. Inside the test function, we'll decode the sentence word by word using the output state that we retrieved from the encoder (which becomes our decoder's initial hidden state). We'll also update the decoder hidden state after each word so that we use previously decoded words to help decode new ones.

To tackle one word at a time, we need a `while` loop that will run until one of two things happens (we don't want the model generating words forever):
- The current token is `"<END>"`.
- The decoded Spanish sentence length hits the maximum target sentence length.

Inside the `while` loop, the decoder model can use the current target sequence (beginning with the `"<START>"` token) and the current state (initially passed to us from the encoder model) to get a bunch of possible next words and their corresponding probabilities. In Keras, it looks something like this:
```py
output_tokens, new_decoder_hidden_state, new_decoder_cell_state = 
    decoder_model.predict(
    [target_seq] + decoder_states_value)
```

Next, we can use NumPy's `.argmax()` method to determine the token (word) with the highest probability and add it to the decoded sentence:
```py
# slicing [0, -1, :] gives us a
# specific token vector within the
# 3d NumPy matrix
sampled_token_index = np.argmax(
    output_tokens[0, -1, :])

# The reverse features dictionary
# translates back from index to Spanish
sampled_token = reverse_target_features_dict[
    sampled_token_index]

decoded_sentence += " " + sampled_token
```

Our final step is to update a few values for the next word in the sequence:
```py
# Move to the next timestep 
# of the target sequence:
target_seq = np.zeros((1, 1, num_decoder_tokens))
target_seq[0, 0, sampled_token_index] = 1.

# Update the states with values from
# the most recent decoder prediction:
decoder_states_value = [
    new_decoder_hidden_state,
    new_decoder_cell_state]
```

And now we can test it all out!

You may recall that, because of platform constraints here, we're using very little data. As a result, we can only expect our model to translate a handful of sentences coherently. Luckily, you will have an opportunity to try this out on your own computer with far more data to see some much more impressive results.

Test Function (part 2)

Now our model is ready for testing! Yay! However, to generate some original output text, we need to redefine the seq2seq architecture in pieces. Wait, didn't we just define and train a model? 

Well, yes. But the model we used for training our network only works when we already know the target sequence. This time, we have no idea what the Spanish should be for the English we pass in! So we need a model that will decode step-by-step instead of using teacher forcing. To do this, we need a seq2seq network in individual pieces. 

To start, we'll build an encoder model with our encoder inputs and the placeholders for the encoder's output states:
```py
encoder_model = Model(encoder_inputs, encoder_states)
```

Next up, we need placeholders for the decoder's input states, which we can build as input layers and store together. Why? We don't know what we want to decode yet or what hidden state we’re going to end up with, so we need to do everything step-by-step. We need to pass the encoder's final hidden state to the decoder, sample a token, and get the updated hidden state back. Then we'll be able to (manually) pass the updated hidden state back into the network:
```py
latent_dim = 256
decoder_state_input_hidden = Input(shape=(latent_dim,))

decoder_state_input_cell = Input(shape=(latent_dim,))

decoder_states_inputs = [decoder_state_input_hidden, decoder_state_input_cell]
```
Using the decoder LSTM and decoder dense layer (with the activation function) that we trained earlier, we'll create new decoder states and outputs:
```py
decoder_outputs, state_hidden, state_cell = 
    decoder_lstm(decoder_inputs, 
    initial_state=decoder_states_inputs)

# Saving the new LSTM output states:
decoder_states = [state_hidden, state_cell]

# Below, we redefine the decoder output
# by passing it through the dense layer:
decoder_outputs = decoder_dense(decoder_outputs)
```

Finally, we can set up the decoder model. This is where we bring together:
- the decoder inputs (the decoder input layer)
- the decoder input states (the final states from the encoder)
- the decoder outputs (the NumPy matrix we get from the final output layer of the decoder)
- the decoder output states (the memory throughout the network from one word to the next)
```py
decoder_model = Model(
    [decoder_inputs] + decoder_states_inputs,
    [decoder_outputs] + decoder_states)
```


Setup for Testing

If you're feeling a bit nervous about building this all on your own, never fear. You don't need to start from scratch — there are a few neural network libraries at your disposal. In our case, we'll be using [TensorFlow](https://github.com/tensorflow/tensorflow) with the [Keras API](https://keras.io/) to build a pretty limited English-to-Spanish translator (we'll explain this later and you'll get an opportunity to improve it).

We can import Keras from Tensorflow like this:
```py
from tensorflow import keras
```

Also, do not worry about memorizing anything we cover here. The purpose of this lesson is for you to make sense of what each part of the code does and how you can modify it to suit your own needs. In fact, the code we'll be using is mostly derived from [Keras's own tutorial on the seq2seq model](https://keras.io/examples/nlp/lstm_seq2seq/).

First things first: preprocessing the text data. Noise removal depends on your use case — do you care about casing or punctuation? For many tasks they are probably not important enough to justify the additional processing. This might be the time to make changes.

We'll need the following for our Keras implementation:
- vocabulary sets for both our input (English) and target (Spanish) data
- the total number of unique word tokens we have for each set
- the maximum sentence length we're using for each language

We also need to mark the start and end of each document (sentence) in the target samples so that the model recognizes where to begin and end its text generation (no book-long sentences for us!). One way to do this is adding `"<START>"` at the beginning and `"<END>"` at the end of each target document (in our case, this will be our Spanish sentences). For example, `"Estoy feliz."` becomes `"<START> Estoy feliz. <END>"`.

Before you dig into the instructions, read through the existing code in **script.py** and try to make sense of each line.


Preprocessing for seq2seq

Finally, we can get to testing our model! To do this, we need to build a function that:
- accepts a NumPy matrix representing the test English sentence input
- uses the encoder and decoder we've created to generate Spanish output

Inside the test function, we'll run our new English sentence through the encoder model. The `.predict()` method takes in new input (as a NumPy matrix) and gives us output states that we can pass on to the decoder:
```py
# test_input is a NumPy matrix
# representing an English sentence
states = encoder.predict(test_input)
```

Next, we'll build an empty NumPy array for our Spanish translation, giving it three dimensions:
```py
# batch size: 1
# number of tokens to start with: 1
# number of tokens in our target vocabulary
target_sequence = np.zeros((1, 1, num_decoder_tokens))
```

Luckily, we already know the first value in our Spanish sentence — `"<Start>"`! So we can give `"<Start>"` a value of `1` at the first timestep:
```py
target_sequence[0, 0, target_features_dict['<START>']] = 1.
```
Before we get decoding, we'll need a string where we can add our translation to, word by word:
```py
decoded_sentence = ''
```
This is the variable that we will ultimately return from the function.

The Test Function

For each sentence, Keras expects a NumPy matrix containing _one-hot vectors_ for each token. What's a one-hot vector? In a one-hot vector, every token in our set is represented by a `0` except for the current token which is represented by a `1`. For example given the vocabulary `["the", "dog", "licked", "me"]`, a one-hot vector for "dog" would look like `[0, 1, 0, 0]`. 

In order to vectorize our data and later translate it from vectors, it's helpful to have a features dictionary (and a reverse features dictionary) to easily translate between all the 1s and 0s and actual words. We'll build out the following:
- a features dictionary for English
- a features dictionary for Spanish
- a reverse features dictionary for English (where the keys and values are swapped)
- a reverse features dictionary for Spanish

Once we have all of our features dictionaries set up, it's time to vectorize the data! We're going to need vectors to input into our encoder and decoder, as well as a vector of target data we can use to train the decoder. 

Because each matrix is almost all zeros, we'll use `numpy.zeros()` from the NumPy library to build them out.

```py
import numpy as np

encoder_input_data = np.zeros(
    (len(input_docs), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
```
Let's break this down: 

We defined a NumPy matrix of zeros called `encoder_input_data` with two arguments:
- the shape of the matrix — in our case the number of documents (or sentences) by the maximum token sequence length (the longest sentence we want to see) by the number of unique tokens (or words)
- the data type we want — in our case NumPy's `float32`, which can speed up our processing a bit


Training Setup (part 1)

The decoder looks a lot like the encoder (phew!), with an input layer and an LSTM layer that we use together: 
```py
decoder_inputs = Input(shape=(None, num_decoder_tokens))
decoder_lstm = LSTM(100, return_sequences=True, return_state=True)
# This time we care about full return sequences
```
However, with our decoder, we pass in the state data from the encoder, along with the decoder inputs. This time, we'll keep the output instead of the states:
```py
# The two states will be discarded for now
decoder_outputs, decoder_state_hidden, decoder_state_cell = 
    decoder_lstm(decoder_inputs, initial_state=encoder_states)
```
We also need to run the output through a final activation layer, using the Softmax function, that will give us the probability distribution — where all probabilities sum to one — for each token. The final layer also transforms our LSTM output from a dimensionality of whatever we gave it (in our case, 10) to the number of unique words within the hidden layer's vocabulary (i.e., the number of unique target tokens, which is definitely more than 10!).

```py
decoder_dense = Dense(num_decoder_tokens, activation='softmax')

decoder_outputs = decoder_dense(decoder_outputs)
```

Keras's implementation could work with several layer types, but `Dense` is the least complex, so we'll go with that. We also need to modify our import statement to include it before running the code:
```py
from keras.layers import Input, LSTM, Dense
```


Decoder Training Setup

Alright! Let's get model-building!

First, we define the seq2seq model using the `Model()` function we imported from Keras. To make it a seq2seq model, we feed it the encoder and decoder inputs, as well as the decoder output:
```py
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
```
Finally, our model is ready to train. First, we compile everything. Keras models demand two arguments to compile: 
- An optimizer (we're using RMSprop, which is a fancy version of the widely-used [gradient descent](https://www.codecademy.com/courses/machine-learning/lessons/linear-regression/exercises/gradient-descent-b)) to help minimize our error rate (how bad the model is at guessing the true next word given the previous words in a sentence).
- A loss function (we're using the logarithm-based cross-entropy function) to determine the error rate.

Because we care about accuracy, we're adding that into the metrics to pay attention to while training. Here's what the compiling code looks like:

```py
model.compile(optimizer='rmsprop', 
              loss='categorical_crossentropy',
              metrics=['accuracy'])
```
Next we need to fit the compiled model. To do this, we give the `.fit()` method the encoder and decoder input data (what we pass into the model), the decoder target data (what we expect the model to return given the data we passed in), and some numbers we can adjust as needed: 
- batch size (smaller batch sizes mean more time, and for some problems, smaller batch sizes will be better, while for other problems, larger batch sizes are better)
- the number of epochs or cycles of training (more epochs mean a model that is more trained on the dataset, and that the process will take more time)
- validation split (what percentage of the data should be set aside for validating — and determining when to stop training your model — rather than training)

Keras will take it from here to get you a (hopefully) nicely trained seq2seq model:
```py
model.fit([encoder_input_data, decoder_input_data], 
          decoder_target_data,
          batch_size=10,
          epochs=100,
          validation_split=0.2)
```


Build and Train seq2seq

LSTMs are pretty extraordinary, but they're only the tip of the iceberg when it comes to actually setting up and running a neural language model for text generation. In fact, an LSTM is usually just a single component in a larger network. 

One of the most common neural models used for text generation is the sequence-to-sequence model, commonly referred to as _seq2seq_ (pronounced "seek-to-seek"). A type of encoder-decoder model, seq2seq uses recurrent neural networks (RNNs) like LSTM in order to generate output, token by token or character by character.

So, where does seq2seq show up? 
- Machine translation software like Google Translate
- Text summary generation
- Chatbots
- Named Entity Recognition (NER)
- Speech recognition

seq2seq networks have two parts:

- An _encoder_ that accepts language (or audio or video) input. The output matrix of the encoder is discarded, but its state is preserved as a vector.

- A _decoder_ that takes the encoder's final state (or memory) as its initial state. We use a technique called "teacher forcing" to train the decoder to predict the following text (characters or words) in a target sequence given the previous text.


"Knowledge is power" is translated from Hindi to English using a seq2seq network. The encoder moves from word to word and then passes the state to the decoder at each decoder timestep. All input tokens are considered at each timestep within the decoding process.