Recurrent Neural Netword(RNN) Using Tensorflow

Pre-processing

Datasets

MNIST database of handwritten digits. Click here
Input data: Image shape(28*28)
Output label: 0~9

Parameters

  • Data_Size
    • Input_dimension: Dimension of each image
    • Output_dimension: Dimension of predicted label
    • Classes: The number of different outputs
  • Model_Parameter
    • Training_iter: The number of iterations for training
    • Batch_size: The length of inputeach epoch

Requirement

  • Python 2.7
  • Tensorflow 0.12.1

Model(RNN + LSTM)

We use a Recurrent Neural Network with LSTM Cell to implement this model.

  • LSTM (Long Short Term Memory):
LSTM_MODEL
LSTM_MODEL
LSTM
LSTM

LSTM Composed of three gates which called INPUT_GATE, FORGET_GATE and OUTPUT_GATE.

More information about how to implement LSTM Model is here.

  • Initialize Step

First we should initialize the placeholder and weights of our neural network.
placeholder: just like the x of the function:

weights: the weight for converting input data to output label.

1
2
3
4
5
6
7
8
9
x = tf.placeholder("float", [None, n_steps, n_input])
y = tf.placeholder("float", [None, n_classes])
weights = {
    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
}
biases = {
    'out': tf.Variable(tf.random_normal([n_classes]))
}
  • Training Step

First we define a RNN_Model function.
Using linear relationship to combine the output parameters.

1
2
3
4
def RNN_Model(x, weights, biases):
    lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden, forget_bias = 1.0)
    output, states = tf.nn.rnn(lstm_cell, x, dtype = tf.float32)
    return tf.matmul(output[-1], weights['out']) + biases['out']

Second we have to define the loss function and optmizer of our model.
loss fuction: softmax_cross_entropy
optimizer: 

1
2
3
4
prediction = RNN_Model(x, weights, biases)
result = tf.nn.softmax(prediction)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = prediction, labels = y))
optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(loss)

Third in order to evaluate the efficiency of this model, we define the function to calculate accuracy.

1
2
correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Finally we can start training after all the initialization.

  • We can use session to run our tensorflow function.
1
2
3
4
5
6
7
8
9
10
11
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    while "./epoch" < training_iters
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        batch_x = batch_x.reshape((batch_size, n_steps, n_input))
        sess.run(optimizer, feed_dict = {x: batch_x, y: batch_y}
        if "./batch_size":
            acc = sess.run(accuracy, feed_dict = {x: batch_x, y: batch_y})
            los = sess.run(loss, feed_dict = {x: batch_x, y: batch_y})

Tips: “./“ represent the parameters defined by user own.

  • Testing Step

After training we get a weights in the tensorflow session which can be used to predict our test data.

First generate the testing dataset from mnist generator.

1
2
3
test_data = mnist.test.images[:"./test_length"].reshape(-1, n_steps, n_input)
res = sess.run(result, feed_dict = {x: test_data})
predict_label = sess.run(tf.argmax(res, 1))

Finally because tensorflow mnist test dataset have its own ground-truth. So we can estimate if our “predict_label” is correct. 

1
2
test_label = mnist.test.labels[:"./test_lenght"]
sess.run(accuracy, feed_dict = {x: predict_label, y: test_label})

Usage

  1. Install tensorflow.

    • If we will run our model on GPU we have to install cuda and cuDNN.
      1
      pip install tensorflow(-gpu)==0.12.1

  2. Import tensorflow package.

  3. Import tensorflow mnist dataset and read the dataset as a generator.
  4. Run our model
    1
    python "./model_name".py

Reference

Challenges

In this experiment we use a simple RNN(LSTM) model to predict the handwritten digits which also catch a good consequence in CNN.
RNN model is good for using in NLP processing. But how to explore the most useful determines whether our model can get an excellent result or not.