In this article, I’m going to show how to implement GRU and LSTM units and how to build deeper RNNs using TensorFlow. I will start by explaining a little theory about GRUs, LSTMs and Deep RNNs, and then explain the code snippet by snippet. This article is meant to be a continuation to my previous article about RNNs:
I’ll suggest you to read that article first. But if you already know about simple RNNs and just want to learn about the content of this article, that’s fine too.
The simple type of RNN that was used in my previous article…
In this article I’m going to explain first a little theory about Recurrent Neural Networks (RNNs) for those who are new to them, then I’ll show the implementation that I did using TensorFlow. We’re going to see the code snippet by snippet along with the explanations and the output that it produced.
The dataset used is A Million News Headlines.
Let’s first recall what feed-forward neural networks are: they are functions that map the input x to an output ŷ which is an estimate for the true label y. They can be represented like this:
Before seeing “how” to use C code from Python let’s see first “why” one may want to do this. If you are reading this article right now you probably want to do one or more of these 3 things:
Sometimes when writing code in a Jupyter notebook, you may be in a hurry and don’t want to spend time on arranging your code, using proper indentation, and stuff like that. So, wouldn’t be nice if there would be a tool to solve this issue? Well… there is. There is a Jupyter extension that does exactly that: Code Prettify Jupyter Extension.
Let’s see below how we can install and use it.
Firstly, we need to install the
jupyter_contrib_nbextensions python package. This package will then be used to install the extensions and to activate/deactivate a certain extension.
pip install jupyter_contrib_nbextensions
By default, a Jupyter Notebook saves your work every 2 minutes, and if you want to change this time interval you can do so by using the
%autosave n magic command; where n is the number of seconds, and if n=0 this means no autosaving.
%autosave magic command produces effects only inside the notebook in which this command is run, and only in that particular session. If you shut down the notebook and re-open it you have to run the
%autosave command again, or if you open another notebook, also you have to run the command again.
To understand better what we’re going to do next, you can read my previous article about logistic regression:
So, what’s our plan for implementing Logistic Regression with PyTorch?
Let’s first think of the underlying math that we want to use.
There are many ways to define a loss function and then find the optimal parameters for it, among them, here we will implement in our
LogisticRegression class the following 3 ways for learning the parameters:
If you are here for a quick solution that just works, then here it is in just 5 lines of code:
model = tf.keras.models.Sequential([ tf.keras.layers.Dense(1, activation='sigmoid')])model.compile(loss='bce')model.fit(x_train, y_train, epochs=100)
Now, if you’re still with me it means that you don’t want just to copy + paste 5 lines of code, but to see how you can actually implement this method yourself from scratch.
TensorFlow is a rich library; it has many APIs that you can use. Among them is the Keras API which can be used to build a logistic regression model very quickly, as you can see…
By default, when the output of a code cell in a Jupyter notebook gets too long, the content remains in its initial position and you have to manually scroll down to see the latest output. And this can be very annoying.
For example, if you train a Machine Learning model from a Jupyter notebook and you want some info to be printed after each batch or epoch, you would like to have the output automatically scrolling so that you can see the message about the last printed epochs, but instead, nothing happens, you have to scroll down manually.
Disclosure: below you’ll find affiliate links to these two books
C and C++ are, despite their age, some of the most used programming languages across the globe and are among the first ones taught in schools and universities. If you are taking a programming course right now, there are big chances you are struggling to learn one of these 2 languages.
I first started studying C++ (actually, it was mostly C, with just a few elements of C++) when I went into high school. Back then, I found this programming language quite confusing and difficult. After 4 years of studying…