Sign in

Passionate about Data Science, AI, Programming & Math | Owner of ∇²

In this article, I’m going to show how to implement GRU and LSTM units and how to build deeper RNNs using TensorFlow. I will start by explaining a little theory about GRUs, LSTMs and Deep RNNs, and then explain the code snippet by snippet. This article is meant to be a continuation to my previous article about RNNs:

I’ll suggest you to read that article first. But if you already know about simple RNNs and just want to learn about the content of this article, that’s fine too.

A little theory

The simple type of RNN that was used in my previous article…

And using it to build a language model for news headlines

In this article I’m going to explain first a little theory about Recurrent Neural Networks (RNNs) for those who are new to them, then I’ll show the implementation that I did using TensorFlow. We’re going to see the code snippet by snippet along with the explanations and the output that it produced.

The dataset used is A Million News Headlines.

A little theory about RNNs

Let’s first recall what feed-forward neural networks are: they are functions that map the input x to an output ŷ which is an estimate for the true label y. They can be represented like this:

… and should we do this?

Background image by Johnson Martin from Pixabay

Before seeing “how” to use C code from Python let’s see first “why” one may want to do this. If you are reading this article right now you probably want to do one or more of these 3 things:

  1. Reusing existing C code from a Python program
  2. Speed up your Python program
  3. Do some low-level stuff that cannot be done directly in Python

A shortcut…

If all you want is just to speed up your Python program, then there is actually an easier way rather than writing certain parts of your program in C. You can just use PyPy instead of Python…

Use the Code Prettify Jupyter Extension

Background image by Free-Photos from Pixabay

Sometimes when writing code in a Jupyter notebook, you may be in a hurry and don’t want to spend time on arranging your code, using proper indentation, and stuff like that. So, wouldn’t be nice if there would be a tool to solve this issue? Well… there is. There is a Jupyter extension that does exactly that: Code Prettify Jupyter Extension.

Let’s see below how we can install and use it.


Firstly, we need to install the jupyter_contrib_nbextensions python package. This package will then be used to install the extensions and to activate/deactivate a certain extension.

pip install jupyter_contrib_nbextensions


A brief introduction to Generative Adversarial Networks

Image source:

Why should we care about Generative Adversarial Networks (GANs for short) in the first place?

Well… take a look at the below images.

You can use this Jupyter extension

Background image by Free-Photos from Pixabay

By default, a Jupyter Notebook saves your work every 2 minutes, and if you want to change this time interval you can do so by using the %autosave n magic command; where n is the number of seconds, and if n=0 this means no autosaving.

But the %autosave magic command produces effects only inside the notebook in which this command is run, and only in that particular session. If you shut down the notebook and re-open it you have to run the %autosave command again, or if you open another notebook, also you have to run the command again.


Understand Logistic Regression and sharpen your PyTorch skills

To understand better what we’re going to do next, you can read my previous article about logistic regression:

So, what’s our plan for implementing Logistic Regression with PyTorch?

Let’s first think of the underlying math that we want to use.

There are many ways to define a loss function and then find the optimal parameters for it, among them, here we will implement in our LogisticRegression class the following 3 ways for learning the parameters:

  • We will rewrite the logistic regression equation so that we turn it into a least-squares linear regression problem with different labels and then, we use…

…something not as hard as you may think


If you are here for a quick solution that just works, then here it is in just 5 lines of code:

model = tf.keras.models.Sequential([    tf.keras.layers.Dense(1, activation='sigmoid')])model.compile(loss='bce'), y_train, epochs=100)

The long way

Now, if you’re still with me it means that you don’t want just to copy + paste 5 lines of code, but to see how you can actually implement this method yourself from scratch.

TensorFlow is a rich library; it has many APIs that you can use. Among them is the Keras API which can be used to build a logistic regression model very quickly, as you can see…

Try this Jupyter extension

Background image by Free-Photos from Pixabay

By default, when the output of a code cell in a Jupyter notebook gets too long, the content remains in its initial position and you have to manually scroll down to see the latest output. And this can be very annoying.

For example, if you train a Machine Learning model from a Jupyter notebook and you want some info to be printed after each batch or epoch, you would like to have the output automatically scrolling so that you can see the message about the last printed epochs, but instead, nothing happens, you have to scroll down manually.

Below is…

How I learned more in 3 months rather than 4 years

Image by Sasin Tipchai from Pixabay

Disclosure: below you’ll find affiliate links to these two books

C and C++ are, despite their age, some of the most used programming languages across the globe and are among the first ones taught in schools and universities. If you are taking a programming course right now, there are big chances you are struggling to learn one of these 2 languages.

I first started studying C++ (actually, it was mostly C, with just a few elements of C++) when I went into high school. Back then, I found this programming language quite confusing and difficult. After 4 years of studying…

Dorian Lazar

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store