12. Lab: Variables
How would you initialize the variables that you set up in your TensorFlow program? This lecture will help you find the answer to this question. We are continuing to work on very simple math operations in TensorFlow. This time we’ll introduce a new concept variables. Variables in TensorFlow, just like variables in any other programming language, are a constant construct whose value can be changed at runtime, the values can be updated and new values can be assigned to those variables. Variables are a very core construct in TensorFlow, and they are extremely important for machine learning operations.
If you think of how machine learning works, those programs are essentially iterative. You run through the code multiple times, you perform multiple computations, feedback the error back to the beginning of your program, find new values which you assign to variables and run through the iteration again. Iterative machine learning is possible using variables in TensorFlow. For this lab, we’ll work on the code that’s stored in the file. Simple math with variables import the TensorFlow libraries at the very top. And then let’s look at all the tensors that we’ve instantiated. We’ve set up a simple equation which represents a line.
The equation for a line, if you remember, is y is equal to WX plus b in this initial setup. Notice that w and b are both variables. Both have some initial values that we’ve specified in the form of NSRs. Both are float 32, and we’ve given both of them names VAR underscore w and VAR underscore b. X is a placeholder when we compute the value of y, the value of x has to be fed in, and y, which is the result of WX plus b, will also be a variable. If you use variables in your TensorFlow program, and the chances are you will, you have to run a special computation to initialize their values. Variables cannot be used.
Before running this explicit initialization step, call EF global variables initializer to initialize all the variables that you’ve set up in your program so far instantiate a session and within a session, the very first thing that you need to do is call session run on in it. This is the node that will initialize all your variables. With your variables initialized, you can now call session run on y. Why requires the value of x before it can be computed, which is why you’ll pass in the value of x using a feed dictionary. With these values of x and the initialized value of w and b, we get the final result for WX plus b as 30 comma 410.
Let’s assume that you have a whole bunch of variables in your program and you don’t require all of them to be initialized for a particular computation. You can initialize just certain variables by using PF variables initializer pass in the variables that you want to initialize. As an argument to this function here, we just want to initialize the variable w. Note that variable b has not been initialized instantiate a session and within it try to compute the value of Y. If you remember, y is equal to WX plus B. It requires the variable W as well as the variable P. Build this run successfully? Well, it won’t. We only initialize the variable W. Computing Y required the variable W as well as the variable B to be initialized.
Our code will throw an error, and if you scroll down and look carefully at the error message, you’ll find the correct reason there. Attempting to use uninitialized value VAR underscore p so far we’ve initialized a bunch of variables, but we haven’t seen how we can update the value of a variable within a particular session. Here we set up two variables, one referenced by number and the other, which is a multiplier. Number has been initialized to ten, and multiplier has been initialized to the value one. Call TF global variables initializer to initialize all the variables. We also set up another computation. We multiply number by multiplier and store the result in number instantiate a session.
Initialize all the variables by calling session run on in it, and we are now ready to change the values of our variables. Set up a for loop for I in range five. It will iterate through its body five times. And within the for loop, the very first line of code computes the value of a result called session run on result. In the second line, within the body of the for loop, increment the value of the multiplier by one multiplier assign add one. Notice that we are changing the value of the variables in each of these session run executions. Run this bit of code and let’s see what the output looks like. The very first result is ten. The number was ten. The multiplier was one. Number is equal to number multiplied by multiplier, which is equal to ten.
In the next step, we increment the value of the multiplier by one. The multiplier was originally one. It now has the value two. Now when you compute the value of result, number is equal to ten, multiplier is equal to two, and number is equal to number multiplied by multiplier gives you 20, so on. In the next step, we increment the value of multiplier. It is now three. We use this new value of multiplier, perform the multiplication with the current value of number, which is 20 to get a result of 60. Okay, we’ve performed a whole host of computations. Let’s write out summary information to TensorBoard and view it in our visualization tool.
Click on the Graphs tab and here is the graph for our TensorFlow computation. The graph is pretty big because we performed a number of computations. I suggest you explore this and see if it makes sense to you. There is one interesting thing that I’ll point out though. Notice the variable W and notice that it is an input to three nodes with the prefix. In it, in it, in it underscore one, and in it underscore two. If you remember the program that we just looked at, we initialized W three times, twice using the global variables initializer method and once using just initialized variables. Those are three initialization compute nodes, and W is an input to all three.
On the other hand, if you look at the variable B, it is an input to only two initialization compute nodes in it and in it underscore two, in it underscore one. Use the variables initializer to only initialize W. It did not initialize B. That’s why we got the error in the first place. In this lecture, we learned two ways in which we can initialize variables that we instantiate in TensorFlow. One is using PF global variables initializer, which initializes all the variables that we’ve set up so far, and the second is using variables initializer, where we specify the variables that we want to initialize. Variables in TensorFlow have to be initialized before you can use them in your computation.
13. Lab: Linear Regression with Made-up Data
What is an example of an optimizer library available in TensorFlow that you can use to minimize loss while computing your linear regression model? What you’re going to see in this lecture is a very basic example of linear regression in TensorFlow. We are not going to use any real data. In fact, we are just plotting four points on our graph and we’ll try and find the best fit line that passes through these four points. In the real world, of course, you’ll be running these models on huge data sets, but working with fake data will allow you to understand the basic steps in machine learning without worrying about the complexity of the data set. This is the first example that you will see.
This serves as an introduction to TensorFlow. The import statements here are as usual. Let’s move on to setting up the model parameters. In our regression model, we want to calculate the values of W and B. We set up both of these as variables within our TensorFlow program. W is the slope of a straight line and B is the y intercept. These are the variables that the model will tweak and give you a final result. For once you have these values of W and B from a trained model, they can be used for prediction given any new value of x. Predict the value of y using WX plus B. The values of x is the data that we feed into this model, which is why x is instantiated as a placeholder.
Linear model is equal to W multiplied by x plus B. That is the formula of a straight line. That is our regression formula. Y is the placeholder which holds the labels for our training data. For every x value which is the input, this is the true y value. Labels are used in a certain class of machine learning algorithms called supervised learning, which we’ll look at in just a little bit. These labels are what you will compare with when you minimize the loss. When you try and fit your regression line, you feed in these y values that you have available, which is why this is a placeholder. You then need to set up a function which calculates the loss of your regression model.
Remember that the entire objective of your model is to minimize this loss function.Smaller the loss of your model, the more accurate it is and better its predictions will be. We’ll calculate this loss using mean square errors, which is the sum of the square of the difference between the predicted values and the actual values. The TensorFlow libraries offer you a bunch of optimizers. Optimizers are essentially functions which allow you to minimize your losses. An optimizer follows a method by which it minimizes the loss of your machine learning model, thus allowing you to tweak your model parameters to improve the model. A standard optimizer library to use is the gradient descent.
Optimizer. Why gradient descent? And what exactly it does is out of scope for right now. We’ll be learning all of this in just a little bit. The training step for your algorithm is when we use the optimizer to minimize your mean square error loss. Optimizer minimize loss instantiate our training data. X underscore train are the xvalues one, two, three and four, and y underscore train are the y values or the labels for the corresponding x values. A training loop starts off by initializing all the variables that you’ve set up so far using the global variables. Initializer instantiate a session using a with statement. We then set up a for loop which runs a thousand times here and run your training model pass in your training data and your corresponding labels.
Session dot run computes our training model and x train and y train are passed in via the feed dictionary. Once our model is fully trained, let’s examine the values of the parameters. The current value of W, the current value of B, and the loss. The model should have converged to some value of all these variables. Print these out to screen. Just execute this bit of code and there you see it. Our regression model parameter. A value of W that is very close to minus one, a value of B that’s very close to one, and very small loss. A standard optimizer library that TensorFlow offers you in order to minimize your loss functions is the gradient descent optimizer. That’s what we used in this very simple linear regression model implemented in TensorFlow.
14. Image Processing
Here is a question that I’d like you to ponder over as we go through the contents of this video. What kind of neural network has become really popular for image processing? Think different types of neural networks, such as convolutional, recurrent, bidirectional, and so on. Let’s now change tracks a little bit and switch from regression to working with images. We are not done with regression, not by a long way. We will get back to linear and logistic regression in TensorFlow in a great deal of detail. But before that, let’s just talk about working with images. Because this is a fairly typical use case of TensorFlow in the real world, the regression is pretty much a solved problem.
Even Microsoft Excel has really powerful functionality on regression and has had this for the past few decades. Working with images, on the other hand, is something which is really new to the world of machine learning. And this is why TensorFlow is so hot, working with hard to represent data like images and videos. This is where neural networks and deep learning really come into their own. Indeed, image recognition is really the starter problem in TensorFlow, as we are going to explore in just a moment. Working with images requires us to understand some basic operations. We need to know how to represent images in the form of tensors.
And these tensors need to encode color as well as grayscale information. We will also learn some basics of image operations, such as cropping, transposing and resizing images. All of these will come in handy once we actually start using TensorFlow for more advanced stuff. So again, the basic idea is image recognition. The basic idea is to detect what an image represents from the pixels that constituted. In case you’re not familiar with the idea of pixels, every little region of an image can be decomposed into specific rectangular shapes called pixels. Pixels contain within them a wealth of information. They are a form of encoding. Let’s understand how pixels can help with image recognition.
In fact, pixels really are the features in our feature vector. During the image recognition machine learning problem solution. We had hinted at this while discussing the use of TensorFlow for image recognition. There we had spoken about how images are represented as pixels. Those pixels have higher level features such as edges, colors, and shapes. And on the basis of all of these, the basic building blocks, including the pixels, the more intermediate building blocks, the colors, shapes and tints, we are able to recognize what that image represents. This is a process which the human brain is really good at. We achieve an image at a glance.
In fact, the human brain is able to encode information with images far more efficiently than using text. The efficiency of that encoding is exactly what gives rise to statements of proverbs like a picture is worth a thousand words. Automating this, allowing machines or computers to do this, is what image recognition and machine learning are all about. We want a computer program which will take in the pixels, identify the higher level features and ultimately tell us what that picture represents. Neural networks, specifically convolutional neural networks are really useful. They are really popular for image recognition problems.
And because neural networks are best implemented in TensorFlow, this is the tool of choice. These days, image recognition has pretty much become a bread and butter use case for TensorFlow. Or to put it differently, TensorFlow is becoming the dominant tool, the technology of choice for folks working with image recognition. Let’s understand this in just a little more detail. Let’s see how image recognition can be achieved using Neural networks at a high level here. The problem that we are going to set out to solve is a classification one. We have a corpus of images. We then have a feature selection and classification algorithm.
This is going to rely on Neural networks. Inside, we will pass the images in our corpus into this algorithm one by one. At the initial level, we will have a layer which deals with the pixels. Then intermediate layers within the neural network will deal with higher level constructs such as the edges, the contours and so on. And then at the end of it all will come a final output layer, a classification layer which will hopefully have the ability to classify the corpus of input images based on their content. So, for instance, maybe we would like to classify these images as belonging to either fish or mammals. This is something which the final step of the Neural network would do.
And that would imply that the output as a whole would be a trained classifier. This machine learning based classifier after training could then be passed in a new image and it would tell us whether that represented the image of a fish or of a mammal. It’s important to keep in mind that this is a two step process. So if you were surprised that the output of this neural network was not a label, but rather that it was a classifier, understand that the way machine learning and Neural network algorithms work. You first train a model. You use the training data, the corpus of images to tweak or optimize the values of the variables, the parameters in that model.
And at the end of that process, you get a model which you can then use to actually classify new images. We had discussed that these layers in the Neural network are called either visible or hidden based on whether we as the end user actually understand what they are operating on. This is a quick introduction to neural networks. Do keep in mind how these layers in general are opaque to the programmer. And a big part of the challenge in designing good machine learning algorithms using Neural networks is to wire up those individual neurons which are present inside these different layers, even though the working of those layers is more or less opaque to you.
It’s more or less unknown. As an example, you might have a neural network with a very dense set of interconnections. You might then go ahead and find that this dense set of interconnections causes your neural network to work well with your training data, but to fare poorly when given new data. You then will specify something known as a dropout factor. You will break completely arbitrarily at random a bunch of the interconnections in your network. This almost represents an act of faith because you have no idea whether this is going to help or hurt with the test data.
This is something that you’ve just got to do because the science and the technology has shown that it works. TensorFlow is optimized. It’s aimed at building neural networks like this one, image recognition or x processing, or any of these use cases using neural networks. All of these are preferred dominant use cases of TensorFlow. But do be aware that while you are using TensorFlow, you are going to have to accept that there is a lot going on under the hood within that network which you will not understand. Let’s return now to the question we posed at the start of this video. Convolutional neural networks have become most popular for image processing.
In fact, there is an algorithm called Inception which makes use of convolutional neural networks and has currently defined the best possible performance, the best performance out there for image classification tasks. Other types of neural networks have their own specific use cases. For instance, recurrent neural networks tend to be used heavily in text processing for applications such as autocomplete or generating sequences of text which are similar to those written by an author. In cases where we have sequences of data kind of arranged in time, such as time series data, recurrent neural networks are more popular. Where we have spatially arranged data, such as pixels in an image. Convolutional neural networks currently are the tool of choice.
