5. Lab: Simple Math Operations
At the end of this lecture, you should be able to clearly define the two phases in any TensorFlow program. TensorFlow is a very powerful numerical computation framework that was open sourced by Google in November 2015. Representation of all computations as a directed acyclic graph makes it very suitable for machine learning applications such as neural networks. In order to get an understanding of how TensorFlow works and how we would set up operations in TensorFlow, we’ll start off with some very simple numerical computations. Simple math using TensorFlow constants. We are going to create a brand new Data Lab instance where we’ll run our TensorFlow programs.
But before that, let’s set up some default configuration properties for our Google Cloud platform account. We use the Gcloud command line utilities to send these configuration properties. We want to set the default project to be my test project. That’s the project we’ve been working on so far. We also set the default computation zone to be US central, one f from your cloud shell command line, you can use the Data Lab Create command to set up a Data Lab VM instance. The code that we execute within this instance is temporary. We don’t want to commit those code files to GitHub, which is why we can say no. Create repository as a command line argument to Data Lab Create and the last argument here is the name of your Data Lab instance.
This is the name that we’ll use to connect to Datalab. This command will not exit on this terminal window. We simply use the web preview to connect to port 80 81. This is the connection to our Data Lab instance. Once you have the browser pointed to your Data Lab instance, you can create a new folder where you store all your TensorFlow code files. So create a new folder called Learning TensorFlow. The source code for this demo and for the next few demos, maybe about ten or twelve of them, should be available for you to download. Along with this lecture, you can also choose to download the source code from this short link. This is a Google Drive folder, and it’s publicly accessible to anyone.
This source code is not a part of Google’s Code Lab series and is not available on GitHub anywhere. Once you’ve downloaded the resources that are associated with the next few demos, you can unzip it and then upload it to this particular folder. Click on the upload button. I’m going to select all the Python notebook files that are present in the source code folder, and I’ll upload it to Datalab. Let’s start off with our very first TensorFlow code. Open up the simple math python notebook. The TensorFlow libraries are available in a bunch of different languages python, Java, C, and even Go. Of all these APIs, Python is the most stable, and that is what is recommended for production systems.
In the first code cell, we import TensorFlow as TF. It’s very common to refer to TensorFlow as TF within your program. That’s the short form. And we import the ML library from Google data lab. We then instantiate a whole bunch of constants A-B-C and B. This is simple scalar data. Here it has just one value. Constants are essentially exactly what their name implies. These values cannot be changed. Over the course of this program, I’ve specified a scalar value for every constant and also associated a name with each of these constants. The name help identify these constants. When you view the visualization of this computation graph on TensorBoard, we’ll now perform some very simple math computations using the simple math functions that TensorFlow provides.
The operations that we perform are add A plus B subtract B minus C, and we square D, and each of these operations also have an associated name. The results of these operations then feed into final sum. Final sum adds up the result of add, subtract, and square. Any TensorFlow program that you work with, whether it’s very complex neural networks or these simple math operations that you see on screen, they are executed in two phases. In the very first step, you build up a computation graph that you want to execute. At this point in our simple TensorFlow program, we’ve completed the very first step building the computation graph. This computation graph the operations that we perform on the data such as add, subtract, and square.
These form the vertices and the data that’s operated on from the edges. In order to run this computation graph, we require the session object. This Session object represents the environment in which these computation graphs are executed, which is why we set up a session using PF Session and store it in the Cess variable. Within the session, we first compute the intermediate add, subtract, and square operations by calling Session run add cess run subtract, and Cess run Square. And finally, we compute the final sum. Note that it’s not required that we compute the intermediate value separately. You can call Session Run on any node of the computation graph, and TensorFlow will compute all nodes which feed into that node.
We also set up another calculation. Here another sum which adds up. ABCD the constants that we initialized and square. This is also executed via Session Run, and this forms the second step in any TensorFlow program. Once you build up the computation graph, you will then execute it or run it. One thing to ensure when you use a session variable is to close it when you’re done with it. That is good practice, which is why generally session variables are instantiated. Using the width statement, a shift enter on the cell will print out the results. To screen. Session run has successfully executed the nodes in our computation graph.
If you’re working on a TensorFlow program of any level of complexity, you typically want to visualize the computation graph you’ve set up and see how data flows through it. You can do this using a very cool tool that TensorFlow provides. The tensor board. The way you use TensorBoard is by generating summaries of quantitative metrics within your program. You can also use it to display images. Using TensorBoard requires you to instrument your program to write out summaries. This we do by using the TF summary file writer. This writes out the computation graph for this program to a directory called Simple Math, which is under our current working directory.
Remember to close the writer once you’re done with it. Once you’ve written out the summaries, you can use the ML library from the Google Data Lab namespace. In order to run TensorBoard, simply call Mlboard Start and pass to it the directory where the TensorBoard events have been written out. When you execute this command, you will get a link right there in your output to access TensorBoard. Click on that link and it will open up the web browser. To show you this TensorBoard start screen, click on the graphs tab, and there you’ll see a visualization of the Simple Math program that we implemented in TensorFlow. Here are some of the computation nodes in our graph. The ad underscore AB adds A and B, the constants that we initialized.
The name of this node is the name that we passed in when we set up this operation. Click on the various nodes. You’ll see what the inputs to that nodes are and where the output flows. The view pane that you see on the top right gives you the detailed information attributes of this particular operation, the inputs and the outputs, the output flow to final sum. The TensorBoard visualization is very cool indeed, and especially when you have massive neural networks set up within your TensorFlow computation graph. This is a very useful debugging and monitoring tool. In this lecture, you saw in practice the two phases of a TensorFlow program, building up the computation graph and then executing it using a session variable. You.
6. Computation Graph
Here is a question that I’d like us to think about as we go through this video. In TensorFlow, a program is represented using a computation graph, which is a tag that’s a directed asylic graph. The question is, what do the nodes and the edges in this directed a cyclic computation graph represent? Again, what do the edges and the nodes in a TensorFlow computation graph represent? End, the key to working with TensorFlow and the key to making TensorFlow work for you is understanding the computation graph. This is the central abstraction of all operations in TensorFlow. If you understand this, we have a fighting chance of creating pretty complex neural networks with a minimal amount of effort.
In TensorFlow, everything is a graph. This is an abstraction that works really well with neural networks because, after all, neurons tend to be interconnected in a form of a network. That’s the whole point of a neural network. And this is also why TensorFlow lends itself so naturally to working with neural networks. In a computation graph, the edges are going to be data items and the nodes are going to be computations, as we shall see in just a little bit. An individual neuron can be thought of as just two computations. So really, each neuron is a pair of operators. Those operators or operations can be pretty simple, but they transform data. And the edges in that computation graph are the data items.
Once again, the idea of data items being the edges in a graph where transformations on those data items are the nodes. This is hardly unique to TensorFlow, but it does lend itself really well to machine learning and neural network applications. So TensorFlow has made this paradigm its own, and the data items which flow between the nodes are called tensors. This is a standard world, again, not specific to TensorFlow. Tensors refer to any data of different dimensionality. More on that in a moment. So tensors are flowing through the computation graph. This flow of tensors is set up for you by the tool, and that’s where the name comes from, TensorFlow, which is all about the transformation of tensors via a computation graph.
Let’s understand how this could work with a simple operation which has nothing to do with machine learning. Let’s just say that we have a really simple arithmetic operation. We like to calculate a certain function. This is represented using a computation graph with two inputs. Let’s call these inputs A and B. Nothing too profound here. And this computation graph that we see represents a specific function. That function is Y and given by this expression that you see on screen. Now, the crucial bit to remember is that individual neurons perform operations very similar to these rounding and flooring addition and multiplication. This is not all that different from what an actual neural network would do.
It’s just that the number of nodes would be far greater than what we see on screen right now. More on neurons later. For now, let’s just understand the computation graph. We can see that this is clearly a directed acyclic graph. That’s a type of graph in which there are no cycles and in which all of the edges point in a certain direction. Here the edges point forward, ie. Towards the result. The directed asylic graph clearly has dependencies, ie. Nodes can depend on other nodes. For instance, here the rounding node is going to send its output into two other nodes, the addition and the multiplication nodes, respectively. This also illustrates the fact that a node can send its output to multiple nodes. It can also receive inputs from multiple nodes.
Dependencies are important in TensorFlow for a couple of reasons. One, they help with parallelization, because a graph can be split into components which can then be evaluated in parallel, provided each of them has no dependencies on any of the other subgraphs understanding dependencies also help with lazy evaluation. This means that TensorFlow can only evaluate that portion of a graph which really needs to be calculated for the result to be returned to a user. Imagine a network where an input comes into a node. This node represents the floor operator. The output of that node, that is, the floor of the input that came in gets passed to an intermediate node. This is an addition.
This will carry out the addition of that flowed input with something else, and then the output of that in turn goes to yet another node which is a multiplication operator, and so on and so forth. The process continues until at some point a final output is calculated. As far as that final node here is concerned, all of the nodes which come before it are dependencies. The nodes which lie immediately before it are direct dependencies, but all of the other nodes are indirect dependencies. So in order to calculate the output of that final node, we are going to need to know the outputs of all of the dependencies of that node.
It’s important to realize that in a directed asylum graph, control or data will only flow from the input towards the output. This is basically true for any sub portion of the graph. It’s not possible for any directed asylum graph to complete its computations if it has a cycle. So this is really important. A directed asyclic graph, by definition, does not have any cycles. If a graph did have cycles in some way, and if we created a cycle by somehow artificially reversing the direction of one of the arrows, then our computation would never finish. Because clearly in order to calculate the value of each of the nodes, we would require its dependencies, and those dependencies would require the original node.
The net effect of all of this is that a computation graph with cycles will just never finish. It will never terminate. It is important to make a distinction between the asyclic nature of a directed acyclic graph and the inherent cyclic nature of the process of machine learning. Let’s see why we make that statement. Let’s see why we describe machine learning as a cyclical process. The way machine learning works, we have a corpus of data. We pass that into some bit of software which performs feature selection and machine learning and we get an output. So far so good. All of the directionality here is forward. But now remember that a machine learning algorithm is one which changes its parameters or it changes itself in some way based on the input data.
So given a set of training data, we are going to calculate the loss or we are going to calculate the performance of our machine learning classifier in some way and feed that information back into the machine learning algorithm itself. And straight away you can see that we now have a cycle in this graph. This is not a computation graph, just to be clear, but we do have information which is flowing backwards and it’s important for us to conceptually understand how this backward flow of information changes the machine learning algorithm. Any machine learning algorithm is going to have some internal state. For instance, in a neural network we have different layers of neurons which are connected to each other in complex connections.
But that could also be a lot simpler. We might just have one linear regression going on where the only state that we have is the equation of a straight line. Whatever that internal state, that internal state needs to change based on the performance that we had versus the training data. For instance, we might change the slope of a regression line or the weights of neurons in a neural network. And those changes are carried out for us by TensorFlow during the training process, provided that we set up the loss function and the feedback loop correctly. So how does this feedback mechanism relate to that directed acylic graph which we discussed a moment ago? Well, it happens via a mechanism known as graph unrolling.
Let’s see how this plays out. Clearly, as always, we are going to have a directed acyclic graph. This will have data flowing into computation nodes and the results flowing forward. But now during the training process, the outputs of those results later on in that process are going to be fed back into a new iteration, a new version of the graph, which in a sense will have some different weights, some different parameters. These form the feedback mechanism. That feedback mechanism might, for instance, change the slope of a regression line based on the data in the training set.
The way this happens is by unrolling the graph through time so that each training step seems to represent a different graph and the output of each training step is then fed in as the input of the subsequent training steps graph. Just to be clear, once again, this process is handled for you by TensorFlow. So you don’t really interact with it or see how it’s playing out. But really, this is how training a machine learning model takes place. The computation graph is unrolled. The output from an earlier iteration is fed in as the input parameters into the next iteration. And because the training process can consist of thousands or even millions of steps, it’s up to us how often we wish to unroll this. That is a choice which we exercise based on the number of iterations in our training data.
Let’s return to the question we posed at the start of this video. In a TensorFlow computation graph, the edges represent data items that’s tensors. The nodes represent operations on those tensors. This is important, and this is actually common to most bag representations of data transformations in TensorFlow. The edges are tensors. In a technology like data flow on the Google cloud platform, the edges will represent ecollections that’s some kind of distributed collection. The idea is common when you have a set of operations represented using a directed acquisition graph, data flows between the nodes. The nodes represent operations on those data items. Those data items constitute the edges, which, of course, in TensorFlow are the tensors.
