14159

TensorFlow learning rate decay - how to properly supply the step number for decay?

Question:

I am training my deep network in TensorFlow and I am trying to use a learning rate decay with it. As far as I see I should use train.exponential_decay function for that - it will calculate the proper learning rate value for current training step using various parameters. I just need to provide it with a step which is performed right now. I suspected I should use tf.placeholder(tf.int32) as usual when I need to provide something into the network, but seems like I am wrong. When I do this I get the below error:

TypeError: Input 'ref' of 'AssignAdd' Op requires l-value input

What am I doing wrong? Unfortunately, I haven't managed to find some good example of network training with decay. My whole code is below. Network has 2 hidden ReLU layers, has L2 penalty on weights and has dropout on both hidden layers.

#We try the following - 2 ReLU layers #Dropout on both of them #Also L2 regularization on them #and learning rate decay also #batch size for SGD batch_size = 128 #beta parameter for L2 loss beta = 0.001 #that's how many hidden neurons we want num_hidden_neurons = 1024 #learning rate decay #starting value, number of steps decay is performed, #size of the decay start_learning_rate = 0.05 decay_steps = 1000 decay_size = 0.95 #building tensorflow graph graph = tf.Graph() with graph.as_default(): # Input data. For the training data, we use a placeholder that will be fed # at run time with a training minibatch. tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size * image_size)) tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels)) tf_valid_dataset = tf.constant(valid_dataset) tf_test_dataset = tf.constant(test_dataset) #now let's build our first hidden layer #its weights hidden_weights_1 = tf.Variable( tf.truncated_normal([image_size * image_size, num_hidden_neurons])) hidden_biases_1 = tf.Variable(tf.zeros([num_hidden_neurons])) #now the layer 1 itself. It multiplies data by weights, adds biases #and takes ReLU over result hidden_layer_1 = tf.nn.relu(tf.matmul(tf_train_dataset, hidden_weights_1) + hidden_biases_1) #add dropout on hidden layer 1 #we pick up the probabylity of switching off the activation #and perform the switch off of the activations keep_prob = tf.placeholder("float") hidden_layer_drop_1 = tf.nn.dropout(hidden_layer_1, keep_prob) #now let's build our second hidden layer #its weights hidden_weights_2 = tf.Variable( tf.truncated_normal([num_hidden_neurons, num_hidden_neurons])) hidden_biases_2 = tf.Variable(tf.zeros([num_hidden_neurons])) #now the layer 2 itself. It multiplies data by weights, adds biases #and takes ReLU over result hidden_layer_2 = tf.nn.relu(tf.matmul(hidden_layer_drop_1, hidden_weights_2) + hidden_biases_2) #add dropout on hidden layer 2 #we pick up the probabylity of switching off the activation #and perform the switch off of the activations hidden_layer_drop_2 = tf.nn.dropout(hidden_layer_2, keep_prob) #time to go for output linear layer #out weights connect hidden neurons to output labels #biases are added to output labels out_weights = tf.Variable( tf.truncated_normal([num_hidden_neurons, num_labels])) out_biases = tf.Variable(tf.zeros([num_labels])) #compute output #notice that upon training we use the switched off activations #i.e. the variaction of hidden_layer with the dropout active out_layer = tf.matmul(hidden_layer_drop_2,out_weights) + out_biases #our real output is a softmax of prior result #and we also compute its cross-entropy to get our loss #Notice - we introduce our L2 here loss = (tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( out_layer, tf_train_labels) + beta*tf.nn.l2_loss(hidden_weights_1) + beta*tf.nn.l2_loss(hidden_biases_1) + beta*tf.nn.l2_loss(hidden_weights_2) + beta*tf.nn.l2_loss(hidden_biases_2) + beta*tf.nn.l2_loss(out_weights) + beta*tf.nn.l2_loss(out_biases))) #variable to count number of steps taken global_step = tf.placeholder(tf.int32) #compute current learning rate learning_rate = tf.train.exponential_decay(start_learning_rate, global_step, decay_steps, decay_size) #use it in optimizer optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step) #nice, now let's calculate the predictions on each dataset for evaluating the #performance so far # Predictions for the training, validation, and test data. train_prediction = tf.nn.softmax(out_layer) valid_relu_1 = tf.nn.relu( tf.matmul(tf_valid_dataset, hidden_weights_1) + hidden_biases_1) valid_relu_2 = tf.nn.relu( tf.matmul(valid_relu_1, hidden_weights_2) + hidden_biases_2) valid_prediction = tf.nn.softmax( tf.matmul(valid_relu_2, out_weights) + out_biases) test_relu_1 = tf.nn.relu( tf.matmul( tf_test_dataset, hidden_weights_1) + hidden_biases_1) test_relu_2 = tf.nn.relu( tf.matmul( test_relu_1, hidden_weights_2) + hidden_biases_2) test_prediction = tf.nn.softmax(tf.matmul(test_relu_2, out_weights) + out_biases) #now is the actual training on the ANN we built #we will run it for some number of steps and evaluate the progress after #every 500 steps #number of steps we will train our ANN num_steps = 3001 #actual training with tf.Session(graph=graph) as session: tf.initialize_all_variables().run() print("Initialized") for step in range(num_steps): # Pick an offset within the training data, which has been randomized. # Note: we could use better randomization across epochs. offset = (step * batch_size) % (train_labels.shape[0] - batch_size) # Generate a minibatch. batch_data = train_dataset[offset:(offset + batch_size), :] batch_labels = train_labels[offset:(offset + batch_size), :] # Prepare a dictionary telling the session where to feed the minibatch. # The key of the dictionary is the placeholder node of the graph to be fed, # and the value is the numpy array to feed to it. feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, keep_prob : 0.5, global_step: step} _, l, predictions = session.run( [optimizer, loss, train_prediction], feed_dict=feed_dict) if (step % 500 == 0): print("Minibatch loss at step %d: %f" % (step, l)) print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels)) print("Validation accuracy: %.1f%%" % accuracy( valid_prediction.eval(), valid_labels)) print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

Answer1:

Instead of using a placeholder for global_step, try using a Variable.

global_step = tf.Variable(0)

You will have to remove global_step from the feed_dict. Note that you don't have to increment global_step manually, tensorflow will do it automatically for you.

Recommend

  • TensorFlow learning rate decay - how to properly supply the step number for decay?
  • Tensorflow vs. Numpy Performance
  • Multiple image upload using php [closed]
  • How to correctly add the record.image_integrity_error to existing condition?
  • Tensorflow for image segmentation: Changing minibatch size stops learning
  • Why does this Keras model require over 6GB of memory?
  • TensorFlow: The tensor is not the element of this graph
  • Tensorflow - Loss increases to NaN
  • Writing and reading from a virtual i2c using C++ and i2c-tools
  • Laravel 5.5 - Handling PostTooLargeException for large base64 image?
  • Part 3: Switching between multiple contexts - no error and a bad exit code
  • Edit and crop uploaded image
  • Tensor indexing in custom loss function
  • Change input resolution for QCamera
  • How to get images filenames from minibatch?
  • Tensorflow: What is the output node name in Cifar-10 model?
  • Django static file and nginx
  • do I even need `htmlspecialchars()` for textarea's value
  • TensorFlow/TFLearn: ValueError: Cannot feed value of shape (256, 400, 400) for Tensor u'Targets
  • LLVM-5.0 Makefile undefined reference fail
  • Shopify: Why does Liquid sometimes use {%- instead of {%?
  • Minimum Cost Flow - network optimization in R
  • tensorflow embeddings don't exist after first RNN example
  • K Shortest Path Python Not Working
  • Rely on Facebook user id as a permanent user identifier
  • How to add learning rate to summaries?
  • Double dispatch in Java example
  • Redirect STDERR in OPEN pipe comand. Perl Linux
  • How to recover from a Spring Social ExpiredAuthorizationException
  • Does CUDA 5 support STL or THRUST inside the device code?
  • SVN: Merging two branches together
  • Hibernate gives error error as “Access to DialectResolutionInfo cannot be null when 'hibernate.
  • How to CLICK on IE download dialog box i.e.(Open, Save, Save As…)
  • embed rChart in Markdown
  • Can Visual Studio XAML designer handle font family names with spaces as a resource?
  • How can I remove ASP.NET Designer.cs files?
  • Are Kotlin's Float, Int etc optimised to built-in types in the JVM? [duplicate]
  • How to get NHibernate ISession to cache entity not retrieved by primary key
  • How can I use `wmic` in a Windows PE script?
  • Unable to use reactive element in my shiny app