Selecting Trainable Variables To Compute Gradient "No Variables To Optimize"

I am trying to select a subset of the trainable variables, the ones in the first and second hidden layer, to perform some operations with their gradients (for example, clipping the

Solution 1:

If you write

print train_vars

you would see, that this collection is indeed empty. You can get all variables by

print tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)

To get a subset I suggest to do

filter_vars = ['hidden1', 'hidden2']
train_vars = []
for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES):
    for filter_var in filter_vars:
        if filter_var in var.name:
            train_vars.append(var)
# print train_vars

or

train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope="hidden1")
train_vars += tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope="hidden2")
# print train_vars

Both gives you:

[<tf.Variable 'hidden1/weights:0' shape=(50, 20) dtype=float32_ref>, 
 <tf.Variable 'hidden1/biases:0' shape=(20,) dtype=float32_ref>, 
 <tf.Variable 'hidden2/weights:0' shape=(20, 10) dtype=float32_ref>, 
 <tf.Variable 'hidden2/biases:0' shape=(10,) dtype=float32_ref>]

But my favorite one is using RegEx:

import re
regex_pattern = 'hidden[12]'
train_vars = []
for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES):
    if re.match(regex_pattern, var.op.name):
        train_vars.append(var)
print train_vars