Tensorflow - softmax returning only 0 and 1

Question

I'm training a CNN on tensorflow but I'm having trouble with my loss that is not improving; I've noticed that tf.nn.softmax() is returning a tensor with only 0 and 1 and not a distribution as I'd expect. Here's the repo, I believe that's the reason of my incapability of training the network but I don't know how to solve it.

Answer 1

Looking at the 2nd box under The Neural Network:

# output layer
with tf.variable_scope('output_lay') as scope:
    weights = weight_variable([4096, CLASSES])
    bias = bias_variable([CLASSES], 0.)
    activation = tf.nn.relu(tf.matmul(out, weights)+bias, name=scope.name)
    out = tf.nn.softmax(activation)
return tf.reshape(out, [-1, CLASSES])

NB : ReLu activation is only used for hidden layers not output layer.

Then you are feeding this to cross-entropy in your train function

logits=AlexNet(x_tr)

# loss function
cross_entropy = -tf.reduce_sum(tf.squeeze(y_tr)*tf.log(tf.clip_by_value(tf.squeeze(logits),1e-10,1.0)))
loss = tf.reduce_mean(cross_entropy)

Re-visiting cross entropy:

C= −1/n * (∑[y*ln(a)+(1−y)*ln(1−a)])

where a = sigmoid(W(x)+b), So I suggest :

with tf.variable_scope('output_lay') as scope:
    weights = weight_variable([4096, CLASSES])
    bias = bias_variable([CLASSES], 0.)
    return tf.matmul(out, weights)+bias

and for simplicity just use inbuilt softmax function:

logits=AlexNet(x_tr)

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=ground_truth_input, logits=logits)
loss = tf.reduce_mean(cross_entropy)

tf.nn.softmax_cross_entropy_with_logits takes in W(x)+b and efficiently calculates the cross entropy.