Sentence Similarity Using Keras

I'm trying to implement sentence similarity architecture based on this work using the STS dataset. Labels are normalized similarity scores from 0 to 1 so it is assumed to be a regr

Solution 1:

The nan is a common issue in deep learning regression. Because you are using Siamese network, you can try followings:

1. check your data: do they need to be normalized?
2. try to add an Dense layer into your network as the last layer, but be careful picking up an activation function, e.g. relu
3. try to use another loss function, e.g. contrastive_loss
4. smaller your learning rate, e.g. 0.0001
5. cos mode does not carefully deal with division by zero, might be the cause of NaN

It is not easy to make deep learning work perfectly.

Solution 2:

I didn't run into the nan issue, but my loss wouldn't change. I found this info check this out

def cosine_distance(shapes):
    y_true, y_pred = shapes
    def l2_normalize(x, axis):
        norm = K.sqrt(K.sum(K.square(x), axis=axis, keepdims=True))
        return K.sign(x) * K.maximum(K.abs(x), K.epsilon()) /     K.maximum(norm, K.epsilon())
    y_true = l2_normalize(y_true, axis=-1)
    y_pred = l2_normalize(y_pred, axis=-1)
    return K.mean(1 - K.sum((y_true * y_pred), axis=-1))