Keras Model To Predict Probability Distribution

We are trying to build a keras model to predict a vector with probablity rates from a vector of features. The output vector should be of probabilty rates which are between 0 and on

Solution 1:

The last activation function to guarantee that the sum is 1 is "softmax".

Now, a frozen loss may be caused by "relu" in this case where you have so few neurons in each layer. (Also a improper weight initialization)

I suggest instead of relu you use "softplus", "tanh" or even "sigmoid".

EDIT:

As @nuric suggested, it's really a good idea to use "categorical_crossentropy" as loss when you're using "softmax".