classifier accuracy

This assumption (and the corresponding intuition) breaks down in cases of class imbalance: if we have a dataset where, say, 90% of samples are of class 0 (i.e. P(class=0)=0.9), then it doesn't make much sense to use the above definition of a random binary classifier; instead, we should use the percentages of the class distributions themselves as the probabilities of our random classifier, i.e.:

As I already said, AFAIK there are no clear-cut definitions of a random classifier in the literature. Sometimes the "naive" random classifier (always flip a fair coin) is referred to as a "random guess" classifier, while what I have described is referred to as a "weighted guess" one, but still this is far from being accepted as a standard...

The bottom line here is the following: since the main reason for using a random classifier is as a baseline, it makes sense to do so only in relatively balanced datasets. In your case of a 60-40 balance, the result turns out to be 0.52, which is admittedly not far from the naive one of 0.5; but for highly imbalanced datasets (e.g. 90-10), the usefulness itself of the random classifier as a baseline ceases to exist, since the correct baseline has become "always predict the majority class", which here would give an accuracy of 90%, in contrast to the random classifier accuracy of just 82% (let alone the 50% accuracy of the naive approach)...

Stack Exchange network consists of 177 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.

The accuracy given by Keras is the training accuracy. This is not a proper measure of the performance of your classifier, as it is not fair to measure accuracy with the data that has been fed to the NN. On the other hand, the test accuracy is a more fair measure of the real performance. As there is a big gap between them, you are overfitting very badly, and you should regularize your model. Consider using dropout or weight decay.

Stack Exchange network consists of 177 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Apologies if this is a very obvious question, but I have been reading various posts and can't seem to find a good confirmation. In the case of classification, is a classifier's accuracy = 1- test error rate? I get that accuracy is TP+TNP+N, but my question is how exactly are accuracy and test error rate related.

In principle yes, accuracy is the fraction of properly predicted cases thus 1-the fraction of misclassified cases, that is error (rate). Both terms may be sometimes used in a more vague way, however, and cover different things like class-balanced error/accuracy or even F-score or AUROC -- it is always best to look for/include a proper clarification in the paper or report.

Stack Exchange network consists of 177 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

I am currently implementing a simple feed forward ANN to a classification problem with 3 possible outcomes/classes. The results don't look great, therefore, I am currently thinking about whether my ANN even learns anything. All accuracy numbers in the following are related to the cross validation accuracy.

From my point of view, if it would not learn anything, it would either assign classes randomly or it would assign each observation the same class. As I have a balanced dataset, this would mean that the accuracy should be at 33.3% if the ANN is not learning anything. Is this a correct analysis? Or would 50% imply not learning anything because it is a coin toss whether our prediction fits or not? I am really confused that this is not a binary class setting but that we have three classes.

An accuracy of 50% means it learns something in this scenario assuming a balanced dataset. If you can edit your question with the architecture of your NN I can perhaps give advice on how to make some architectural improvements.