How to chose the number of clusters in a model?

After a quick reminder of how it should be done theoretically from an article in Data Science Central (The full article can be found below) https://www.datasciencecentral.com/profiles/blogs/how-to-automatically-determine-the-number-of-clusters-in-your-dat

 We will explain how we actually do it in the real world of marketing below.

 1 - The theory of determining clusters

Determining the number of clusters when performing unsupervised clustering is a tricky problem. Many data sets don't exhibit well separated clusters, and two human beings asked to visually tell the number of clusters by looking at a chart, are likely to provide two different answers. Sometimes clusters overlap with each other, and large clusters contain sub-clusters, making a decision not easy.

For instance, how many clusters do you see in the picture below? What is the optimum number of clusters? No one can tell with certainty, not AI, not a human being, not an algorithm.

How many clusters here? (source: see here)

In the above picture, the underlying data suggests that there are three main clusters. But an answer such as 6 or 7, seems equally valid. A number of empirical approaches have been used to determine the number of clusters in a data set. They usually fit into two categories:

• Model fitting techniques: an example is using a mixture model to fit with your data, and determine the optimum number of components; or use density estimation techniques, and test for the number of modes (see here.) Sometimes, the fit is compared with that of a model where observations are uniformly distributed on the entire support domain, thus with no cluster; you may have to estimate the support domain in question, and assume that it is not  made of disjoint sub-domains; in many cases, the convex hull of your data set, as an estimate of the support domain, is good enough. 
• Visual techniques: for instance, the silhouette or elbow rule (very popular.)

In both cases, you need a criterion to determine the optimum number of clusters. In the case of the elbow rule, one typically uses the percentage of unexplained variance. This number is 100% with zero cluster, and it decreases (initially sharply, then more modestly) as you increase the number of clusters in your model. When each point constitutes a cluster, this number drops to 0.  Somewhere in between, the curve that displays your criterion, exhibits an elbow (see picture below), and that elbow determines the number of clusters. For instance, in the chart below, the optimum number of clusters is 4.

The elbow rule tells you that here, your data set has 4 clusters (elbow strength in red)

Good references on the topic are available. Some R functions are available too, for instance fviz_nbclust. However, I could not find in the literature, how the elbow point is explicitly computed. Most references mention that it is mostly hand-picked by visual inspection, or based on some predetermined but arbitrary threshold. In the next section, we solve this problem.

2 - How we actually find clusters

The article is longer and explains how to actually do it.
 https://www.datasciencecentral.com/profiles/blogs/how-to-automatically-determine-the-number-of-clusters-in-your-dat

But in real life, and if you have any understanding of your data at all, this is useless!

It is essential to understand why with a quick example.

In the chart above, the optimum number of clusters is 4. But what if we chose 6 as indicated or 14 as the underlying data in grey seems to indicate? In that case, what we find is that the balance of the clusters (not just the mode) changes quite dramatically, and the 4 original clusters become unrecognizable which is normal since we have a rather large overlap of data.

And unfortunately, the result as indicated by the market (on-line client use) is bad. Our clusters are under-performing.

To solve the problem, we inverted the question: What are the "known" clusters in our population? Whatever the data you work with, there will be some accumulated knowledge about that population. In our case, marketing, clusters are clear and defined by the clients: "Young working women" or "Retired urban couples" for example.

Our task is now simpler and far more accurate: Which "known" clusters can we identify in our data?

This has proved extremely helpful and made our clusters very popular with clients because they recognize them right-away and because they perform well!

The only risk is that often you cannot identify the "known" clusters in you data because of the difficulty to link parameters. In that case the solution is quite simple: Vary your parameters and see how your data distribution changes.