Summary Notes on Unsupervised Learning

 Target: Clustering | Anomaly detection 


in contrast: clustering



Applications of clustering:
Grouping similar news
Market segmentation
DNA analysis
Astronomical data analysis



K-means intuition:






Repeat the steps above to achieve the following result.


Continue until no further changes occur, indicating convergence.




K-means algorithm:


If a cluster has zero points, then k=k-1 clusters. or reinitialize k cluster centroids.


K-means Optimize the distortion cost function:



The distortion cost function should go down.

How to initialize k-means?



Always use multiple random initializations, as this significantly improves K-means' ability to minimize the distortion cost function and select better cluster centroids.


Choosing the number of clusters:

Elbow method is not frequently used





You can use K-means to determine t-shirt sizes by clustering data. Using three clusters may categorize sizes as small, medium, and large, while five clusters could define extra small, small, medium, large, and extra large. Both approaches are valid, but the choice depends on the trade-off between better fit with more sizes and higher manufacturing and shipping costs. A practical approach is to run K-means with K=3K  and K=5K , compare the results, and decide based on what best balances fit and cost for the business.

Comments

Post a Comment

Popular posts from this blog

Analysis of Repeated Measures Data using SAS (1)

Image
 1. Basic Concepts of Repeated Measures: In this basic setup of a completely randomized design with repeated measures, there are two factors, treatments and time. Treatment is called the between-subjects factor because levels of treatment can change only between subjects; all measurements on the same subject will represent the same treatment. Time is called a within-subjects factor because different measurements on the same subject are taken at different times.  In repeated measures experiments, interest centers on (1) how treatment means differ, (2) how treatment means change over time, and (3) how differences between treatment means change over time. 2. Four-step procedure for mixed model analysis: Step 1: Model the mean structure, usually by specification of the fixed effects. Step 2: Specify the covariance structure, between subjects as well as within subjects. Step 3: Fit the mean model accounting for the covariance structure. Step 4: Make statistical inference bas...
Read more

Medical information for Melanoma, Merkel cell carcinoma and tumor mutation burden

  Background: Melanoma Also called: malignant melanoma The most serious type of skin cancer. Melanoma occurs when the pigment-producing cells that give color to the skin become cancerous. Symptoms might include a new, unusual growth or a change in an existing mole. Melanomas can occur anywhere on the body. Treatment may involve surgery, radiation, medications, or in some cases chemotherapy. Merkel cell carcinoma  (MCC) it is a very rare ,  highly aggressive, neuroendocrine carcinoma of the skin ;  it has a worse survival probability compared with more common cutaneous malignancies, including metastatic melanoma, and has been termed the “most lethal” skin cancer  as   p atients with metastatic MCC have a historical 5-year overall survival (OS) rate of 18% What Is Tumor Mutational Burden? TMB is defined as the frequency of certain mutations within a tumor’s genes.  To be counted towards TMB, mutations have to alter the protein that is made from a gene. ...
Read more

Four essential statistical functions for simulation in SAS

  PDF function : This function is the probability density function. It returns the probability density at a given point for a variety of distributions. (For discrete distribution, the PDF function evaluates the probability mass function.) CDF function : This function is the cumulative distribution function. The CDF returns the probability that an observation from the specified distribution is less than or equal to a particular value. For continuous distributions, this is the area under the PDF up to a certain point. QUANTILE function : This function is closely related to the CDF function, but solves an inverse problem. Given a probability, P, it returns the smallest value,  q , for which CDF( q ) is greater than or equal to P. RAND function : This function generates a random sample from a distribution. In SAS/IML software, use the  RANDGEN subroutine , which fills up an entire matrix at once.
Read more