Isotonic Regression : Another Level of Regression Method

Introduction

Regression analysis is a statistic method which investigates the relationship between a dependent (target) and independent variable (s) (predictor). Independent variables can be one or more, for one independent variable we called Simple Regression, for more than one independent variable we called Multiple Regression. For example, relationship between height and weight in American women aged 30–39 years is best studied through regression.

Linear Regression

The linear regression model is used to determine the relationship between the dependent and independent variables. It assumes a linear relationship, represented by the best fit line, between the two variables.

The equation of linear regression model :

Isotonic Regression

Isotonic regression is a regression technique in which the predictor variable is monotonically related to the target variable. This means that as the value of the predictor variable increases, the value of the target variable either increases or decreases in a consistent, non-oscillating manner. (like optimization problem in mathematic)

Isotonic Regression for Probability Calibration

Isotonic regression can be used to adjust the predicted probabilities produced by a classifier so that they are more accurately calibrated to the true probabilities to minimize LogLoss Error. We can use isotonic regression to calibrate the probabilities in a postprocessing step. As isotonic regression is overconfident and predict 0.0 or 1.0 when it shouldn’t. For example :

and

Isotonic regression has a number of applications, including:

· Ordinal regression: Isotonic regression can be used to model ordinal variables, which are variables that can be ranked in order

· Non-parametric regression: Because isotonic regression does not make any assumptions about the functional form of the relationship between the predictor and target variables, it can be used as a non-parametric regression method.

· Outlier detection: Isotonic regression can be used to identify outliers in a dataset by identifying points that are significantly different from the overall trend of the data.

Isotonic Regression Code

Here is an example of how to implement the IsotonicRegression with scikit-learn in Python :

Import library and create sample data

Import Isotonic Regression library and fit the model

Import Linear Regression library and fit the model

Import Polynomial Regression and fit the model

Compare Different Regression

Advantage

· It offers multidimension scaling which means we can analyse each and every dimension and then can interpolate in a function in a linear way.

· It does not constraint on any function such as linearity as the function is monotonically increasing.

Disadvantage

· While using isotonic regression there can be problem of over fitting because monotonic constraint and prediction value increases.

· Real life application and data related to same may not have increasing trend data as required by Isotonic used cases.

· Isotonic regression is monotonic increasing or monotonic decreasing. Thus, it would not be appropriate for fitting distributions that have left and right tails. Also, unlike B-splines, it does not fit derivatives, so it will not approximate smooth curves like most distribution functions.

References : https://www.geeksforgeeks.org/isotonic-regression-in-scikit-learn/