Mark The Disease

In this article, we will learn how to predict the disorder of knee using X-rays of knee and do classification using Convolutional neural network and then deploy the model using Web-based Flask application.

OSTEOARTHRITIS DETECTION IN KNEE USING CNN

Arthritis is one of the disorder that causes swelling, stiffness, tenderness and disability etc. It may be happen at one or more joints of the people. Arthritis is more common in older people and typically worsens with age. It is gradually increases in globally day by day. More than 350 million people have arthritis globally, this accounts for almost 4.43% of the human population. Although as of now there is no known cure for arthritis, the benefits of early detection can’t be understated.

Sample of Normal and Osteoarthritis Knee

Idea to Make This application

The idea is to build an application that will help the patients or doctors to detect the arthritis of the joints. Along with helping with early detection, this web application also will detects the severity of the disorder and by which they will do precaution and cure about the arthritis as soon as possible.

Datasets of Knee x-ray Images

The data used for the experiments and analysis in this study are downloaded from site of Mendeley :-https://data.mendeley.com/datasets/t9ndx37v5h/1

The dataset consists of 3300 digital X-ray images of knee joint which are collected from Mendeley. Original images are 8-bit grayscale image. Each radiographic knee X-ray image is manually annotated /labelled as per Kellgren and Lawrence grades by 2 medical experts. A novel approach has been developed to automatically extract the Cartilage region (region of interest) based on density of pixels. The target is to evaluate the performance of the deep learning algorithm to predict per Kellgren and Lawrence grades.

The Kellgren and Lawrence classification system:

grade 0 (normal): definite absence of x-ray changes of osteoarthritis. In total there are 3300 knee images and the distribution as per the KL grades is in normal case:- 1017 X-ray images.

grade 1 (doubtful): doubtful joint space narrowing and possible osteophytes lipping. In total there are 3300 knee images and the distribution as per the KL grades is in doubtful case:- 965 X-ray images.

grade2 (mild): definite osteophytes and possible joint space narrowing. In total there are 3300 knee images and the distribution as per the KL grades is in mild case:- 464 X-ray images.

grade 3 (moderate): moderate multiple osteophytes, definite narrowing of joint space and some sclerosis and possible deformity of bone ends. In total there are 3300 knee images and the distribution as per the KL grades is in moderate case:- 442 X-ray images.

grade 4 (severe): large osteophytes, marked narrowing of joint space, severe sclerosis and definite deformity of the bone end. In total there are 3300 knee images and the distribution as per the KL grades is in severe case:- 412 X-ray images.

The KL grading system to assess the severity of knee OA.

Platform Used to Preprocessing and Train The Model

Google Colab

Colaboratory or “Colab” for short, is a product from Google Research. Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning, data analysis and education.

Methods and Classification

This section introduces the methodology used for quantifying radiographic knee OA severity. This involves two steps: automatically detecting knee joints using a fully convolutional network (FCN), and simultaneous classification and regression of localized knee images using a convolutional neural network (CNN).

Classification is a method to extract information from data sets. This is done by dividing the data into categories based on some features. The idea is to derive a model which can perform the sorting process by training it on data objects where the category, or label, is known. The model should then be able to classify unlabeled data with sufficient accuracy. There are many different models that are used for classification, e.g. neural networks.

Library Used

Tensorflow library : Used for fast numerical computing created and released by Google. It is a foundation library that can be used to create Deep Learning models directly or by using wrapper libraries that simplify the process built on top of TensorFlow.

Numpy library :NumPy is the fundamental package for scientific computing in Python. NumPy arrays facilitate advanced mathematical and other types of operations on large numbers of data. Typically, such operations are executed more efficiently and with less code than is possible using Python’s built-in sequences.

OpenCV library : OpenCV is a cross-platform library using which we can develop real-time computer vision applications. It mainly focuses on image processing, video capture and analysis including features like face detection and object detection.

CNN Architecture: CNNs are a class of Deep Neural Networks that can recognize and classify particular features from images and are widely used for analyzing visual images. Their applications range from image and video recognition, image classification, medical image analysis, computer vision and natural language processing. The term ‘Convolution” in CNN denotes the mathematical function of convolution which is a special kind of linear operation wherein two functions are multiplied to produce a third function which expresses how the shape of one function is modified by the other. In simple terms, two images which can be represented as matrices are multiplied to give an output that is used to extract features from the image.

There are two main parts to a CNN architecture

• A convolution tool that separates and identifies the various features of the image for analysis in a process called as Feature Extraction.
• A fully connected layer that utilizes the output from the convolution process and predicts the class of the image based on the features extracted in previous stages.

Convolutional Neural Network Architecture

There are three types of layers that make up the CNN which are the convolutional layers, pooling layers, and fully-connected (FC) layers. When these layers are stacked, a CNN architecture will be formed. In addition to these three layers, there are two more important parameters which are the dropout layer and the activation function.

1. Convolutional Layer :- This layer is the first layer that is used to extract the various features from the input images.
2. Pooling Layer :- In most cases, a Convolutional Layer is followed by a Pooling Layer. The primary aim of this layer is to decrease the size of the convolved feature map to reduce the computational costs.
3. Fully Connected Layer :- The Fully Connected (FC) layer consists of the weights and biases along with the neurons and is used to connect the neurons between two different layers. These layers are usually placed before the output layer and form the last few layers of a CNN Architecture.
4. Dropout :- Usually, when all the features are connected to the FC layer, it can cause overfitting in the training dataset. To overcome this problem, a dropout layer is utilised wherein a few neurons are dropped from the neural network during training process resulting in reduced size of the model.
5. Activation Functions :- Finally, one of the most important parameters of the CNN model is the activation function. They are used to learn and approximate any kind of continuous and complex relationship between variables of the network. In simple words, it decides which information of the model should fire in the forward direction and which ones should not at the end of the network.

# Some activation functions are :-

sigmoid/Logistic Activation Function.

ReLU Function

Tanh Function(hyperbolic Tangent)

Preprocessing

Data preprocessing is a process of preparing the raw data and making it suitable for a machine learning model. It is the first and crucial step while creating a machine learning model.

A real-world data generally contains noises, missing values, and maybe in an unusable format which cannot be directly used for machine learning models. Data preprocessing is required tasks for cleaning the data and making it suitable for a machine learning model which also increases the accuracy and efficiency of a machine learning model.

data_path=’/content/drive/MyDrive/Knee-project/Knee-Dataset/’
categories=os.listdir(data_path)
labels=[i for i in range(len(categories))]
label_dict=dict(zip(categories,labels)) #empty dictionary
print(label_dict)
print(categories)
print(labels)

• data_path − This is the directory, which needs to be explored.
• os.listdir() method in python is used to get the list of all files and directories in the specified directory.
• for i in range() iterates through numbers. for element in my_list iterates through whatever is in the list (they could be numbers, too) for i in range(len(a_list)) iterates through numbers (which can be used for index access of a_list ).
• When using LabelEncoder to encode categorical variables into numerics,
• how does one keep a dictionary in which the transformation is tracked?
• i.e. a dictionary in which I can see which values became :- {'A':1,'B':2,'C':3}

The input knee X-ray image is initially checked for the proper quality and then it is smoothened with anisotropic filter. All the input images are resized into 256 × 256 for the standardization of all the four categories of KOA.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
resized=cv2.resize(gray,(img_size,img_size))

cv2.cvtColor(cv2.COLOR_BGR2GRAY) method is used to convert an color image to grayscale image. we convert RGB image to grayscale in image processing because to store a single color pixel of an RGB color image we will need 8*3 = 24 bits (8 bit for each color component), but when we convert an RGB image to grayscale image, only 8 bit is required to store a single pixel of the image.

resizing the image into 256 x 256, since we need a fixed common size for all the images in the dataset.

CNN MODEL

In machine learning, a convolutional neural network (ConvNets or CNNs) is one of the main categories to do images recognition, images classifications, Objects detections, recognition faces etc. are some of the areas where CNNs are widely used. In this project, the classification model is also based on CNN for image classification.

The model type that we will be using is Sequential. Sequential is the easiest way to build a model in tensorflow. It allows you to build a model layer by layer.

We use the ‘add()’ function to add layers to our model.

Our first 2 layers are Conv2D layers. These are convolution layers that will deal with our input images, which are seen as 2-dimensional matrices.

Defining Our Neural Network

model=Sequential()
model.add(Conv2D(128,(3,3),input_shape=data.shape[1:]))
model.add(Activation(‘relu’))
model.add(MaxPooling2D(pool_size=(2,2)))
#The first CNN layer followed by Relu and MaxPooling layers
model.add(Conv2D(64,(3,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
#The second convolution layer followed by Relu and MaxPooling layers
model.add(Conv2D(32,(3,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
#The thrid convolution layer followed by Relu and MaxPooling layers
model.add(Flatten())
#Flatten layer to stack the output convolutions from 3rd convolution layer
model.add(Dropout(0.2))
model.add(Dense(128,activation='relu'))
#Dense layer of 128 neurons
model.add(Dropout(0.1))
model.add(Dense(64,activation='relu'))
#Dense layer of 64 neurons
model.add(Dense(5,activation='softmax'))
#The Final layer with two outputs for two categories

We will now define our neural network model, which as mentioned will use TensorFlow as the model’s backend.

The structure of our convolutional neural network architecture is defined below:

1. The first hidden layer Conv2D is a convolutional layer that has 128 feature maps, each with a size of 3 x 3 and we are using a rectified linear activation function - relu
2. We then add another convolutional layer with 64 feature maps, each with a size of 3 x 3 and we are using a rectified linear activation function — relu
3. We add a third convolutional layer with 32 feature maps, each with a size of 3 x 3 and we are using a rectified linear activation function — relu
4. We then add a pooling layer MaxPooling2D1 that is configured with a pool size of 2 x 2. Max pooling operation for 2D spatial data. Down samples the input along its spatial dimensions (height and width) by taking the maximum value over an input window (of size defined by pool_size ) for each channel of the input.
5. We then convert the 2-dimensional matrix into a vector using Flatten - this allows our output to be processed by fully connected layers. Flattening is used to convert all the resultant 2-Dimensional arrays from pooled feature maps into a single long continuous linear vector. The flattened matrix is fed as input to the fully connected layer to classify the image.
6. We then apply a regularization layer using Dropout that is set to randomly exclude 20% of the neurons in the layer - this is used to reduce overfitting.
7. Next, we add a fully connected layer that has 128 neurons and a ReLU activation function.
8. We will then add another regularization layer to reduce overfitting, this time we’re randomly excluding 10% of the neurons.
9. Next, we add a fully connected layer that has 64 neurons and a ReLU activation function.
10. We finish the neural network with an output layer that has 5 neurons — the same as the number of classes in our classification problem and a softmax activation function. This will output a prediction of the probability that a digit belongs to each class.

Compiling the Model

Before training the model we need to compile it and it define the loss function, optimizers, and metrics for prediction.

model.compile(loss=’categorical_crossentropy’,optimizer=’adam’,metrics=[‘accuracy’])

Loss Function:- The loss function is a method of evaluating how well your machine learning algorithm models your featured data set. In other words, loss functions are a measurement of how good your model is in terms of predicting the expected outcome.

Loss functions can be grouped into two major categories

1. classification
2. regression

In classification, we are trying to predict output from set of finite categorical values i.e Given large data set of images of hand written digits, categorizing them into one of 0–9 digits.

Regression, on the other hand, deals with predicting a continuous value for example given floor area, number of rooms, size of rooms, predict the price of room.

Here, we use Cross Entropy Loss this is the most common setting for classification problems. Cross-entropy loss increases as the predicted probability diverges from the actual label.

Optimizers:- Optimizers are algorithms or methods used to change the attributes of your neural network such as weights and learning rate in order to reduce the losses.

There are different optimizers used in building a deep learning model.

1. Gradient Descent
2. Stochastic Gradient Descent
3. Adam
4. Mini-Batch Gradient Descent
5. Adagrad

Here, we use Adam optimizer it is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.

Metrics:- Metrics are used to monitor and measure the performance of a model (during training and testing), and don’t need to be differentiable.

There are generally two types of Metrics

Regression Metrics

Regression models have continuous output. So, we need a metric based on calculating some sort of distance between predicted and ground truth.

In order to evaluate Regression models, these metrics are used:

• Mean Absolute Error (MAE),
• Mean Squared Error (MSE),
• Root Mean Squared Error (RMSE),
• R² (R-Squared).

Classification Metrics

Classification models have discrete output, so we need a metric that compares discrete classes in some form. Classification Metrics evaluate a model’s performance and tell how good or bad the classification is, but each of them evaluates it in a different way.

In order to evaluate Classification models, these metrics are used:

• Accuracy
• Confusion Matrix (not a metric but fundamental to others)
• Precision and Recall
• F1-score
• AU-ROC

Here, we use classification matrix accuracy because Classification accuracy is perhaps the simplest metric to use and implement and is defined as the number of correct predictions divided by the total number of predictions, multiplied by 100.

Model Summary

Now, we can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as we go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, we can afford (computationally) to add more output channels in each Conv2D layer.

Convolution and pooling layer(Hidden Layer)

Add Dense layers on top

To complete the model, we will feed the last output tensor from the convolutional base (of shape (60, 60, 32)) into one or more Dense layers to perform classification. Dense layers take vectors as input (which are 1D), while the current output is a 3D tensor. First, we will flatten (or unroll) the 3D output to 1D, then add one or more Dense layers on top. Arthritis classification has 5 output classes, so you use a final Dense layer with 5 outputs.

Classification (Flatten, Fully connected layer and Dense layer)

Splitting The Data into Training and Testing

Train test split is a model validation procedure that allows you to simulate how a model would perform on new/unseen data. procedure to works of train test split:

Train test split procedure

1. Arrange the data

Make sure our data is arranged into a format acceptable for train test split. In scikit-learn, this consists of separating our full data set into “Features” and “Target.”

from sklearn.model_selection import train_test_split

2. Split the data

Split the data set into two pieces — a training set and a testing set. This consists of random sampling without replacement about 90 percent of the rows and putting them into training set. The remaining 10 percent is put into test set. Note that the colors in “Features” and “Target” indicate where their data will go (“X_train,” “X_test,” “y_train,” “y_test”) for a particular train test split.

x_train,x_test,y_train,y_test=train_test_split(data,new_label,test_size=0.1)

3. Train the Model

Train the model on the training set. This is “X_train” and “y_train” in the image.

4. Test the model

Test the model on the testing set (“X_test” and “y_test” in the image) and evaluate the performance.

Fit the Model

Model fitting is a measure of how well a machine learning model generalizes to similar data to that on which it was trained. A model that is well-fitted produces more accurate outcomes.

history=model.fit(x_train,y_train,epochs=10,validation_split=0.2)

We will fit the model using 10 epochs.

epochs mean how many times we go through your training set. The model is updated each time a batch is processed, which means that it can be updated multiple times during one epoch.

Accuracy in training and testing Phase

validation_split:- Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch.

We achieve 70% test accuracy with this simple CNN. That’s a really pretty good accuracy.

Evaluating The Model

Finally, we will evaluate our model and print the accuracy:

Visualization of loss and accuracy

As we can see in the above diagram, the loss on the training set decreases rapidly for the further epochs. For the validate set, the loss does not decrease at the same rate as the training set, but remains almost flat for multiple epochs. This means our model is generalizing well to unseen data.

And we can see in the above diagram, the accuracy increases rapidly in the further epochs, indicating that the network is learning fast. Afterwards, the curve flattens indicating that not too many epochs are required to train the model further. Generally, if the training data accuracy (“acc”) keeps improving while the validation data accuracy (“val_acc”) gets worse, we are encountering overfitting. It indicates that the model is starting to memorize the data.

Confusion Matrix of Results

A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class.

DEPLOYMENT OF THE MODEL

Machine learning model deployment is the process of placing a finished machine learning model into a live environment where it can be used for its intended purpose.

Models can be deployed in a wide range of environments, and they are often integrated with apps through an API or on the web so they can be accessed by end users.

Visual Studio Code

Visual Studio Code (famously known as VS Code) is a free open source text editor by Microsoft. VS Code is available for Windows, Linux, and macOS. Although the editor is relatively lightweight, it includes some powerful features that have made VS Code one of the most popular web development environment tools in recent times.

VS Code supports a wide array of programming languages from Java, C++, and Python to CSS. Moreover, VS Code allows us to add on and even creating new extensions including code linters, debuggers, and cloud and web development support.

The VS Code user interface allows for a lot of interaction compared to other text editors.

Flask Framework

Flask is a Python-based micro framework used for developing small-scale websites. Flask is very easy to make Restful APIs using python. As of now, we have developed a model i.e model.h1 , which can predict a class of the data based on various attributes of the data. The class label is Normal, Doubtful, Mild, Moderate, Severe.

Now we will design a web application where the user will input all the attribute values and the data will be given to the model, based on the training given to the model, the model will predict which type of disorder is happen in person knee i.e Normal, Doubtful, Mild, Moderate, Severe. whose details have been fed.

Flask script:- We need to download flask and some other libraries. Here, we make use of a virtual environment, where all the libraries are managed which makes both the development and deployment job easier.

mkdir app.py
cd app.py
python3 -m venv venv
source venv/bin/activate

Then install Flask.

pip install flask

Then create folder templates. In our application, we will use templates to render HTML which will display in the user’s browser. This folder contains our HTML form file index.html.

mkdir templates

HTML

HTML (HyperText Markup Language) is the code that is used to structure a web page and its content.

For example, content could be structured within a set of paragraphs, a list of bulleted points, or using images and data tables.

Note:- In order to predict the data correctly, the corresponding values of each label should match the value of each input selected. For example — In the attribute Relationship, there are 5 categorical values. These are converted to numerical like this {0 : ‘Normal’, 1 : ‘Doubtful’, 2 : ‘Mild’, 3 : ‘Moderate’, 4 : ‘Severe’}. Therefore we need to put the same values to the HTML form.

Then create folder static. Static files in Flask have a special route. All application URLs that begin with “/static”, by convention, are served from a folder located at “/static” inside our application’s root folder.

This means if we create a “/static” folder inside our main “serving_static” folder, we will be able to serve static files such as CSS, JS, images, and other assets by simply placing them inside that “/static” folder.

CSS

Cascading Style Sheet(CSS) is used to set the style in web pages that contain HTML elements. It sets the background color, font-size, font-family, color etc. property of elements on a web page.

JavaScript

JavaScript is a dynamic computer programming language. It is lightweight and most commonly used as a part of web pages, whose implementations allow client-side script to interact with the user and make dynamic pages. It is an interpreted programming language with object-oriented capabilities.

Display Script in the Browser

Create script.py file in the venv folder and copy the following code.
Here we import the libraries, then using app=Flask(__name__) we create an instance of flask. @app.route(‘/’) is used to tell flask what URL should trigger the function index() and in the function index, we use render_template(‘index.html’) to display the script index.html in the browser.

Now, Let’s run the application.

set FLASK_APP=app.py      # this it the code  for windows.
run flask

This should run the application and launch a simple server. Open http://127.0.0.1:5000/ to see the html form.

Prediction of Knee Osteoarthritis Results:-

When someone select and submit the Knee X-ray images, the webpage should display the predicted value of osteoarthritis i.e Normal, Doubtful, Mild, Moderate, Severe. For this, we require the model file (model.h1) we created before in the same project folder.

Here, after the selecting Knee X-ray images, the form values are stored in the variable to_predict_list in the form of a dictionary. We convert it into a list of the dictionary’s values and pass it as an argument to ValuePredictor() function. In this function, we load the model.h1 file and predict the new values and return the result.

This result/prediction (osteoarthritis results) is then passed as an argument to the template engine with the HTML page to be displayed.

Create the following result.html file and add it to the templates folder.

<!doctype html>
<html>
<body>
 <h1> {{ prediction of Knee Osteoarthritis }}</h1>
</body>
</html>

Output Shown before the Prediction:

Output Shown after the Prediction:-

Here, we can see that our model predict and gives result the Diagnosis of Knee status:- Moderate.

Conclusion

It was very interesting and adopting learning kind of experience to work with medical images in this project as it is playing a very big role in the medical field. This project took us through the various phases of project development and gave us the real insight into the world of machine learning techniques.

Due to this project we came to know how different machine learning techniques and methods can be designed in order to achieve a state of art solution to the problems like image classification and object detection.

In this project we proposed methods to automatically localize knee joints using a fully convolutional network and quantified knee OA severity through a network jointly trained from scratch.

Our model achieve 70% test accuracy and test loss is 77% during model training.

Finally, we deploy the model using Flask and design a web application where the user will give input of Knee X-ray images then the model will predict which type of disorder is happen in person knee.

Further Work

For the improvement on the accuracy of Model we will work on more dataset.

Add some precautions for the type of severity.

Improve the outlook of webpage.

!!!!!! _________________Thank You_________________!!!!!!