Utility of deep learning in radiology

Deep learning has been actively applied to radiology for several reasons [1]. First, deep learning is a powerful tool for image recognition, and radiology begins by recognizing abnormal regions in the image. Secondly, the number of imaging examinations increases rapidly, but there are not enough radiologists to interpret the examinations. Thus, there is a need for a technique that can efficiently improve the radiologist workflow. Lastly, radiology is technology friendly. Over the past 20 years, radiology has grown with technological advancement and thus is accustomed to accepting new technologies.

Deep learning has superior to existing image processing algorithms in image recognition. Traditional image processing algorithms, called computer aided diagnosis (CAD) have been used in clinical practice for the last 20 years, but most of them failed to give a good impression to the radiologists. However, deep Learning is expected to provide a much better user experience than traditional CADs. Deep learning learns important features without the help of experts, and this learning process is known to be a major factor in achieving better performance with existing methods. In the medical imaging, various studies have already proved the superiority of deep learning in the last three years [2].

Examples beyond present value

Although most applications of deep learning are related to disease classification and segmentation, deep learning can create new value beyond helping physicians to interpret. Here, I will briefly review several recent studies that demonstrated the potential capability of deep learning beyond disease diagnosis.

First, deep learning can optimize radiology workflow. Mohammad et al. demonstrated that deep learning could automatically analyze computed tomography (CT), prioritize radiology worklists and reduce time to diagnosis of intracranial hemorrhage (ICH) [3]. ICH is a life-threatening disease that requires immediate diagnosis. The deep learning model  re-prioritize “routine” head CT studies as “stat” on realtime radiology worklists if an ICH was detected (Fig. 1). The model was validated prospectively in clinical practice for 3 months. Of 347 head CT scans, 5 new cases of ICH were identified and median time to diagnosis was significantly reduced from 512 to 19 min. In other worlds, these 5 patients could be treated quickly with the help of deep learning. Like this, deep learning can improve the quality of medical care by increasing the efficiency of workflow beyond the improvement of diagnostic accuracy.

스크린샷 2018-05-20 오후 6.01.20

Fig.1. Clinical implementation

Second, risk assessment is one of the tasks where the efficacy of deep learning is most expected. Google AI  research demonstrated that deep learning could predict cardiovascular problems from retinal images [4]. Using deep learning models trained on data from 284,335 patients, it was demonstrated that the known cardiovascular risk factors (e.g., age, gender, smoking status, etc) could be predicted. These risk factors can be obtained from patients, so predicting them is not the final goal.  The results support the hypothesis that deep learning can extract the relationship between retinal images and the risk factors. Even though doctors have risk factor information, it is difficult to provide the final risk score. However, deep learning can extract features associated with cardiovascular risk from a large number of images, and predict the final risk score.

Another example is breast cancer risk assessment in mammography using deep learning. The primary goal of screening mammography is the detection of breast cancer, but density assessment and notification is also important. Since breast parenchymal density  is associated with breast cancer, breast density is routinely evaluated quantitatively or qualitatively. However, some researchers believed that breast parenchymal texture might have more meaningful information beyond density. Recently, Hui et al.  demonstrated that deep learning (Fig. 2) distinguish between the BRCA1/2 gene-mutation carriers and low-risk women and between unilateral cancer patients and low-risk women, respectively [5]. Deep learning can extract parenchymal characteristics which are relevant to the task of distinguishing between cancer risk populations. Such deep learning application can be used to aid radiologists in cancer risk assessment from mammography.


Fig. 2. Deep learning model for cancer risk assessment in mammography

Finally, deep learning is applicable to outcome prediction from histologic images. The interpretation of digitized tissue samples is very complicated, so visual assessment is limited. Dmitrii et al. demonstrated that deep learning with only small tissue areas (hazard ratio 2.3; CI 95% 1.79–3.03; AUC 0.69) outperforms visual histological assessment performed by human experts (HR 1.67; CI 95% 1.28–2.19; AUC 0.58) in outcome prediction [6]. The features were extracted with convolutional neural network (CNN) from images of tissue microarray (TMA) and Long Short-Term Memory (LSTM) Network predict the patient risk outcome using the extracted features (Fig. 3).  Although further studies for generalizing the prognostic performance are necessary, it is impressive that the possibility of cancer outcome prediction based on deep learning was proved.


Fig. 3. Deep learning for outcome prediction from tissue samples

There is room for improvement in the number of data and algorithms of the above-mentioned studies. However, the value of these studies is not a numerical value of accuracy or how well validated. These studies are of great value in their effort to create new value by applying deep learning in various tasks. Further, it is expected that many more innovative applications will be introduced in the future.

How to discover new value

The value can vary greatly depending on the nature of the problems being addressed by deep learning. Most applications of deep learning are still at the research level and are aimed at better performing the tasks doctors do. If the role of deep learning is limited to help doctors work, creating new markets is difficult. In other words, improving the quality of existing practice may not provide motivation to pay for new technologies. On the other hand, deep learning, which does things that doctors cannot do is likely to make profits. That is why the company should challenge discover new value rather than competing with doctors to do better what they do.

The most important thing to create new value is a problem setting that takes into account both the medical needs and the applicability of deep learning. In this process, researchers must communicate with doctors and listen to various opinions. Also, physicians should be interested in deep learning and make efforts to develop and use it to create new medical value. Therefore, the seamless collaboration between the physicians and deep learning researchers is the basis for generating a deep learning model that will create new value.


[1] www.forbes.com/sites/forbestechcouncil/2018/04/12/which-area-of-medicine-is-most-ripe-for-ai-disruption/

[2] A survey on deep learning in medical image analysis, Medical Image Analysis, 2017, doi:10.1016/j.media.2017.07.005

[3] Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration, npj Digital Medicine, 2018, doi:10.1038/s41746-017-0015-z

[4] Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning, Nature Biomedical Engineering, 2018, doi:10.1038/s41551-018-0195-0

[5] Deep learning in breast cancer risk assessment: evaluation of fine-tuned convolutional neural networks on a clinical dataset of FFDMs, Proc. SPIE 10575, Medical Imaging 2018: Computer-Aided Diagnosis, doi: 10.1117/12.2294536

[6] Deep learning based tissue analysis predicts outcome in colorectal cancer, Scientific Reports, 2018, doi:10.1038/s41598-018-21758-3

Posted by:Ki Hwan Kim

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s