Due date: Nov 4

Part 1: Smartphone-Based Heart Rate Monitoring

In this assignment you will learn how to measure heart rate using the smartphone camera.

Checkpoints and recommended steps:

  1. Open the default camera app on your phone. Turn to video mode. Place the camera on the table so the camera is dark, and you will now be able to turn on the flash/torch. Place your finger over the camera and record a video for 30 seconds. Here is what this would look like on Android and iOS:

  1. In MATLAB/Python, loop through each frame, convert to grayscale using rgb2gray (MATLAB), and compute the mean color value for the frame.
  2. Next we use the algorithm from [4] in step 1 to 3 as a rough guide. Subtract the DC value from the signal and apply a highpass filter to get a signal like below:

  1. Apply FFT to the filtered waveform. Find the bin range which corresponds to a normal heart rate of 60-100bpm and find the bin with the maximum amplitude. This provides an estimate of the heart rate.

  1. Use peak finding with findpeaks to select all the peaks associated with a heartbeat. You can use the MinPeakDistance heuristic to impose a restriction on the minimum number of samples between peaks which can be set to the estimated heart rate computed from the previous step. How much would that be in terms of samples? The camera sampling rate would normally be 30 Hz.

  1. Bonus: How does the signal look without flash? How does it vary based on the amount of pressure your finger places on the camera [5].

Deliverables: Plot of FFT with estimated heart rate in Hz. Plot of time-domain of the heart rate signal with selected peaks corresponding to heartbeats, with title indicating the computed heart rate in bpm. Code used for the assignment.

Part 2: Finetuning vision models for medical image diagnostics

In this assignment you will learn how to finetune a vision foundation model to a real-world medical dataset.

Checkpoints and recommended steps:

  1. Select a pretrained vision foundation model of your choice. Keras offers a list of options here which you can try. Consider trying InceptionV3 which is one of the first ImageNet vision models that was finetuned for medical image diagnostics [3].
  2. Select a medical image dataset of your choice, here are some examples (you can choose others outside from this list)
    1. Google Health Foundation models
    2. Fundus images (Kaggle)
    3. Medical MNIST (Kaggle)
    4. ISIC dataset
    5. Chest Xrays