How do I Input Videos into the Deep Network Designer

I have a Deep Learning project for a class where I have to create a model that inputs 1.5 second videos (.mp4 files) of a baseball pitcher throwing pitches and it classifies wheather the pitch thrown is a fastball or a breaking ball. I have data stored in a folder with fast and break subfolders of those .mp4 files. Does anyone have any tips on where to start in creating the model mainly in starting to get a training session by inputiing these video files. Any help would be appretiated!

Answer 1

[Pratyush]

Hi Roman,

I understand that you want to build a Deep Learning Model to classify the pitches into fastball or breaking ball.

You can use the following steps to create a model for classifying the pitches:

1. Data Preprocessing: Load the video files using "VideoReader" to read the video files into MATLAB. You can use "dir" to get a list of files in the fast and break subfolders. You can label the videos as fast or break based on the folder they are in. You can use "imageDataStore" for this purpose.
2. Feature Extraction: Preprocess the videos to a fixed size and format. You can use "imresize" to resize the frames, and "rgb2gray" to convert them to grayscale if needed. Extract relevant features from the video frames. You can use techniques like optical flow, frame differencing, or extracting statistical features from the frames.
3. Model Building: Consider using a pre-trained model for feature extraction. You can use a pre-trained CNN model like ResNet or VGG and remove the fully connected layers to extract features. Build a model architecture suitable for video classification. You might consider using 3D convolutional neural networks (3D CNNs) which can handle spatiotemporal data.
4. Training: Use data augmentation techniques to increase the diversity of your training data. MATLAB provides functions like "augmentedImageDatastore" for this purpose. Train your model using the preprocessed videos. You can use "trainNetwork" function to train your model.
5. Evaluation: Evaluate the model's performance using a separate validation set. You can use the "classify" function to classify new videos and compare the results with ground truth labels.

These are the general steps to get you started with your project. Hope it helps.

[Roman Skorupa]

I've been attempting to load in .mp4 files into the Deep Network Designer by using filedatastore and Importing via Custom Data. However, I am running into the following error.

Im not sure if this is due to errors in my code loading the files, or if it is another project oversight. I am using my school's supercomputer for this project and it may not have support for MP4. I have tried converting all the data to .avi file extensions, but then I get this following error when importing the data. It seems like it won't read into a 4D Array and leave it all empty when I attempt to store the data that way.

I'll attach the code I'm currently running. This attempts to load .mp4 files into a datastor and results in the first error I have attatched.

Do you see if there is anything I am missing in this solution?

% Define the base folder containing subfolders for fastballs and breaking balls
baseFolder = '/home/skorupar/CSC4651/Applied Project/Data/Trial 1';

% Create a file datastore with custom ReadFcn
fds = fileDatastore(baseFolder, 'ReadFcn', @customReadFunction, 'FileExtensions', '.mp4', 'IncludeSubfolders', true);

% Display the first few rows of the datastore
disp(fds.Files);

% Define the customReadFunction

function data = customReadFunction(filename)
    % Read video
    v = VideoReader(filename);
    videoData = read(v);
    
    % Extract class label from file path
    % Assuming one level of subfolders for class labels
    [path, ~, ~] = fileparts(filename);
    [~, classLabel] = fileparts(path);
    
    % Convert classLabel to categorical
    classLabel = categorical(cellstr(classLabel));
 
    % Structured output as a cell array with two columns
    data = {videoData, classLabel};
end