Create Review Tasks for Labeled Images

Step 3 of 5 in Get Started with Team-Based Labeling

The Image Labeler app enables the project owner of the team to create a labeling project setup and distribute the images into label tasks before assigning them to label task owners. For more information about how to set up a team labeling project as a project owner, see Create Project for a Labeling Team and Publish Tasks. After the label task owners complete the assigned label tasks, they share the labeled images with the project owner. For more information about how to complete an assigned label task as a label task owner, see Label Images and Send for Review.

To ensure that the labels received from the distributed label tasks are correct, the project owner creates distributed review tasks for the review task owners. The review task owners are the team members who review the shared labeled images and provide feedback to the label task owners.

This example shows you the steps that a project owner must follow to:

• Open an existing project.

• Check the progress of shared label task images.

• Create review tasks for labeled images.

• Distribute the labeled images for review among the team members.

Open Existing Labeling Project

Open the Image Labeler app using one of these options:

• MATLAB^® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.

• In the MATLAB command prompt, enter

  imageLabeler

To open an existing team project, such as the one you created in the Create Project for a Labeling Team and Publish Tasks step, follow these steps:

1. In the Image Labeler app interface, select Open Project.

   

2. In the file explorer, navigate to the folder that contains your previously created project. Select the .prj file, and click Open.

Once you select and open the team labeling project, the app enables you to open the project in Project Owner or Task Owner mode. To access the project you created as a project owner, select Project Owner.

Check Status of Images Assigned for Labeling

In project owner mode, the Image Labeler app displays the published label tasks and the individual label task progress in the Tasks pane.

In Create Project for a Labeling Team and Publish Tasks, you created three label tasks. In Label Images and Send for Review step, the task owners labeled the assigned images. The Progress column indicates that all 50 images each of Task 1 and Task 3 have been labeled and shared for review. For Task 2, 44 images have been labeled and shared for review, while the remaining 6 images have not yet been labeled or the labeling is in progress.

Note

Progress is defined as C/N. For a label task, C is the number of labeled images, and N is the total number of images in the label task.

C varies over time in the project. For example:

• If the label task owner shares all the images in the label task for review, then C = N.

• If the project owner creates a review task for labeled images and the review task owner rejects two images and accepts the rest, the label task progress is (C-2)/N.

The Image Browser pane displays the thumbnails of all the images in the project. Using the Image Filters list, you can filter the images based on status. Set the filter to Images that can be assigned to a review task, and the Image Browser displays the 144 images that have been labeled and shared for review.

Note

Changing the Image Filters option updates the current project images with the most recently saved images and labels from other project contributors and re-saves the project.

The symbol on the top right corner of each image thumbnail indicates the status of the image. You can access descriptions for the thumbnail badges by selecting Image Status Legend from the app toolstrip. This table shows the image labeling progress for each image based on the displayed badge.

Image Thumbnail	Image Status	Description
	Image pending in an unpublished label task	Image is assigned to a label task that is not yet published.
	Image locked by a label task owner	Image is assigned to a label task, and labeling is in progress.
	Image labeled and ready for review	Image labeling is complete, and the image is ready to be assigned to a review task.
	Image pending in an unpublished review task	Labeled image is assigned to a review task that is not yet published.

Create Review Tasks and Assign to Task Owners

To ensure labeling accuracy in the labeled data shared by the label task owners, the project owner can assign the labeled data to review task owners for review. To distribute the shared labeled images for review, you must create one or more review tasks for the team members, add labeled images to each review task, and then assign the tasks to the individual reviewers.

Create Review Task

The project owner can assign only images with a Image labeled and ready for review status to a review task. Image labeled and ready for review images are the images that are marked as labeled and shared for review by the label task owner.

1. In the Image Browser pane, set Image Filters to Images that can be assigned to a review task. The Image Browser displays only the Image labeled and ready for review images.

   

2. On the app toolstrip, select Add Review Task. The app adds a new task to the Tasks table, of Type Review.

3. In the Task Information pane, set Name to Task 4.

4. Set Owner to Team Member D.

5. In the Instruction field, enter Review labeled data and modify incorrect labels.

6. Add the first 50 images in the Image Browser pane to this task.

   1. Select the first image in the Image Browser.

   2. In the Task Information pane, in the Select Images field, enter 50 and press Enter. This selects the first 50 images sequentially, starting from the selected image. The Image Browser highlights all selected images in blue.

   3. Click Add Images To List to add the selected images to the task. Once added, the image names show up as entries in the Images List. In the Image Browser, the thumbnail for each added image now has an image status legend, Image pending in an unpublished review task, in its corner, indicating that it is associated with a review task.

   

7. You cannot assign the same image to two label tasks or two review tasks simultaneously. Each image can be part of only one label task and only one review task.

8. Repeat steps 1–5 to create a new task named Task 5 for Team Member E, adding the next 50 labeled images in the project.

9. Use the same process to create a new review task named Task 6 for Team Member F, and add the remaining 44 labeled images.

The Tasks pane now lists Task 4, Task 5, and Task 6 with and their assigned task owners along with the previously published label tasks, Task 1, Task 2, and Task 3.

Publish Review Tasks and Share Project with Review Task Owners

Publish Review Tasks

Publishing the tasks saves the project in a state that enables individual review task owners to open the project and start working on their assigned review tasks.

• For this example, publish all the unpublished review tasks at once. To do so, on the app toolstrip, select Publish Tasks > All Unpublished Tasks.

• Alternatively, to publish a single review task, in the Tasks pane, select the task you want to publish. Then, on the app toolstrip, select Publish Tasks > Selected Task.

  

Share Project with Review Task Owners

Save the project after publishing the review tasks. Then, on the app toolstrip, select Publish Tasks > Share Instructions to open a dialog box with a shareable link to the project location. Ensure that the review task owners have read/write permissions for the shared location, and then share the link with the task owners so they can start working on their review tasks.

Note

Distribute the shareable link to the task owners through email, or as a message outside the Image Labeler app.

Create Executable for Review Task Owners Without Full MATLAB License

For review task owners who do not have access to a full MATLAB^® license, you can create a compiled version of the review project in the form of a standalone executable using these steps:

1. On the Image Labeler tab of the app toolstrip, click Compile.

2. Choose the app options for compilation based on the task. For this example, set App Type to Review as you are creating this executable for review task owners.

3. Specify the App Name, Location and include any automation algorithms necessary for labeling.

4. Click Start Packaging. This creates an executable that you can share with the review task owners.

Review task owners can use the executable to install the standalone app, open the app, and then complete the label tasks.