diff --git a/yolov5/utils/loggers/clearmi/.gitkeep b/yolov5/utils/loggers/clearmi/.gitkeep
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/yolov5/utils/loggers/clearmi/README.md b/yolov5/utils/loggers/clearmi/README.md
deleted file mode 100644
index ca41c040193c1d8817a870404af09871b511f7ed..0000000000000000000000000000000000000000
--- a/yolov5/utils/loggers/clearmi/README.md
+++ /dev/null
@@ -1,237 +0,0 @@
-# ClearML Integration
-
-<img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_dark.png#gh-light-mode-only" alt="Clear|ML"><img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_light.png#gh-dark-mode-only" alt="Clear|ML">
-
-## About ClearML
-
-[ClearML](https://cutt.ly/yolov5-tutorial-clearml) is an [open-source](https://github.com/allegroai/clearml) toolbox designed to save you time ⏱️.
-
-🔨 Track every YOLOv5 training run in the <b>experiment manager</b>
-
-🔧 Version and easily access your custom training data with the integrated ClearML <b>Data Versioning Tool</b>
-
-🔦 <b>Remotely train and monitor</b> your YOLOv5 training runs using ClearML Agent
-
-🔬 Get the very best mAP using ClearML <b>Hyperparameter Optimization</b>
-
-🔭 Turn your newly trained <b>YOLOv5 model into an API</b> with just a few commands using ClearML Serving
-
-<br />
-And so much more. It's up to you how many of these tools you want to use, you can stick to the experiment manager, or chain them all together into an impressive pipeline!
-<br />
-<br />
-
-
-
-<br />
-<br />
-
-## 🦾 Setting Things Up
-
-To keep track of your experiments and/or data, ClearML needs to communicate to a server. You have 2 options to get one:
-
-Either sign up for free to the [ClearML Hosted Service](https://cutt.ly/yolov5-tutorial-clearml) or you can set up your own server, see [here](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server). Even the server is open-source, so even if you're dealing with sensitive data, you should be good to go!
-
-1. Install the `clearml` python package:
-
- ```bash
- pip install clearml
- ```
-
-1. Connect the ClearML SDK to the server by [creating credentials](https://app.clear.ml/settings/workspace-configuration) (go right top to Settings -> Workspace -> Create new credentials), then execute the command below and follow the instructions:
-
- ```bash
- clearml-init
- ```
-
-That's it! You're done 😎
-
-<br />
-
-## 🚀 Training YOLOv5 With ClearML
-
-To enable ClearML experiment tracking, simply install the ClearML pip package.
-
-```bash
-pip install clearml>=1.2.0
-```
-
-This will enable integration with the YOLOv5 training script. Every training run from now on, will be captured and stored by the ClearML experiment manager.
-
-If you want to change the `project_name` or `task_name`, use the `--project` and `--name` arguments of the `train.py` script, by default the project will be called `YOLOv5` and the task `Training`.
-PLEASE NOTE: ClearML uses `/` as a delimiter for subprojects, so be careful when using `/` in your project name!
-
-```bash
-python train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
-```
-
-or with custom project and task name:
-
-```bash
-python train.py --project my_project --name my_training --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
-```
-
-This will capture:
-
-- Source code + uncommitted changes
-- Installed packages
-- (Hyper)parameters
-- Model files (use `--save-period n` to save a checkpoint every n epochs)
-- Console output
-- Scalars (mAP_0.5, mAP_0.5:0.95, precision, recall, losses, learning rates, ...)
-- General info such as machine details, runtime, creation date etc.
-- All produced plots such as label correlogram and confusion matrix
-- Images with bounding boxes per epoch
-- Mosaic per epoch
-- Validation images per epoch
-- ...
-
-That's a lot right? 🤯
-Now, we can visualize all of this information in the ClearML UI to get an overview of our training progress. Add custom columns to the table view (such as e.g. mAP_0.5) so you can easily sort on the best performing model. Or select multiple experiments and directly compare them!
-
-There even more we can do with all of this information, like hyperparameter optimization and remote execution, so keep reading if you want to see how that works!
-
-<br />
-
-## 🔗 Dataset Version Management
-
-Versioning your data separately from your code is generally a good idea and makes it easy to acquire the latest version too. This repository supports supplying a dataset version ID, and it will make sure to get the data if it's not there yet. Next to that, this workflow also saves the used dataset ID as part of the task parameters, so you will always know for sure which data was used in which experiment!
-
-
-
-### Prepare Your Dataset
-
-The YOLOv5 repository supports a number of different datasets by using yaml files containing their information. By default datasets are downloaded to the `../datasets` folder in relation to the repository root folder. So if you downloaded the `coco128` dataset using the link in the yaml or with the scripts provided by yolov5, you get this folder structure:
-
-```
-..
-|_ yolov5
-|_ datasets
- |_ coco128
- |_ images
- |_ labels
- |_ LICENSE
- |_ README.txt
-```
-
-But this can be any dataset you wish. Feel free to use your own, as long as you keep to this folder structure.
-
-Next, ⚠️**copy the corresponding yaml file to the root of the dataset folder**⚠️. This yaml files contains the information ClearML will need to properly use the dataset. You can make this yourself too, of course, just follow the structure of the example yamls.
-
-Basically we need the following keys: `path`, `train`, `test`, `val`, `nc`, `names`.
-
-```
-..
-|_ yolov5
-|_ datasets
- |_ coco128
- |_ images
- |_ labels
- |_ coco128.yaml # <---- HERE!
- |_ LICENSE
- |_ README.txt
-```
-
-### Upload Your Dataset
-
-To get this dataset into ClearML as a versioned dataset, go to the dataset root folder and run the following command:
-
-```bash
-cd coco128
-clearml-data sync --project YOLOv5 --name coco128 --folder .
-```
-
-The command `clearml-data sync` is actually a shorthand command. You could also run these commands one after the other:
-
-```bash
-# Optionally add --parent <parent_dataset_id> if you want to base
-# this version on another dataset version, so no duplicate files are uploaded!
-clearml-data create --name coco128 --project YOLOv5
-clearml-data add --files .
-clearml-data close
-```
-
-### Run Training Using A ClearML Dataset
-
-Now that you have a ClearML dataset, you can very simply use it to train custom YOLOv5 🚀 models!
-
-```bash
-python train.py --img 640 --batch 16 --epochs 3 --data clearml://<your_dataset_id> --weights yolov5s.pt --cache
-```
-
-<br />
-
-## 👀 Hyperparameter Optimization
-
-Now that we have our experiments and data versioned, it's time to take a look at what we can build on top!
-
-Using the code information, installed packages and environment details, the experiment itself is now **completely reproducible**. In fact, ClearML allows you to clone an experiment and even change its parameters. We can then just rerun it with these new parameters automatically, this is basically what HPO does!
-
-To **run hyperparameter optimization locally**, we've included a pre-made script for you. Just make sure a training task has been run at least once, so it is in the ClearML experiment manager, we will essentially clone it and change its hyperparameters.
-
-You'll need to fill in the ID of this `template task` in the script found at `utils/loggers/clearml/hpo.py` and then just run it :) You can change `task.execute_locally()` to `task.execute()` to put it in a ClearML queue and have a remote agent work on it instead.
-
-```bash
-# To use optuna, install it first, otherwise you can change the optimizer to just be RandomSearch
-pip install optuna
-python utils/loggers/clearml/hpo.py
-```
-
-
-
-## 🤯 Remote Execution (advanced)
-
-Running HPO locally is really handy, but what if we want to run our experiments on a remote machine instead? Maybe you have access to a very powerful GPU machine on-site, or you have some budget to use cloud GPUs.
-This is where the ClearML Agent comes into play. Check out what the agent can do here:
-
-- [YouTube video](https://youtu.be/MX3BrXnaULs)
-- [Documentation](https://clear.ml/docs/latest/docs/clearml_agent)
-
-In short: every experiment tracked by the experiment manager contains enough information to reproduce it on a different machine (installed packages, uncommitted changes etc.). So a ClearML agent does just that: it listens to a queue for incoming tasks and when it finds one, it recreates the environment and runs it while still reporting scalars, plots etc. to the experiment manager.
-
-You can turn any machine (a cloud VM, a local GPU machine, your own laptop ... ) into a ClearML agent by simply running:
-
-```bash
-clearml-agent daemon --queue <queues_to_listen_to> [--docker]
-```
-
-### Cloning, Editing And Enqueuing
-
-With our agent running, we can give it some work. Remember from the HPO section that we can clone a task and edit the hyperparameters? We can do that from the interface too!
-
-🪄 Clone the experiment by right-clicking it
-
-🎯 Edit the hyperparameters to what you wish them to be
-
-⏳ Enqueue the task to any of the queues by right-clicking it
-
-
-
-### Executing A Task Remotely
-
-Now you can clone a task like we explained above, or simply mark your current script by adding `task.execute_remotely()` and on execution it will be put into a queue, for the agent to start working on!
-
-To run the YOLOv5 training script remotely, all you have to do is add this line to the training.py script after the clearml logger has been instantiated:
-
-```python
-# ...
-# Loggers
-data_dict = None
-if RANK in {-1, 0}:
- loggers = Loggers(save_dir, weights, opt, hyp, LOGGER) # loggers instance
- if loggers.clearml:
- loggers.clearml.task.execute_remotely(queue="my_queue") # <------ ADD THIS LINE
- # Data_dict is either None is user did not choose for ClearML dataset or is filled in by ClearML
- data_dict = loggers.clearml.data_dict
-# ...
-```
-
-When running the training script after this change, python will run the script up until that line, after which it will package the code and send it to the queue instead!
-
-### Autoscaling workers
-
-ClearML comes with autoscalers too! This tool will automatically spin up new remote machines in the cloud of your choice (AWS, GCP, Azure) and turn them into ClearML agents for you whenever there are experiments detected in the queue. Once the tasks are processed, the autoscaler will automatically shut down the remote machines, and you stop paying!
-
-Check out the autoscalers getting started video below.
-
-[](https://youtu.be/j4XVMAaUt3E)
diff --git a/yolov5/utils/loggers/clearmi/__init__.py b/yolov5/utils/loggers/clearmi/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/yolov5/utils/loggers/clearmi/clearml_utils.py b/yolov5/utils/loggers/clearmi/clearml_utils.py
deleted file mode 100644
index 2764abe90da80a7b270bca9c0fd89b99ec25af3b..0000000000000000000000000000000000000000
--- a/yolov5/utils/loggers/clearmi/clearml_utils.py
+++ /dev/null
@@ -1,164 +0,0 @@
-"""Main Logger class for ClearML experiment tracking."""
-import glob
-import re
-from pathlib import Path
-
-import numpy as np
-import yaml
-
-from utils.plots import Annotator, colors
-
-try:
- import clearml
- from clearml import Dataset, Task
-
- assert hasattr(clearml, '__version__') # verify package import not local dir
-except (ImportError, AssertionError):
- clearml = None
-
-
-def construct_dataset(clearml_info_string):
- """Load in a clearml dataset and fill the internal data_dict with its contents.
- """
- dataset_id = clearml_info_string.replace('clearml://', '')
- dataset = Dataset.get(dataset_id=dataset_id)
- dataset_root_path = Path(dataset.get_local_copy())
-
- # We'll search for the yaml file definition in the dataset
- yaml_filenames = list(glob.glob(str(dataset_root_path / '*.yaml')) + glob.glob(str(dataset_root_path / '*.yml')))
- if len(yaml_filenames) > 1:
- raise ValueError('More than one yaml file was found in the dataset root, cannot determine which one contains '
- 'the dataset definition this way.')
- elif len(yaml_filenames) == 0:
- raise ValueError('No yaml definition found in dataset root path, check that there is a correct yaml file '
- 'inside the dataset root path.')
- with open(yaml_filenames[0]) as f:
- dataset_definition = yaml.safe_load(f)
-
- assert set(dataset_definition.keys()).issuperset(
- {'train', 'test', 'val', 'nc', 'names'}
- ), "The right keys were not found in the yaml file, make sure it at least has the following keys: ('train', 'test', 'val', 'nc', 'names')"
-
- data_dict = dict()
- data_dict['train'] = str(
- (dataset_root_path / dataset_definition['train']).resolve()) if dataset_definition['train'] else None
- data_dict['test'] = str(
- (dataset_root_path / dataset_definition['test']).resolve()) if dataset_definition['test'] else None
- data_dict['val'] = str(
- (dataset_root_path / dataset_definition['val']).resolve()) if dataset_definition['val'] else None
- data_dict['nc'] = dataset_definition['nc']
- data_dict['names'] = dataset_definition['names']
-
- return data_dict
-
-
-class ClearmlLogger:
- """Log training runs, datasets, models, and predictions to ClearML.
-
- This logger sends information to ClearML at app.clear.ml or to your own hosted server. By default,
- this information includes hyperparameters, system configuration and metrics, model metrics, code information and
- basic data metrics and analyses.
-
- By providing additional command line arguments to train.py, datasets,
- models and predictions can also be logged.
- """
-
- def __init__(self, opt, hyp):
- """
- - Initialize ClearML Task, this object will capture the experiment
- - Upload dataset version to ClearML Data if opt.upload_dataset is True
-
- arguments:
- opt (namespace) -- Commandline arguments for this run
- hyp (dict) -- Hyperparameters for this run
-
- """
- self.current_epoch = 0
- # Keep tracked of amount of logged images to enforce a limit
- self.current_epoch_logged_images = set()
- # Maximum number of images to log to clearML per epoch
- self.max_imgs_to_log_per_epoch = 16
- # Get the interval of epochs when bounding box images should be logged
- self.bbox_interval = opt.bbox_interval
- self.clearml = clearml
- self.task = None
- self.data_dict = None
- if self.clearml:
- self.task = Task.init(
- project_name=opt.project if opt.project != 'runs/train' else 'YOLOv5',
- task_name=opt.name if opt.name != 'exp' else 'Training',
- tags=['YOLOv5'],
- output_uri=True,
- reuse_last_task_id=opt.exist_ok,
- auto_connect_frameworks={'pytorch': False}
- # We disconnect pytorch auto-detection, because we added manual model save points in the code
- )
- # ClearML's hooks will already grab all general parameters
- # Only the hyperparameters coming from the yaml config file
- # will have to be added manually!
- self.task.connect(hyp, name='Hyperparameters')
- self.task.connect(opt, name='Args')
-
- # Make sure the code is easily remotely runnable by setting the docker image to use by the remote agent
- self.task.set_base_docker('ultralytics/yolov5:latest',
- docker_arguments='--ipc=host -e="CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1"',
- docker_setup_bash_script='pip install clearml')
-
- # Get ClearML Dataset Version if requested
- if opt.data.startswith('clearml://'):
- # data_dict should have the following keys:
- # names, nc (number of classes), test, train, val (all three relative paths to ../datasets)
- self.data_dict = construct_dataset(opt.data)
- # Set data to data_dict because wandb will crash without this information and opt is the best way
- # to give it to them
- opt.data = self.data_dict
-
- def log_debug_samples(self, files, title='Debug Samples'):
- """
- Log files (images) as debug samples in the ClearML task.
-
- arguments:
- files (List(PosixPath)) a list of file paths in PosixPath format
- title (str) A title that groups together images with the same values
- """
- for f in files:
- if f.exists():
- it = re.search(r'_batch(\d+)', f.name)
- iteration = int(it.groups()[0]) if it else 0
- self.task.get_logger().report_image(title=title,
- series=f.name.replace(it.group(), ''),
- local_path=str(f),
- iteration=iteration)
-
- def log_image_with_boxes(self, image_path, boxes, class_names, image, conf_threshold=0.25):
- """
- Draw the bounding boxes on a single image and report the result as a ClearML debug sample.
-
- arguments:
- image_path (PosixPath) the path the original image file
- boxes (list): list of scaled predictions in the format - [xmin, ymin, xmax, ymax, confidence, class]
- class_names (dict): dict containing mapping of class int to class name
- image (Tensor): A torch tensor containing the actual image data
- """
- if len(self.current_epoch_logged_images) < self.max_imgs_to_log_per_epoch and self.current_epoch >= 0:
- # Log every bbox_interval times and deduplicate for any intermittend extra eval runs
- if self.current_epoch % self.bbox_interval == 0 and image_path not in self.current_epoch_logged_images:
- im = np.ascontiguousarray(np.moveaxis(image.mul(255).clamp(0, 255).byte().cpu().numpy(), 0, 2))
- annotator = Annotator(im=im, pil=True)
- for i, (conf, class_nr, box) in enumerate(zip(boxes[:, 4], boxes[:, 5], boxes[:, :4])):
- color = colors(i)
-
- class_name = class_names[int(class_nr)]
- confidence_percentage = round(float(conf) * 100, 2)
- label = f'{class_name}: {confidence_percentage}%'
-
- if conf > conf_threshold:
- annotator.rectangle(box.cpu().numpy(), outline=color)
- annotator.box_label(box.cpu().numpy(), label=label, color=color)
-
- annotated_image = annotator.result()
- self.task.get_logger().report_image(title='Bounding Boxes',
- series=image_path.name,
- iteration=self.current_epoch,
- image=annotated_image)
- self.current_epoch_logged_images.add(image_path)
diff --git a/yolov5/utils/loggers/clearmi/hpo.py b/yolov5/utils/loggers/clearmi/hpo.py
deleted file mode 100644
index ee518b0fbfc89ee811b51bbf85341eee4f685be1..0000000000000000000000000000000000000000
--- a/yolov5/utils/loggers/clearmi/hpo.py
+++ /dev/null
@@ -1,84 +0,0 @@
-from clearml import Task
-# Connecting ClearML with the current process,
-# from here on everything is logged automatically
-from clearml.automation import HyperParameterOptimizer, UniformParameterRange
-from clearml.automation.optuna import OptimizerOptuna
-
-task = Task.init(project_name='Hyper-Parameter Optimization',
- task_name='YOLOv5',
- task_type=Task.TaskTypes.optimizer,
- reuse_last_task_id=False)
-
-# Example use case:
-optimizer = HyperParameterOptimizer(
- # This is the experiment we want to optimize
- base_task_id='<your_template_task_id>',
- # here we define the hyper-parameters to optimize
- # Notice: The parameter name should exactly match what you see in the UI: <section_name>/<parameter>
- # For Example, here we see in the base experiment a section Named: "General"
- # under it a parameter named "batch_size", this becomes "General/batch_size"
- # If you have `argparse` for example, then arguments will appear under the "Args" section,
- # and you should instead pass "Args/batch_size"
- hyper_parameters=[
- UniformParameterRange('Hyperparameters/lr0', min_value=1e-5, max_value=1e-1),
- UniformParameterRange('Hyperparameters/lrf', min_value=0.01, max_value=1.0),
- UniformParameterRange('Hyperparameters/momentum', min_value=0.6, max_value=0.98),
- UniformParameterRange('Hyperparameters/weight_decay', min_value=0.0, max_value=0.001),
- UniformParameterRange('Hyperparameters/warmup_epochs', min_value=0.0, max_value=5.0),
- UniformParameterRange('Hyperparameters/warmup_momentum', min_value=0.0, max_value=0.95),
- UniformParameterRange('Hyperparameters/warmup_bias_lr', min_value=0.0, max_value=0.2),
- UniformParameterRange('Hyperparameters/box', min_value=0.02, max_value=0.2),
- UniformParameterRange('Hyperparameters/cls', min_value=0.2, max_value=4.0),
- UniformParameterRange('Hyperparameters/cls_pw', min_value=0.5, max_value=2.0),
- UniformParameterRange('Hyperparameters/obj', min_value=0.2, max_value=4.0),
- UniformParameterRange('Hyperparameters/obj_pw', min_value=0.5, max_value=2.0),
- UniformParameterRange('Hyperparameters/iou_t', min_value=0.1, max_value=0.7),
- UniformParameterRange('Hyperparameters/anchor_t', min_value=2.0, max_value=8.0),
- UniformParameterRange('Hyperparameters/fl_gamma', min_value=0.0, max_value=4.0),
- UniformParameterRange('Hyperparameters/hsv_h', min_value=0.0, max_value=0.1),
- UniformParameterRange('Hyperparameters/hsv_s', min_value=0.0, max_value=0.9),
- UniformParameterRange('Hyperparameters/hsv_v', min_value=0.0, max_value=0.9),
- UniformParameterRange('Hyperparameters/degrees', min_value=0.0, max_value=45.0),
- UniformParameterRange('Hyperparameters/translate', min_value=0.0, max_value=0.9),
- UniformParameterRange('Hyperparameters/scale', min_value=0.0, max_value=0.9),
- UniformParameterRange('Hyperparameters/shear', min_value=0.0, max_value=10.0),
- UniformParameterRange('Hyperparameters/perspective', min_value=0.0, max_value=0.001),
- UniformParameterRange('Hyperparameters/flipud', min_value=0.0, max_value=1.0),
- UniformParameterRange('Hyperparameters/fliplr', min_value=0.0, max_value=1.0),
- UniformParameterRange('Hyperparameters/mosaic', min_value=0.0, max_value=1.0),
- UniformParameterRange('Hyperparameters/mixup', min_value=0.0, max_value=1.0),
- UniformParameterRange('Hyperparameters/copy_paste', min_value=0.0, max_value=1.0)],
- # this is the objective metric we want to maximize/minimize
- objective_metric_title='metrics',
- objective_metric_series='mAP_0.5',
- # now we decide if we want to maximize it or minimize it (accuracy we maximize)
- objective_metric_sign='max',
- # let us limit the number of concurrent experiments,
- # this in turn will make sure we do dont bombard the scheduler with experiments.
- # if we have an auto-scaler connected, this, by proxy, will limit the number of machine
- max_number_of_concurrent_tasks=1,
- # this is the optimizer class (actually doing the optimization)
- # Currently, we can choose from GridSearch, RandomSearch or OptimizerBOHB (Bayesian optimization Hyper-Band)
- optimizer_class=OptimizerOptuna,
- # If specified only the top K performing Tasks will be kept, the others will be automatically archived
- save_top_k_tasks_only=5, # 5,
- compute_time_limit=None,
- total_max_jobs=20,
- min_iteration_per_job=None,
- max_iteration_per_job=None,
-)
-
-# report every 10 seconds, this is way too often, but we are testing here
-optimizer.set_report_period(10 / 60)
-# You can also use the line below instead to run all the optimizer tasks locally, without using queues or agent
-# an_optimizer.start_locally(job_complete_callback=job_complete_callback)
-# set the time limit for the optimization process (2 hours)
-optimizer.set_time_limit(in_minutes=120.0)
-# Start the optimization process in the local environment
-optimizer.start_locally()
-# wait until process is done (notice we are controlling the optimization process in the background)
-optimizer.wait()
-# make sure background optimization stopped
-optimizer.stop()
-
-print('We are done, good bye')
diff --git a/yolov5/val.py b/yolov5/val.py
new file mode 100644
index 0000000000000000000000000000000000000000..d4073b42fe781ab3d3fa95d4d9d54159dc3e7f25
--- /dev/null
+++ b/yolov5/val.py
@@ -0,0 +1,409 @@
+# YOLOv5 🚀 by Ultralytics, GPL-3.0 license
+"""
+Validate a trained YOLOv5 detection model on a detection dataset
+
+Usage:
+ $ python val.py --weights yolov5s.pt --data coco128.yaml --img 640
+
+Usage - formats:
+ $ python val.py --weights yolov5s.pt # PyTorch
+ yolov5s.torchscript # TorchScript
+ yolov5s.onnx # ONNX Runtime or OpenCV DNN with --dnn
+ yolov5s_openvino_model # OpenVINO
+ yolov5s.engine # TensorRT
+ yolov5s.mlmodel # CoreML (macOS-only)
+ yolov5s_saved_model # TensorFlow SavedModel
+ yolov5s.pb # TensorFlow GraphDef
+ yolov5s.tflite # TensorFlow Lite
+ yolov5s_edgetpu.tflite # TensorFlow Edge TPU
+ yolov5s_paddle_model # PaddlePaddle
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+from tqdm import tqdm
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[0] # YOLOv5 root directory
+if str(ROOT) not in sys.path:
+ sys.path.append(str(ROOT)) # add ROOT to PATH
+ROOT = Path(os.path.relpath(ROOT, Path.cwd())) # relative
+
+from models.common import DetectMultiBackend
+from utils.callbacks import Callbacks
+from utils.dataloaders import create_dataloader
+from utils.general import (LOGGER, TQDM_BAR_FORMAT, Profile, check_dataset, check_img_size, check_requirements,
+ check_yaml, coco80_to_coco91_class, colorstr, increment_path, non_max_suppression,
+ print_args, scale_boxes, xywh2xyxy, xyxy2xywh)
+from utils.metrics import ConfusionMatrix, ap_per_class, box_iou
+from utils.plots import output_to_target, plot_images, plot_val_study
+from utils.torch_utils import select_device, smart_inference_mode
+
+
+def save_one_txt(predn, save_conf, shape, file):
+ # Save one txt result
+ gn = torch.tensor(shape)[[1, 0, 1, 0]] # normalization gain whwh
+ for *xyxy, conf, cls in predn.tolist():
+ xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist() # normalized xywh
+ line = (cls, *xywh, conf) if save_conf else (cls, *xywh) # label format
+ with open(file, 'a') as f:
+ f.write(('%g ' * len(line)).rstrip() % line + '\n')
+
+
+def save_one_json(predn, jdict, path, class_map):
+ # Save one JSON result {"image_id": 42, "category_id": 18, "bbox": [258.15, 41.29, 348.26, 243.78], "score": 0.236}
+ image_id = int(path.stem) if path.stem.isnumeric() else path.stem
+ box = xyxy2xywh(predn[:, :4]) # xywh
+ box[:, :2] -= box[:, 2:] / 2 # xy center to top-left corner
+ for p, b in zip(predn.tolist(), box.tolist()):
+ jdict.append({
+ 'image_id': image_id,
+ 'category_id': class_map[int(p[5])],
+ 'bbox': [round(x, 3) for x in b],
+ 'score': round(p[4], 5)})
+
+
+def process_batch(detections, labels, iouv):
+ """
+ Return correct prediction matrix
+ Arguments:
+ detections (array[N, 6]), x1, y1, x2, y2, conf, class
+ labels (array[M, 5]), class, x1, y1, x2, y2
+ Returns:
+ correct (array[N, 10]), for 10 IoU levels
+ """
+ correct = np.zeros((detections.shape[0], iouv.shape[0])).astype(bool)
+ iou = box_iou(labels[:, 1:], detections[:, :4])
+ correct_class = labels[:, 0:1] == detections[:, 5]
+ for i in range(len(iouv)):
+ x = torch.where((iou >= iouv[i]) & correct_class) # IoU > threshold and classes match
+ if x[0].shape[0]:
+ matches = torch.cat((torch.stack(x, 1), iou[x[0], x[1]][:, None]), 1).cpu().numpy() # [label, detect, iou]
+ if x[0].shape[0] > 1:
+ matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 1], return_index=True)[1]]
+ # matches = matches[matches[:, 2].argsort()[::-1]]
+ matches = matches[np.unique(matches[:, 0], return_index=True)[1]]
+ correct[matches[:, 1].astype(int), i] = True
+ return torch.tensor(correct, dtype=torch.bool, device=iouv.device)
+
+
+@smart_inference_mode()
+def run(
+ data,
+ weights=None, # model.pt path(s)
+ batch_size=32, # batch size
+ imgsz=640, # inference size (pixels)
+ conf_thres=0.001, # confidence threshold
+ iou_thres=0.6, # NMS IoU threshold
+ max_det=300, # maximum detections per image
+ task='val', # train, val, test, speed or study
+ device='', # cuda device, i.e. 0 or 0,1,2,3 or cpu
+ workers=8, # max dataloader workers (per RANK in DDP mode)
+ single_cls=False, # treat as single-class dataset
+ augment=False, # augmented inference
+ verbose=False, # verbose output
+ save_txt=False, # save results to *.txt
+ save_hybrid=False, # save label+prediction hybrid results to *.txt
+ save_conf=False, # save confidences in --save-txt labels
+ save_json=False, # save a COCO-JSON results file
+ project=ROOT / 'runs/val', # save to project/name
+ name='exp', # save to project/name
+ exist_ok=False, # existing project/name ok, do not increment
+ half=True, # use FP16 half-precision inference
+ dnn=False, # use OpenCV DNN for ONNX inference
+ model=None,
+ dataloader=None,
+ save_dir=Path(''),
+ plots=True,
+ callbacks=Callbacks(),
+ compute_loss=None,
+):
+ # Initialize/load model and set device
+ training = model is not None
+ if training: # called by train.py
+ device, pt, jit, engine = next(model.parameters()).device, True, False, False # get model device, PyTorch model
+ half &= device.type != 'cpu' # half precision only supported on CUDA
+ model.half() if half else model.float()
+ else: # called directly
+ device = select_device(device, batch_size=batch_size)
+
+ # Directories
+ save_dir = increment_path(Path(project) / name, exist_ok=exist_ok) # increment run
+ (save_dir / 'labels' if save_txt else save_dir).mkdir(parents=True, exist_ok=True) # make dir
+
+ # Load model
+ model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
+ stride, pt, jit, engine = model.stride, model.pt, model.jit, model.engine
+ imgsz = check_img_size(imgsz, s=stride) # check image size
+ half = model.fp16 # FP16 supported on limited backends with CUDA
+ if engine:
+ batch_size = model.batch_size
+ else:
+ device = model.device
+ if not (pt or jit):
+ batch_size = 1 # export.py models default to batch-size 1
+ LOGGER.info(f'Forcing --batch-size 1 square inference (1,3,{imgsz},{imgsz}) for non-PyTorch models')
+
+ # Data
+ data = check_dataset(data) # check
+
+ # Configure
+ model.eval()
+ cuda = device.type != 'cpu'
+ is_coco = isinstance(data.get('val'), str) and data['val'].endswith(f'coco{os.sep}val2017.txt') # COCO dataset
+ nc = 1 if single_cls else int(data['nc']) # number of classes
+ iouv = torch.linspace(0.5, 0.95, 10, device=device) # iou vector for mAP@0.5:0.95
+ niou = iouv.numel()
+
+ # Dataloader
+ if not training:
+ if pt and not single_cls: # check --weights are trained on --data
+ ncm = model.model.nc
+ assert ncm == nc, f'{weights} ({ncm} classes) trained on different --data than what you passed ({nc} ' \
+ f'classes). Pass correct combination of --weights and --data that are trained together.'
+ model.warmup(imgsz=(1 if pt else batch_size, 3, imgsz, imgsz)) # warmup
+ pad, rect = (0.0, False) if task == 'speed' else (0.5, pt) # square inference for benchmarks
+ task = task if task in ('train', 'val', 'test') else 'val' # path to train/val/test images
+ dataloader = create_dataloader(data[task],
+ imgsz,
+ batch_size,
+ stride,
+ single_cls,
+ pad=pad,
+ rect=rect,
+ workers=workers,
+ prefix=colorstr(f'{task}: '))[0]
+
+ seen = 0
+ confusion_matrix = ConfusionMatrix(nc=nc)
+ names = model.names if hasattr(model, 'names') else model.module.names # get class names
+ if isinstance(names, (list, tuple)): # old format
+ names = dict(enumerate(names))
+ class_map = coco80_to_coco91_class() if is_coco else list(range(1000))
+ s = ('%22s' + '%11s' * 6) % ('Class', 'Images', 'Instances', 'P', 'R', 'mAP50', 'mAP50-95')
+ tp, fp, p, r, f1, mp, mr, map50, ap50, map = 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
+ dt = Profile(), Profile(), Profile() # profiling times
+ loss = torch.zeros(3, device=device)
+ jdict, stats, ap, ap_class = [], [], [], []
+ callbacks.run('on_val_start')
+ pbar = tqdm(dataloader, desc=s, bar_format=TQDM_BAR_FORMAT) # progress bar
+ for batch_i, (im, targets, paths, shapes) in enumerate(pbar):
+ callbacks.run('on_val_batch_start')
+ with dt[0]:
+ if cuda:
+ im = im.to(device, non_blocking=True)
+ targets = targets.to(device)
+ im = im.half() if half else im.float() # uint8 to fp16/32
+ im /= 255 # 0 - 255 to 0.0 - 1.0
+ nb, _, height, width = im.shape # batch size, channels, height, width
+
+ # Inference
+ with dt[1]:
+ preds, train_out = model(im) if compute_loss else (model(im, augment=augment), None)
+
+ # Loss
+ if compute_loss:
+ loss += compute_loss(train_out, targets)[1] # box, obj, cls
+
+ # NMS
+ targets[:, 2:] *= torch.tensor((width, height, width, height), device=device) # to pixels
+ lb = [targets[targets[:, 0] == i, 1:] for i in range(nb)] if save_hybrid else [] # for autolabelling
+ with dt[2]:
+ preds = non_max_suppression(preds,
+ conf_thres,
+ iou_thres,
+ labels=lb,
+ multi_label=True,
+ agnostic=single_cls,
+ max_det=max_det)
+
+ # Metrics
+ for si, pred in enumerate(preds):
+ labels = targets[targets[:, 0] == si, 1:]
+ nl, npr = labels.shape[0], pred.shape[0] # number of labels, predictions
+ path, shape = Path(paths[si]), shapes[si][0]
+ correct = torch.zeros(npr, niou, dtype=torch.bool, device=device) # init
+ seen += 1
+
+ if npr == 0:
+ if nl:
+ stats.append((correct, *torch.zeros((2, 0), device=device), labels[:, 0]))
+ if plots:
+ confusion_matrix.process_batch(detections=None, labels=labels[:, 0])
+ continue
+
+ # Predictions
+ if single_cls:
+ pred[:, 5] = 0
+ predn = pred.clone()
+ scale_boxes(im[si].shape[1:], predn[:, :4], shape, shapes[si][1]) # native-space pred
+
+ # Evaluate
+ if nl:
+ tbox = xywh2xyxy(labels[:, 1:5]) # target boxes
+ scale_boxes(im[si].shape[1:], tbox, shape, shapes[si][1]) # native-space labels
+ labelsn = torch.cat((labels[:, 0:1], tbox), 1) # native-space labels
+ correct = process_batch(predn, labelsn, iouv)
+ if plots:
+ confusion_matrix.process_batch(predn, labelsn)
+ stats.append((correct, pred[:, 4], pred[:, 5], labels[:, 0])) # (correct, conf, pcls, tcls)
+
+ # Save/log
+ if save_txt:
+ save_one_txt(predn, save_conf, shape, file=save_dir / 'labels' / f'{path.stem}.txt')
+ if save_json:
+ save_one_json(predn, jdict, path, class_map) # append to COCO-JSON dictionary
+ callbacks.run('on_val_image_end', pred, predn, path, names, im[si])
+
+ # Plot images
+ if plots and batch_i < 3:
+ plot_images(im, targets, paths, save_dir / f'val_batch{batch_i}_labels.jpg', names) # labels
+ plot_images(im, output_to_target(preds), paths, save_dir / f'val_batch{batch_i}_pred.jpg', names) # pred
+
+ callbacks.run('on_val_batch_end', batch_i, im, targets, paths, shapes, preds)
+
+ # Compute metrics
+ stats = [torch.cat(x, 0).cpu().numpy() for x in zip(*stats)] # to numpy
+ if len(stats) and stats[0].any():
+ tp, fp, p, r, f1, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)
+ ap50, ap = ap[:, 0], ap.mean(1) # AP@0.5, AP@0.5:0.95
+ mp, mr, map50, map = p.mean(), r.mean(), ap50.mean(), ap.mean()
+ nt = np.bincount(stats[3].astype(int), minlength=nc) # number of targets per class
+
+ # Print results
+ pf = '%22s' + '%11i' * 2 + '%11.3g' * 4 # print format
+ LOGGER.info(pf % ('all', seen, nt.sum(), mp, mr, map50, map))
+ if nt.sum() == 0:
+ LOGGER.warning(f'WARNING ⚠️ no labels found in {task} set, can not compute metrics without labels')
+
+ # Print results per class
+ if (verbose or (nc < 50 and not training)) and nc > 1 and len(stats):
+ for i, c in enumerate(ap_class):
+ LOGGER.info(pf % (names[c], seen, nt[c], p[i], r[i], ap50[i], ap[i]))
+
+ # Print speeds
+ t = tuple(x.t / seen * 1E3 for x in dt) # speeds per image
+ if not training:
+ shape = (batch_size, 3, imgsz, imgsz)
+ LOGGER.info(f'Speed: %.1fms pre-process, %.1fms inference, %.1fms NMS per image at shape {shape}' % t)
+
+ # Plots
+ if plots:
+ confusion_matrix.plot(save_dir=save_dir, names=list(names.values()))
+ callbacks.run('on_val_end', nt, tp, fp, p, r, f1, ap, ap50, ap_class, confusion_matrix)
+
+ # Save JSON
+ if save_json and len(jdict):
+ w = Path(weights[0] if isinstance(weights, list) else weights).stem if weights is not None else '' # weights
+ anno_json = str(Path('../datasets/coco/annotations/instances_val2017.json')) # annotations
+ pred_json = str(save_dir / f'{w}_predictions.json') # predictions
+ LOGGER.info(f'\nEvaluating pycocotools mAP... saving {pred_json}...')
+ with open(pred_json, 'w') as f:
+ json.dump(jdict, f)
+
+ try: # https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb
+ check_requirements('pycocotools>=2.0.6')
+ from pycocotools.coco import COCO
+ from pycocotools.cocoeval import COCOeval
+
+ anno = COCO(anno_json) # init annotations api
+ pred = anno.loadRes(pred_json) # init predictions api
+ eval = COCOeval(anno, pred, 'bbox')
+ if is_coco:
+ eval.params.imgIds = [int(Path(x).stem) for x in dataloader.dataset.im_files] # image IDs to evaluate
+ eval.evaluate()
+ eval.accumulate()
+ eval.summarize()
+ map, map50 = eval.stats[:2] # update results (mAP@0.5:0.95, mAP@0.5)
+ except Exception as e:
+ LOGGER.info(f'pycocotools unable to run: {e}')
+
+ # Return results
+ model.float() # for training
+ if not training:
+ s = f"\n{len(list(save_dir.glob('labels/*.txt')))} labels saved to {save_dir / 'labels'}" if save_txt else ''
+ LOGGER.info(f"Results saved to {colorstr('bold', save_dir)}{s}")
+ maps = np.zeros(nc) + map
+ for i, c in enumerate(ap_class):
+ maps[c] = ap[i]
+ return (mp, mr, map50, map, *(loss.cpu() / len(dataloader)).tolist()), maps, t
+
+
+def parse_opt():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='dataset.yaml path')
+ parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model path(s)')
+ parser.add_argument('--batch-size', type=int, default=32, help='batch size')
+ parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='inference size (pixels)')
+ parser.add_argument('--conf-thres', type=float, default=0.001, help='confidence threshold')
+ parser.add_argument('--iou-thres', type=float, default=0.6, help='NMS IoU threshold')
+ parser.add_argument('--max-det', type=int, default=300, help='maximum detections per image')
+ parser.add_argument('--task', default='val', help='train, val, test, speed or study')
+ parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
+ parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
+ parser.add_argument('--single-cls', action='store_true', help='treat as single-class dataset')
+ parser.add_argument('--augment', action='store_true', help='augmented inference')
+ parser.add_argument('--verbose', action='store_true', help='report mAP by class')
+ parser.add_argument('--save-txt', action='store_true', help='save results to *.txt')
+ parser.add_argument('--save-hybrid', action='store_true', help='save label+prediction hybrid results to *.txt')
+ parser.add_argument('--save-conf', action='store_true', help='save confidences in --save-txt labels')
+ parser.add_argument('--save-json', action='store_true', help='save a COCO-JSON results file')
+ parser.add_argument('--project', default=ROOT / 'runs/val', help='save to project/name')
+ parser.add_argument('--name', default='exp', help='save to project/name')
+ parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
+ parser.add_argument('--half', action='store_true', help='use FP16 half-precision inference')
+ parser.add_argument('--dnn', action='store_true', help='use OpenCV DNN for ONNX inference')
+ opt = parser.parse_args()
+ opt.data = check_yaml(opt.data) # check YAML
+ opt.save_json |= opt.data.endswith('coco.yaml')
+ opt.save_txt |= opt.save_hybrid
+ print_args(vars(opt))
+ return opt
+
+
+def main(opt):
+ check_requirements(exclude=('tensorboard', 'thop'))
+
+ if opt.task in ('train', 'val', 'test'): # run normally
+ if opt.conf_thres > 0.001: # https://github.com/ultralytics/yolov5/issues/1466
+ LOGGER.info(f'WARNING ⚠️ confidence threshold {opt.conf_thres} > 0.001 produces invalid results')
+ if opt.save_hybrid:
+ LOGGER.info('WARNING ⚠️ --save-hybrid will return high mAP from hybrid labels, not from predictions alone')
+ run(**vars(opt))
+
+ else:
+ weights = opt.weights if isinstance(opt.weights, list) else [opt.weights]
+ opt.half = torch.cuda.is_available() and opt.device != 'cpu' # FP16 for fastest results
+ if opt.task == 'speed': # speed benchmarks
+ # python val.py --task speed --data coco.yaml --batch 1 --weights yolov5n.pt yolov5s.pt...
+ opt.conf_thres, opt.iou_thres, opt.save_json = 0.25, 0.45, False
+ for opt.weights in weights:
+ run(**vars(opt), plots=False)
+
+ elif opt.task == 'study': # speed vs mAP benchmarks
+ # python val.py --task study --data coco.yaml --iou 0.7 --weights yolov5n.pt yolov5s.pt...
+ for opt.weights in weights:
+ f = f'study_{Path(opt.data).stem}_{Path(opt.weights).stem}.txt' # filename to save to
+ x, y = list(range(256, 1536 + 128, 128)), [] # x axis (image sizes), y axis
+ for opt.imgsz in x: # img-size
+ LOGGER.info(f'\nRunning {f} --imgsz {opt.imgsz}...')
+ r, _, t = run(**vars(opt), plots=False)
+ y.append(r + t) # results and times
+ np.savetxt(f, y, fmt='%10.4g') # save
+ subprocess.run(['zip', '-r', 'study.zip', 'study_*.txt'])
+ plot_val_study(x=x) # plot
+ else:
+ raise NotImplementedError(f'--task {opt.task} not in ("train", "val", "test", "speed", "study")')
+
+
+if __name__ == '__main__':
+ opt = parse_opt()
+ main(opt)