Secure learning-based coordinated UAV–UGV framework design for medical waste transportation

A cost-effective solution with less human involvement must be developed for medical waste (MW) transportation. A learning-based coordinated unmanned aerial vehicle–unmanned ground vehicle (UAV–UGV) (CUU) framework, currently unavoidable use, with a transfer learning algorithm is suggested. A transfer learning algorithm is implemented for collision-free optimal path planning. In the framework, mobile ground robots collect medical waste from waste disposal centers through the pick-and-place technique. Then, networked drones lift the collected medical waste and fly through a predefined optimal trajectory. The framework considers the dynamic behavior of the environment and explores the actions for picking, placing, and dropping medical waste. A deep reinforcement learning mechanism has been incorporated for each successful or unsuccessful action by the framework to provide the rewards. With optimal policies, the coordinated UAV and UGV change their actions in dynamic conditions. An optimal cost of transportation of medical waste by the proposed framework is created by considering the weight of MW packets as the payload capacity of a CUU framework, the cost of steering the UAV and UGV, and the time required to transport the MW. The effectiveness of the CUU framework for MW transportation has been tested using MATLAB. The MW transportation data have been encrypted using an encryption key for security and authenticity.

1 Introduction

In recent years, the amount of medical waste has increased due to increased numbers of unmanned aerial vehicle healthcare centers. These centers produce a large amount of medical waste. The congestion in cities creates problems in the transportation of medical waste. Much medical waste is disposed of using ordinary and contemporary methods, including incineration, steam sterilization, microwave, and landfill disposal (Mishra et al., 2020). The medical wastes include hypodermic needles, hypodermic needles with attached syringes, and needles with attached tubing, blades, broken glass, acupuncture needles, and pipettes, some contaminated with biohazardous or pharmaceutical material. These are generated from most patient care and clinical support areas. The healthcare industry is increasing rapidly, so an effective solution must be developed for the transportation of medical waste to the disposal centers with less human contact. Improved education and standards must be defined for adequate medical waste management (Windfeld and Brooks, 2015). Much planning and logistics work must be done to manage medical waste efficiently in hospital systems.

A peer-to-peer scheme could be created for hazardous medical waste treatment to avoid illegal usage in medical waste treatment. Inadequate management of medical waste in healthcare centers creates health issues for employers of these centers and draws global attention. Due to storage capacity limitations at medical centers, collection services should occur at regular intervals. A periodic and minimum route scheme must be developed to collect medical waste from dispersed hospitals. The major process of medical waste management is segregation, labeling, and separation (Shareefdeen et al., 2022).

Medical waste management comprises the process of separation, short-term storage, safe disposal, and transfer of medical waste. Standards for medical waste management must be defined. The cost that may be incurred in medical waste transportation should be defined, and new schemes must be developed to minimize this cost (Koçak et al., 2016).

A hybrid decision-making approach comprising interval 2-tuple induced distance operators with similarity to the ideal solution (TOPSIS) has been suggested to tackle the healthcare treatment. A trash system equipped with a camera and quick response (QR) code was created for medical waste management to address the uncertainty and diversity noted in the assessment report (Lu et al., 2016). In that study, artificial intelligence techniques were implemented in the classification of medical wastes that have different features. The molecular structure of the wastes was to be analyzed with a spectrum analyzer.

Major problems associated with hazardous medical waste management are inadequate and unstandardized supervision. Current methods of detecting hidden dangers in hazardous waste disposal are inefficient. Implementing fifth-generation (5G) wireless communication technology and big data in the medical waste treatment process is challenging. A platform comprising 5G broadband wireless communication technology and a big data approach is developed for an effective point-to-point (P2P) goods information system (Wang and Nai, 2021).

Radio frequency identification (RFID) technology has been implemented in medical waste management for item collection and transmission without manual intervention. The RFID technology can perform object recognition, multi-object recognition, and object tracking [8]. Existing healthcare waste treatment methods are steam sterilization, microwave, plasma pyrolysis, chemical disinfection, and incineration (Liu et al., 2019). RFID can be used to detect lost medical waste items and prevent illegal recycling and poor supervision. The RFID technology can be used to establish a tracking and processing system, incineration center subsystems, and effective supervision of medical waste (Sun et al., 2019; Wang et al., 2021). The interaction rules are designed for medical waste management. Blockchain-based methods are suggested for medical waste disposal to create trust among different stakeholders (Ahmad et al., 2021). Multi-criteria decision-making methodology has been implemented. The associated uncertainty present in multi-criterion decision-making is tackled by the Pythagorean fuzzy sets.

The process of medical waste management comprises medical waste generation from hospitals and testing centers, waste collection at waste store rooms, waste transportation, waste segregation and sorting, waste treatment, and waste disposal and recycling (Ahmad et al., 2021). A blockchain-based process is implemented to trace the medical equipment and corresponding medical waste. The main stockholders in medical waste management are doctors, medical test operators, patients, health workers, and waste collection, transportation, and disposal companies.

A literature review revealed the limited use of drones in the transfer of medical waste. A collection problem could be considered to obtain an optimal trajectory. Manual collection of waste from collection points may cause health problems for employees. A coordinated multi-drone mobile robot framework could collect medical waste and transfer it to disposal centers with minimum human intervention.

1.1 Research gap

Identifying research gaps is crucial for advancing knowledge in any field. In the context of medical waste transportation, the following potential research gaps could be explored:

• Efficiency and optimization: Explore methods to optimize medical waste transportation routes to minimize costs and reduce environmental impact. Consider factors such as fuel efficiency, vehicle capacity utilization, and scheduling.

• Technological innovations: Investigate the integration of emerging technologies (such as IoT, RFID, or GPS tracking) to enhance the efficiency and tracking capabilities of medical waste transportation. Evaluate how these technologies can improve real-time monitoring, route planning, and overall logistics.

The article is organized as follows: Section 1 provides the introduction. Section 2 discusses the occupational and transportation risks in medical waste transportation. Section 3 overviews a coordinated unmanned aerial vehicle (UAV) and unmanned ground vehicle (UGV) framework, and Section 4 discusses the application of a UAV and UGV framework in medical waste transportation. Section 5 discusses secure medical waste data. Section 6 includes problem formulation with a proposed currently unavoidable use (CUU) framework for medical waste transportation. Section 7 suggests a transfer learning mechanism for MW transportation. Section 8 includes the reinforcement learning mechanism for the framework. Simulation and discussions are covered in Section 9. Section 10 provided the conclusion.

2 Occupational and transportation risks in medical waste

In the context of medical waste management, occupational risk and transportation risk take on specific considerations due to the unique nature of medical waste. Medical waste includes materials generated in healthcare facilities that may threaten human health and the environment if not handled properly. The following sections discuss how occupational and transportation risks relate to medical waste.

2.1 Occupational risks in medical waste management

2.1.1 Exposure to infectious agents

Exposure to infectious agents: Healthcare workers involved in the collection, handling, and disposal of medical waste are at risk of exposure to infectious agents, such as bacteria, viruses, and other pathogens.

2.1.2 Needlestick injuries

Improper disposal of sharps, such as needles and syringes, can lead to needlestick injuries, posing a risk of infection and other health issues.

2.1.3 Chemical exposure

Some medical waste may contain hazardous chemicals or pharmaceuticals, exposing workers to potential health hazards if not managed properly.

2.1.4 Heavy lifting and ergonomic hazards

Workers involved in lifting and moving heavy containers of medical waste may face ergonomic risks, leading to musculoskeletal issues. Occupational safety measures in medical waste management include the use of personal protective equipment (PPE), proper training, safe handling protocols, and the implementation of engineering controls to minimize exposure.

2.2 Transportation risks in medical waste management

2.2.1 Accidental spills and leaks

During transportation, medical waste containers may be at risk of spills or leaks, especially if not securely sealed. This can lead to contamination and exposure risks for transport personnel and the general public.

2.2.2 Inadequate packaging

Improperly packaging medical waste during transportation can lead to breakage, spillage, or damage to containers, increasing the risk of exposure to infectious agents.

2.2.3 Traffic accidents

Transportation accidents can result in the release of medical waste into the environment, creating potential hazards for first responders and others involved in the cleanup.

2.2.4 Regulatory compliance

Failure to comply with transportation regulations for medical waste can lead to legal consequences and increased risks. Transportation safety measures involve proper packaging, labeling, and securing of medical waste during transit. Compliance with transportation regulations, including those related to hazardous materials, is crucial to mitigating risks.

2.2.5 Integrated approach

Medical facilities and waste management companies must adopt an integrated approach to occupational safety and transportation risk management. Comprehensive training programs for healthcare workers and transport personnel are essential to ensure awareness of risks and proper handling procedures. Regular inspections, maintenance of transport vehicles, and adherence to relevant regulations contribute to minimizing risks associated with the transportation of medical waste. By addressing both occupational and transportation risks in medical waste management, organizations can promote the safety of their workers, protect public health, and prevent environmental contamination.

3 Coordinated UAV and UGV framework: an overview

Coordination between unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) is essential for the successful implementation of various applications, ranging from surveillance and reconnaissance to disaster response and agriculture. Key aspects of UAV–UGV coordination are discussed in the sections that follow.

3.1 Inter vehicle communication

UAVs and UGVs must be equipped with communication systems to exchange real-time information. This can include data on the environment, mission status, and obstacles.

3.2 Command and control

A robust command and control (C2) system enables seamless communication between the UAV operator, the UGV operator, and the vehicles themselves. This facilitates coordination in mission planning and execution.

3.3 Collaborative mission planning

3.3.1 Shared objectives

UAVs and UGVs must have shared mission objectives. Collaborative planning involves defining roles, tasks, and responsibilities for each vehicle based on its capabilities.

3.3.2 Adaptive planning

Adapting the mission plan in response to changing conditions or unexpected events is crucial. Both vehicles should be capable of receiving and executing updated mission parameters.

3.4 Sensing and perception

3.4.1 Sensor fusion

Combining data from various sensors on both UAVs and UGVs enhances the overall perception of the environment. This can include visual cameras, lidar, radar, and other sensors.

3.4.2 Obstacle avoidance

Vehicles should be equipped with obstacle detection and avoidance systems to navigate through complex environments without collisions.

3.5 Localization and mapping

3.5.1 Shared maps

UAVs and UGVs should share a common map of the environment. This allows each vehicle to understand the other’s position, helping in collaborative decision-making.

3.5.2 Simultaneous localization and mapping

Advanced localization techniques, such as simultaneous localization and mapping (SLAM), enable the vehicles to map their surroundings in real time and localize themselves within the shared map.

3.5.3 Swarm intelligence

Coordination can benefit from principles of swarm intelligence, where multiple vehicles work together in a decentralized manner. This enables flexibility, scalability, and resilience in dynamic environments.

3.5.4 Dynamic formation control

UAVs and UGVs can coordinate their movements to maintain a specific formation, adapting to changes in the environment or mission requirements.

3.6 Data fusion and sharing

3.6.1 Data integration

Combining data collected by both UAVs and UGVs enhances overall situational awareness. Integrated data can be valuable for decision-making.

3.6.2 Real-time data sharing

Establishing a reliable data sharing mechanism ensures that information collected by one vehicle is quickly transmitted to others, facilitating coordinated response.

3.6.3 Redundancy and fault tolerance

3.6.3.1 Redundant systems

To enhance reliability, both UAVs and UGVs should have redundant systems. In case of a failure in one vehicle, the others can compensate and continue the mission.

3.6.4 Fault tolerance

The coordination system should be designed to handle faults gracefully, allowing the vehicles to adapt and continue functioning even in the presence of failures. Effective UAV–UGV coordination enhances the capabilities of autonomous systems, making them more versatile and adaptable to a variety of applications. This coordination is particularly valuable in scenarios where a combination of aerial and ground-based perspectives is required for comprehensive information gathering and mission execution.

4 UAV and UGV in medical waste transportation

Within a healthcare facility, regulated storing and transporting of medical waste strategies must be created. Safe transport and storage facilities ensure health and environmental safety. The medical waste disposal containers are classified based on color: red for biohazardous or sharp waste, yellow for chemotherapy waste and soiled and contaminated linen, black for Resource Conservation and Recovery Act (RCRA)-regulated waste, and blue for non-RCRA-regulated waste. Challenges accompany the handling of medical waste because it has impacts on the environment and risks to public health. Figure 1 shows some of these challenges and impacts.

FIGURE 1

FIGURE 1. Challenges associated with the transportation of medical waste.

Drones are revolutionizing medical waste transportation. A coordinated robot drone framework minimizes the transportation and occupational risks concerning the temporary storage of hazardous wastes at the medical centers, as shown in Figure 2. The framework minimizes the risks associated with the transportation of potentially dangerous microorganisms. The framework collects and transports infectious waste from scattered collection points to disposal centers.

FIGURE 2

FIGURE 2. Risks minimization with the CUU framework.

The flying operating time of drones is restricted by payload weight and battery capacity. Turbulence is also a hindrance. The latency-guaranteed multi-hop wireless communication system is considered for the flight of a drone moving from the line of sight (LOS) to beyond the line of sight (BLOS) with an obstacle avoidance facility. The obstacles may be trees, buildings, small hills, etc. The drone control mechanism comprises a multi-hop link or direct link to obtain the adaptive flying route (Kagawa et al., 2017).

5 Secure medical waste data

Efficient MW management must be developed to avoid dangerous microorganisms that affect public health and the environment. During the treatment and disposal of medical waste, pathogens, and toxins are released. It is mandatory to develop an optimal logistical solution for medical waste transportation to minimize contact with operators and health workers. Relevant data include optimal and secure MW transportation solution suggests the MW data structure as weight of the waste, the time of waste pickup, sensor state data, and order status.

Rq_mw: Medical waste shipment request.

T_p: Waste pickup time.

T_d: Waste disposal time.

W_mw: Weight of the waste.

Id_mw: ID of medical waste (created with Secure Hash Algorithm (SHA)-256).

Au_mw: Medical waste authorization.

O_mw: Location of medical waste.

Q_k = { Rq_k,mw, T_k,p, T_k,d, W_k,mw, Id_k,mw, Au_k,mw, O_k,mw}.

5.1 Hash key with SHA-256

In the context of hash functions, a “hash key” usually refers to the input (or key) provided to a hash function to generate a hash value. A hash function takes an input (or key) and produces a fixed-size string of characters, which is typically a hash code. SHA-256-bit is a cryptographic hash function that belongs to the SHA-2 family of hash functions. It produces a fixed-size output of 256 bits (32 bytes) and is commonly used for various security applications and protocols.

5.2 Proposed MW shipment process

A medical waste shipment process with the following steps has been suggested:

Input: Medical waste data

Create: Medical waste shipment request

Set: Rq_mw = 1

Load: Pick up MW packet by CUU framework

Time: Generate the timestamp of MW pickup

Carry: Ship the MW packet to a disposal center

Time: Generate the timestamp of MW disposal

Set: Au_mw = 1 after authorization

Encrypt: Do encrypt Q_k with a secure key

Output: Encrypted Q_k.

6 Proposed CUU framework for medical waste transportation

After it is collected from health centers, medical waste must be transported to the transit points. From transit points, the CUU framework comprising UGVs and multiple UAVs collects the medical wastes and transports them to the disposal centers Figure 3. The whole “seek” area is divided into subareas. Each subarea is assigned to a different drone. The flight time of each drone is estimated according to the charging of the battery. The control center acquires the drones’ real-time data, such as battery information, coordinates, and sensor data. The seek area is divided into the n unit cube parts with 1-m dimension. The WiFi communication zone comprises $(\frac{n}{2}+1)$ parts that are within the control range (Celtek et al., 2018).

FIGURE 3

FIGURE 3. CUU framework for medical waste transportation.

6.1 Problem formulation

An appropriate framework must be designed for the robot drone MW transportation system. One such framework is shown in Figure 3. Generally, drones have limited hovering time because of the battery capacity. The costs associated with the total distances are minimized. In this framework, a MW robot drone accomplishes the assigned tasks of MW transportation. The coordinated UGV–UAV MW transport problem is formulated as shown below:

$min{Z}_f=\left(a_{mr}{T}_{mr}+a_{dr}{T}_{dr}\right)+\left({C_{dr}W}_{mw}^{dr}+{C_{mr}W}_{mw}^{mr}\right)+\left(a_L{L}_{\mathit{ugv}}+a_H{H}_{\mathit{uav}}\right),(1)$

where T_mr and T_dr are the time of transportation of medical waste taken by a mobile robot and a drone, respectively. C_mr and C_dr are the cost per unit weight incurred in transportation of medical waste by a mobile robot and a drone, respectively. a_mr, a_dr, a_L, and a_H are the scaling factors. The $W_{mw}^{mr}$ and $W_{mw}^{dr}$ are weights of the medical waste that are being carried by a mobile robot and a drone, respectively. The maximum waste transportation capacities of a mobile ground robot and an independent drone are $W_{mw}^{mr,max}{W}_{mw}^{tr,max}$ and $W_{mw}^{dr,max}$, respectively. L_ugv and H_uav are the costs of UGV steering and UAV steering, respectively, and these are defined in the subsequent section.

Let dc_mw = {O_1,mw, O_2,mw, , O_m,mw } represent the set of all medical waste disposal centers and all the nodes in the network, where $O_{k,mw}=\left\{x_k^p,y_k^p,z_k^p\right\}$, and $x_k^p,y_k^p,z_k^p$ are the distances of medical waste disposal centers k from the transit point p. l_tr = { l_tr.1,… l_{tr. p} } be the set of transit points (TPs) that are to be visited by a truck for medical waste collection and disposal. Let $W_{mw}^{mr}=\{W_{mw,1}^{mr}{W}_{mw,2}^{mr},\dots .W_{mw,m}^{mr}\}$ be the weight of MW packets, where $W_{mw,1}^{mr}$ represents the weight of the first MW packet to be carried by a drone. The speed of the mobile robot and the independent drone are denoted as v_mr and v_dr, respectively (Celtek et al., 2018).

According to battery capacity (state of charge, SoC) and payload capacity W) of the independent drone, the predicted route of UAV is denoted as $Rdr\left(SoC,W_{mw}\right)$, and the predicted route of UGV is denoted as $Rmr\left(SoC,W_{mw}\right)$.

The predicted routes of the mobile robot and the drone comprise the set of nodes and are defined as shown below:

$R_{dr}={\displaystyle \sum _{i=1}^n}\left|\left|d_{i+1}^a-d_i^a\right|\right|,i=\in {\mathcal{R}}^{\mathit{n}},(2)$

$R_{mr}={\displaystyle \sum _{i=1}^n}\left|\left|d_{i+1}^g-d_i^g\right|\right|,i=\in {\mathcal{R}}^{\mathit{n}},(3)$

where ${\mathcal{R}}^{\mathit{n}}$ is the set of the cube in the travel space of the CUU framework; d_i+1 and d_i are the centers of the cube i + 1 and cube i in the ground space and the flying airspace of the CUU framework, respectively. Thus, the total route length of the MW transportation is R_mw = {R_mr + R_dr}. The times taken by the mobile robot (T_mr) and drone (T_dr) in medical waste transportation are shown below:

$T_{dr}=\frac{R_{dr}}{v_{dr}},(4)$

$T_{mr}=\frac{R_{mr}}{v_{mr}},(5)$

where v_dr is the average velocity of drone and v_mr is the average velocity of a mobile robot (Wang et al., 2019).

6.2 Proposed learning-based coordinated UAV and UGV framework

Drone path planning considers the zones at different altitudes with a total n number of waypoints $d_i^a=(d_1^a,d_2^a,d_n^a)$, and $d_i^g=(d_1^g,d_2^g,d_n^g)$ is the set of nodes to be traveled by UGV. Each waypoint and node is defined by the 3D coordinates d_i = (x_i, y_i, z_i), i = 1 … , n. An objective is to decide route lengths R_mr R_dr comprising a starting point, different nodes/waypoints, and a termination point. The following criteria are considered in path planning. Let another variable h_i be defined to represent both $d_i^a$ and $d_i^g$.

6.2.1 UGV steering

The cost of steering is defined according to the number of nodes of the path, as the more nodes, the higher the cost of steering. The cost of steering is defined as shown below:

$L_{\mathit{ugv}}=\frac{1}{\left|h_s,h_T\right|}{\displaystyle \sum _{i=1}^n}\left|h_i,h_{i+1}\right|,(6)$

where h_s and h_T are starting and terminating waypoints.

6.2.2 UAV steering

The cost of UAV steering depends on the altitude of the path. The cost will be higher if the altitude of the path is higher. At lower altitudes, the drone may fly with improved efficiency. The cost of UAV steering is defined as follows:

$H_{\mathit{uav}}=\frac{1}{\left|z_{\mathit{max}}-z_{\mathit{min}}\right|}{\displaystyle \sum _{i=1}^n}|\left(z_m\left(h_i,h_{i+1}\right)-z_{\mathit{min}}\right)|,(7)$

where z_m (h_i, h_i+1) is the mean flying height at segments (h_i, h_i+1). z_max and z_min are maximum and minimum altitudes, respectively. The following algorithm has been proposed for the coordinated steering of UAV and UGV. The weight of an edge between two waypoints is determined by Z_f, as shown in Equation (6.1).

Coordinated UAV and UGV steering algorithm is mentioned below.

Input: A weighted undirected graph G = (V, E, v)

create 3D coordinates $h=h_1,h_2,{\dot{h}}_{n}i=1\dots ,n$, in the steer (h_i+1, h_i)

h_i+1 ← steerUAVandUGV (h_i+1, h_i)

Sort the edges between two waypoints in E in decreasing order by weight and check mapping (h_i+1)

if path (h_i+1, h_i) is obstacle free

Check if Z_f decreases and r(t) increases

Return True (h_i+1)

Output: A minimum spanning tree for the UGV–UAV route

end

7 Proposed transfer learning mechanism for MW transportation with CUU

7.1 Bayes model

The Bayesian linear regression learning model implements the adaptive approach to enhance the performance of the predictive model with new data available along with available posterior data. The Bayesian learning model is described as follows:

${\pi }_i=\frac{e^{{\beta }_i{x}_i}}{\left(1+e^{{\displaystyle {\sum }_{i=0}^n}{\beta }_i{x}_i}\right)},(8)$

where x_i = [SoC_i, W_mw,i].

The likelihood function of ${Y=(h_1,h_{2,\dots \dots .},h_n)}^T$ can be obtained:

$f\left(Y\left|\right.\beta \right)=\prod _{i=1}^N{\pi }_i=\prod _{i=1}^N\left[{\left(\frac{e^{{\displaystyle {\sum }_{i=1}^n}{\beta }_i{x}_i}}{\left(1+e^{{\displaystyle {\sum }_{i=1}^n}{\beta }_i{x}_i}\right)}\right)}^{y_i}{\left(1-\frac{e^{{\displaystyle {\sum }_{i=1}^n}{\beta }_i{x}_i}}{\left(1+e^{{\displaystyle {\sum }_{i=i}^n}{\beta }_i{x}_i}\right)}\right)}^{y_i}\right].(9)$

As per the Bayesian theorem, the posterior distribution of the data can be obtained with the following. The predictive model of successful and unsuccessful delivery by drone incorporates the Bayesian inference, which is updated to improve its accuracy. The predictive model considers the new data Y₁ and the previous data set Y₂. The posterior distribution on model parameter β depending on data Y₁ and data Y₂ is formulated with the Bayesian theorem and given below (Xing et al., 2021):

$\mathrm{l}\mathrm{n}\left(\frac{{\pi }_{\mathit{optimal}}}{{\pi }_{\mathit{nonoptimal}}}\right)={\beta }_0+{\beta }_{1}SoC+{\beta }_2{W}_{mw}.(10)$

7.2 Multinomial logistic regression

The multinomial logistic regression represents the logistic regression extension of the class of many systems. The multinomial logistic regression model is developed based on the available data, and this can be used for the predictions based on previous data. The hyperparameter of the multinomial logistic regression must be tuned. The multinomial logit model is defined in terms of original probability π_i. The multinomial logit model algorithm for the probabilistic optimal cost of MW transportation is defined below:

Multinomial logit model algorithm:

Initialization: medical waste data

Generate data of transportation costs C_T = {C_mr, C_dr}, and route of transportation R_T = { R_mr: R_dr} and

Given multi-logit (Y_k|β) obtained for a past class of MW

Let $(\widehat{Y})$ for the new class of MW

Define prior $(\widehat{\beta })$ for a new class of $\widehat{Y}$

Procedure PARTITION (Z_f, Y) → partition of the data set into training and testing

Update the model using the training data set

Compute Z_f

Compute the maximum likelihood of Y as $f\left(Y|\widehat{\beta }\right)$ for Z_f

Obtain posterior distribution as $f\left(\widehat{\beta }|Y\right)=f(Y|\widehat{\beta })$

Randomly generate n samples from $f\left(\widehat{\beta }|Y\right)$

Update $\widehat{\beta }$ with the mean of n samples

Procedure EVALUATION Evaluating performance using the testing data set

Make a prediction $\widehat{Y}=MultiLogit(Y|\widehat{\beta })$

8 Reinforcement learning process of the framework

Let S and A represent a discrete set of environment states and a set of actions, respectively. In every state s ∈ S, a framework of multi-drone mobile robots takes feasible actions a ∈ A over the finite learning horizon. The framework transits to the next state s¹ ∈ S, and the framework receives an immediate reward for performing the desired action. However, the framework is penalized for each undesired action leading to a collision. The goal of the framework is to maximize its total reward. It does this by learning which action is optimal for each state. With available information of $<s,a,s^{\prime },r^t>$, the Q-learning process is updated according to the following equation:

$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+{\alpha }_t\left(s_t,a_t\right)\times \left[r_{t+1}+\gamma \underset{a_{t+1}}{\mathrm{m}\mathrm{a}\mathrm{x}}Q\left(s_{t+1},a_{t+1}\right)-Q\left(s_t,a_t\right)\right],(11)$

where ${\alpha }_t\left(s_t,a_t\right)(0<\alpha \le 1)$ is the learning rate and R_t+1 is the reward observed after performing a_t in s_t. The learning rate may be the same for all pairs, and the discount factor γ is such that 0 ≤ γ < 1. The Bellman equation that describes the optimal action-value function Q*(s, a) is given below:

$Q^{*}\left(s,a\right)=\underset{s^{\prime }\sim P}{\mathrm{E}}\left[r\left(s,a\right)+\gamma \underset{a^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}{Q}^{*}\left(s^{\prime },a^{\prime }\right)\right],(12)$

where s′ ∼ P represents the next state, which is sampled from a distribution P (.|s, a). This Bellman function starts the learning approach of approximately $Q^{*}\left(s,a\right).$ The set (s, a, r, s′, d) ∈ D represents the transitions, and d tells about the terminal of s′. To keep Q_Φ(s, a)

close to the Bellman equation, a mean square Bellman error (MSBE) is defined below:

$L\left(\Phi ,D\right)=\underset{\left(s,a,r,s^{\prime },d\right)\sim D}{\mathrm{E}}\left[Q_{\Phi }\left(s,a\right)-{\left(r+\gamma \left(1-d\right)\underset{a^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}{Q}_{\Phi }\left(s^{\prime },a^{\prime }\right)\right)}^2\right],(13)$

where d equals 1 if s′ is terminal; else, it is 0. In the target networks, the target term is defined below:

$r+\gamma \left(1-d\right)\underset{a^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}{Q}_{\Phi }\left(s^{\prime },a^{\prime }\right).(14)$

By minimizing the MSBE loss, it is possible to track this target by $Q_{\Phi }\left(s,a\right)$. An optimal action a* can be obtained by solving the following equation:

$a^{*}=arg\underset{a}{\mathrm{m}\mathrm{a}\mathrm{x}}{Q}^{*}\left(s,a\right),(15)$

where $a={[v_{mr},v_{dr}]}^T$. The reward function for the optimal trajectory of the drone is as follows:

$r_{dr}\left(t\right)={\eta }_d\times \frac{‖R_{dr}^p\left(t\right)-R_{dr}\left(t\right)‖}{‖R_{dr}^p\left(t\right)‖},(16)$

where $R_{dr}^p\left(t\right)$, R_dr(t) is the predicted optimal and actual route length of the drone, and η_P is the binary scaling factor.

$r_{mr}\left(t\right)={\eta }_m\times \frac{‖R_{mr}^p\left(t\right)-R_{mr}\left(t\right)‖}{‖R_{mr}^p\left(t\right)‖},(17)$

where $R_{mr}^p\left(t\right)$, R_mr(t) is the predicted optimal and actual route length of the mobile robot, and η_m is the binary scaling factor.The total reward function is defined below:

${r\left(t\right)=r}_{mr}\left(t\right)+r_{dr}\left(t\right).(18)$

Collecting medical waste manually may cause health problems for the employees of the medical centers. A coordinated multi-drone mobile robot framework can collect medical waste and transfer it to disposal centers with minimum human intervention.

9 Simulations and case study

A hospital located in Bareilly, India, is considered for a medical waste transportation study. Currently, medical waste is transported in trucks. This system could expose the people involved in MW transportation to risks. In some situations, unauthorized transportation may occur. The drone possesses a camera and an inertial measurement unit (IMU) to create reference elevation movement, and the robot generates a probabilistic network of paths. The ground robot and aerial drone work collectively to create the multi-level platform. The real-time appearance-based mapping technique is used to localize the robot and map the environment. The IMU estimates the attitude using multinomial logistic regression. Both visual and inertial data are combined to produce odometry estimates. The ground robot with four mecanum wheels is a custom-designed platform. The UAV creates a 3D map with a ground path and altitude information using the IMU and a camera. The ground robot follows the tracks within the 3D map of the environment generated by the UAV. Synergistically, the robot drone platform is equipped with sensors such as a PIR sensor, infrared camera, and Intel RealSense D435 with IMU. The ground robot moves on predefined paths to slow, stop, or reroute its path for collection of medical waste. The UAV is controlled in tandem to collect the packet of medical waste and then steer away for transportation of the packet to the MW disposal centers.

The types of medical waste are classified and represented by colors (red, yellow, blue, and black). The considered medical wastes comprises 110 gauze $(\sim 0.02kg)$, 120 gloves$(\sim 0.002-0.007kg)$, 140 infusion bags $(\sim 0.5-1kg)$, 135 infusion bottles $(\sim 3-5kg)$, 132 infusion apparatus$(\sim 0.001-0.004kg)$, 125 syringe needles $(\sim 0.02-0.004kg)$, 132 tweezers $(\sim 0.015-0.020kg)$, and 128 syringe (∼0.002–0.004 kg). The mobile robot helps to form the MW packets with weights ${W_{mw}^{mr}\le W}_{mw}^{mr,max}$ and $W_{mw}^{tr}\le W_{mw}^{tr,max}$. Python programming has been used to encrypt medical waste packets. Hash keys for these packets have been obtained and are mentioned below.

In a similar fashion, the next subsequent encrypted MW packet data are obtained. Let maximums of T_mr and T_dr be taken as 10 min and 25 min. The maximum velocities of the robot and drone are chosen as 5 km/h and 80 km/h, respectively. The various maximum values of transportation costs are taken as C_mr = 5 Rs/kg, C_dr = 50 Rs/kg, and C_dr = 40 Rs/kg. One hundred (100) samples of MW data are generated. After taking the mean velocities of the robot and drone at different time instants, the average velocities of the robot and drone are obtained as 2.48 km/h and 47.8 km/h, respectively. Next, the prediction error is calculated as 0.4%.

The posterior density was obtained with the transfer learning algorithm, and a predicted route for the mobile robots and drones is generated in a 3D environment Figure 4. Using the data related to the SoC of the battery and the MW weight, the proportional logit model is obtained:

$\mathrm{l}\mathrm{n}\left(\frac{{\pi }_{\mathit{optimal}}}{{\pi }_{\mathit{nonotimal}}}\right)=1.47+2.54SoC+1.24W_{mw}.(19)$

FIGURE 4

FIGURE 4. Predicted nodes of the drone’s route from the transit point to the disposal center.

The SoC of the battery of the drone and the weight of the medical waste packet are considered for the prediction of the optimal route length of the framework (18). The SoC of batteries varies from 0.2 to 0.8, and the weight of MW varies from 1 kg to 2 kg. The optimal predicted velocity of the robot and drone are shown in Figures 5 and 6, respectively (14)–(15). The rewards of actions done by robots and drones are obtained with a Q-learning mechanism using (16). The accumulative rewards are estimated as per (17) and shown in Figure 7. The coordinated framework has been rewarded for the optimal and successful delivery of medical waste from the collection point to the disposal center through transit points. At various iterations, the rewards vary from 0.7 to 0.9. The limitations of the various UAV–UGV coordination algorithms are mentioned in Table 1. None of these algorithms is applied in medical waste transportation. The effectiveness of the proposed learning-based UAV–UGV coordinated algorithm is shown by obtaining the results, and the features of this algorithm are mentioned in Table 2.

FIGURE 5

FIGURE 5. Predicted velocity (km/hr) of the mobile robot.

FIGURE 6

FIGURE 6. Predicted velocity (km/hr) of the drone.

FIGURE 7

FIGURE 7. Rewards of the proposed coordinated framework of the robot and drone.

TABLE 1

TABLE 1. Comparative study on UAV–UGV coordination algorithms.

TABLE 2

TABLE 2. Features of learning-based UGV–UAV coordination algorithm.

10 Conclusion

In this paper, a coordinated UAV–UGV framework is proposed for medical waste transportation to minimize human involvement. For optimal medical waste transportation, a learning-based mechanism has been implemented to predict the route length followed by the robots and the drone. The time-stamped medical waste packets are encrypted with hash keys. The predicted route length from the transit point to the disposal center has been obtained. The reinforcement learning algorithm has been considered for optimal successful and unsuccessful transportation of medical waste from collection points to disposal centers.

11 Future research direction

As the field of medical waste transportation continues to evolve, several potential future research directions can be explored to address emerging challenges and promote sustainable practices. Here are some areas that researchers might consider:

• Green technologies and sustainable practices: Examine and create eco-friendly technology for the transportation of medical waste. To lessen the influence of transportation on the environment, this could involve using hybrid or electric cars, renewable energy sources, and other environmentally friendly methods.

• Circular economy approaches: Examine medical waste management circular economy models, emphasizing transportation. To reduce total waste generation, investigate how waste materials and by-products can be recycled, repurposed, or reused in a closed-loop system.

• Integration of artificial intelligence (AI) and machine learning (ML): Analyze the possibilities for applying AI and ML to forecast transportation requirements, optimize medical waste transit routes, and enhance overall logistics effectiveness. Real-time tracking, predictive analytics, and data-driven decision-making may all be considered.

• Blockchain technology for traceability: Examine how blockchain technology can improve the transparency and traceability of the transfer of medical waste. Create safe, decentralized systems to monitor the flow of medical waste, guaranteeing adherence to rules and lowering the possibility of unauthorized disposal.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

DS: conceptualization, methodology, and writing–original draft. JL: resources, supervision, and writing–review and editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was supported by the funded project by the Council of Science and Technology, Lucknow, India (CST/D-3246).

Acknowledgments

The authors acknowledge the affiliating institutions of authors and the Council of Science and Technology, Lucknow, India.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Ahmad, R. W., Salah, K., Jayaraman, R., Yaqoob, I., Omar, M., and Ellahham, S. (2021). Blockchain-based forward supply chain and waste management for covid-19 medical equipment and supplies. IEEE Access 9, 44905–44927. doi:10.1109/ACCESS.2021.3066503

PubMed Abstract | CrossRef Full Text | Google Scholar

Arbanas, I. A. C. M. e. a., Antun, I., Car, M., Orsag, M., Petrovic, T., and Bogdan, S. (2018). Decentralized planning and control for uav–ugv cooperative teams. Auton. Robot. 42, 1601–1618. doi:10.1007/s10514-018-9712-y

CrossRef Full Text | Google Scholar

Celtek, S. A., Durdu, A., and Kurnaz, E. (2018). “Design and simulation of the hierarchical tree topology based wireless drone networks,” in International Conference on Artificial Intelligence and Data Processing (IDAP), USA, 21-22 Sept. 2019 (IDAP), 1–5. doi:10.1109/IDAP.2018.8620755

CrossRef Full Text | Google Scholar

Chen, J., Zhang, X., Xin, B., and Fang, H. (2016). Coordination between unmanned aerial and ground vehicles: a taxonomy and optimization perspective. IEEE Trans. Cybern. 46, 959–972. doi:10.1109/TCYB.2015.2418337

PubMed Abstract | CrossRef Full Text | Google Scholar

Ding, Y., Xin, B., Dou, L., Chen, J., and Chen, B. M. (2022). A memetic algorithm for curvature-constrained path planning of messenger uav in air-ground coordination. IEEE Trans. Automation Sci. Eng. 19, 3735–3749. doi:10.1109/TASE.2021.3135044

CrossRef Full Text | Google Scholar

Ha, C. S., and Lee, D. (2013). “Preliminary result of vision-based teleoperation of an uav and ugvs team,” in 10th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), China, Oct. 30 - Nov 02, 2013 (URAI). 128–128. doi:10.1109/URAI.2013.6677492

CrossRef Full Text | Google Scholar

Kagawa, T., Ono, F., Shan, L., Takizawa, K., Miura, R., Li, H.-B., et al. (2017). “A study on latency-guaranteed multi-hop wireless communication system for control of robots and drones,” in 2017 20th international symposium on wireless personal multimedia communications (New York: WPMC), 417–421. doi:10.1109/WPMC.2017.8301849

CrossRef Full Text | Google Scholar

Koçak, O., Kurtuldu, H., Akpek, A., Koçoğlu, A., and Eroğul, O. (2016). A medical waste management model for public private partnership hospitals. Med. Technol. Natl. Congr. (TIPTEKNO), 1. doi:10.1109/TIPTEKNO.2016.7863138

CrossRef Full Text | Google Scholar

Liu, Z., Xu, H., Zhao, X., Liu, P., and Li, J. (2019). Multi-attribute group decision making based on intuitionistic uncertain linguistic hamy mean operators with linguistic scale functions and its application to health-care waste treatment technology selection. IEEE Access 7, 20–46. doi:10.1109/ACCESS.2018.2882508

CrossRef Full Text | Google Scholar

Lu, C., Xian, X., and Hu, C. (2016). Health-care waste treatment technology selection using the interval 2-tuple induced topsis method. Int. J. Environ. Res. Public Health 13, 10127–10134. doi:10.3389/fnins.2013.12345

CrossRef Full Text | Google Scholar

Mishra, A. R., Mardani, A., Rani, P., and Zavadskas, E. K. (2020). A novel edas approach on intuitionistic fuzzy set for assessment of health-care waste disposal technology using new parametric divergence measures. J. Clean. Prod. 272, 122807. doi:10.1016/j.jclepro.2020.122807

CrossRef Full Text | Google Scholar

Shareefdeen, Z., Ashoobi, N., and Ilyas, U. (2022). Medical waste management and treatment technologies. Cham: Springer International Publishing. 113–133. doi:10.1007/978-3-030-95262-4_5

CrossRef Full Text | Google Scholar

Sun, S., Hu, J., Cao, Y., and Zhou, W. (2019). “Discussion on the application of rfid technology in medical waste management,” in 10th International Conference on Information Technology in Medicine and Education (ITME), South Koren, Aug. 23 2019 (ITME), 120–124. doi:10.1109/ITME.2019.00037

CrossRef Full Text | Google Scholar

Wang, C., Ma, Y., and Zhong, G. (2021). “Iot monitoring system of medical waste based on artificial intelligence,” in 2021 7th Annual International Conference on Network and Information Systems for Computers (ICNISC), USA, 23-25 July 2021 (ICNISC), 139–143. doi:10.1109/ICNISC54316.2021.00034

CrossRef Full Text | Google Scholar

Wang, D., Hu, P., Du, J., Zhou, P., Deng, T., and Hu, M. (2019). Routing and scheduling for hybrid truck-drone collaborative parcel delivery with independent and truck-carried drones. IEEE Internet Things J. 6, 10483–10495. doi:10.1109/JIOT.2019.2939397

CrossRef Full Text | Google Scholar

Wang, G. J., Guo, J., Tian, G., Chen, Y., and Huang, J. (2021). Event-triggered intervention framework for uav-ugv coordination systems. Machines 9, 371. doi:10.3390/machines9120371

CrossRef Full Text | Google Scholar

Wang, K., and Nai, W. (2021). “Application of 5g wireless communication technology in hazardous medical waste treatment,” in IEEE International Conference on Software Engineering and Artificial Intelligence (SEAI), China, 10-12 June 2022 (IEEE), 2021. 87–90. doi:10.1109/SEAI52285.2021.9477538

CrossRef Full Text | Google Scholar

Wang, Q., Lu, Q., Chen, H., Tian, J., and Su, Y. (2020). Path planning for uav/ugv collaborative systems in intelligent manufacturing. IET Intell. Transp. Syst. 14, 1475–1483. doi:10.1049/iet-its.2019.0688

CrossRef Full Text | Google Scholar

Waslander, N. K., Steven, L., Kartidjo, M., and Yoon, A.Kwang-Joonand Budiyono (2013). Unmanned aerial and ground vehicle teams: recent work and open problems. Japan: Springer, 21. doi:10.1007/978-4-431-54276-6_2

CrossRef Full Text | Google Scholar

Windfeld, E. S., and Brooks, M. S.-L. (2015). Medical waste management – a review. J. Environ. Manag. 163, 98–108. doi:10.1016/j.jenvman.2015.08.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Xing, W., Du, D., Bakhshi, A., Chiu, K.-C., and Du, H. (2021). Designing a transferable predictive model for online learning using a bayesian updating approach. IEEE Trans. Learn. Technol. 14, 474–485. doi:10.1109/TLT.2021.3107349

CrossRef Full Text | Google Scholar