Intelligent Predictive Maintenance and Remote Monitoring Framework for Industrial Equipment Based on Mixed Reality

The currently applied maintenance strategies, including Reactive and Preventive maintenance can be considered obsolete. The constant improvements in Information and Communication Technologies as well as in Digital Technologies along with the increase of computational power, have facilitated the development of new Artificial Intelligence algorithms to integrate cognition in computational systems. This trend is posing a great challenge for engineers, as such developments will enable the creation of robust systems that can monitor the current status of the machines and by extension to predict unforeseeable situations. Furthermore, Smart Computers will be capable of examining all possible scenarios and suggest viable solutions in a fraction of time compared to humans. Therefore, in this paper, the modelling, design and development of a Predictive Maintenance and Remote Monitoring system are proposed, based on the utilization of Artificial Intelligence algorithms for data acquisition, fusion, and post-processing. In addition to that, the proposed framework will integrate a Mixed Reality application for the intuitive visualization of the data, that will ultimately facilitate production and maintenance engineers to monitor the condition of the machines, and most importantly to get an accurate prediction of the oncoming failures.

Introduction

Maintenance of industrial equipment as a part of the manufacturing lifecycle, approaches 60–70% of the overall cost of production. Therefore, being able to predict and perform machine maintenance operations in a short period of time can lead to successful troubleshooting, and at the same time increase the availability of machine tools (Mourtzis et al., 2015). Currently, inadequate maintenance techniques can reduce the total productive ability of the plant by between 5 and 20% (Wollenhaupt, 2016). Traditionally, maintenance professionals have combined several techniques, both quantitative and qualitative, with the aim to anticipate potential problems and alleviating downtime in their production plants. Predictive maintenance gives them the potential to optimize maintenance tasks in real time, maximizing the useful life of their equipment while avoiding disruption of operation. Recent studies also show that unplanned downtime is costly, with an estimation of $50 billion per year for global producers (Deloitte, 2017a). In the Industry 4.0 environment, maintenance should do much more than simply prevent the downtime of individual assets. Predictive maintenance increases uptime by 10–20%, while reducing overall maintenance costs by 5–10% and maintenance planning time by 20–50%. Furthermore, due to increased interconnectivity and new opportunities for collecting, processing, and analyzing information, predictive maintenance can be a very powerful strategy (Deloitte, 2017b).

In addition to failure prediction, a significant challenge is the implementation of reliable and error-free maintenance operations and, as a result, the constant validation of fully working equipment as soon as possible. To that end, a significant amount of development work has been made to design and improve real-time technical service systems and software focused on mobile apps to prevent unwanted errors and malfunctions (Masoni et al., 2017). Moreover, the handling of complex cases of smart factories, intelligent maintenance, self-organized adaptive logistics, customer-integrated engineering and smart factory architectures require the integration of production data into modeling that can only be achieved with the use of advanced simulation and Information Technology (IT) (Mourtzis, 2020). Moreover, the value of products will eventually be focused on their software parts not on their specification or implementation functions under the Product Service Systems (PSS) paradigm (Mourtzis et al., 2018). With the advancement of Information and Communication Technology (ICT) and cutting-edge technologies such as Mixed-Reality (MR), Augmented Reality (AR) and Virtual Reality (VR), the academic domain is expanding this strategy by leveraging the advantages of AR for data visualization during maintenance operations (Mourtzis et al., 2017; Mourtzis et al., 2018; Palmarini et al., 2018). Emerging technologies such as the Internet of Things (ΙοΤ), cyber-physical networks and cloud computing have enabled the processing of vast volumes of tracking data, which is intended to significantly increase manufacturing productivity (Tao et al., 2018; Fantini et al., 2020). However, as a core topic in prognostics and health management, the remaining useful life (RUL) prediction based on monitoring data ca be used to prevent a failure triggered (Lei et al., 2018). RUL prediction is thus a hot topic that has drawn more and more interest in recent years (Yang et al., 2019). Most research studies on intelligent prognosis and health management (PHM) analysis using data-driven approaches by deducing correlations between data from different sensors ( e.g. accelerometers, acoustic energy emission) to determine the remaining useful life (e.g. accelerometers, acoustic energy sounders, etc.). In order to limit the complexity inherent in the dynamic updating of online data, Machine Learning has arisen as a way of analyzing vast volumes of data for statistical purposes. Especially in the implementation of neural network-based techniques, complex multidimensional non-linearities can be used for automated learning, allowing for efficient processing of data features in an attempt to provide optimized solutions (Vogl et al., 2019).

Having identified the above-mentioned challenges, this research work presents the design and development of a predictive maintenance framework for industrial equipment. Further to that the contribution of this research work extends to the presentation of a custom Data Acquisition (DAQ) device and a framework for processing the data via the Digital Twin of the equipment for the calculation of Remaining Useful Life of critical components. The remainder of the paper is structured as follows. In State of the Art the most pertinent literature is reviewed, and commercial devices are compared. In Proposed System Architecture, the proposed system architecture is presented. In System Implementation the practical implementation steps are discussed. Then in Case Study, an experimental case study that has derived from Industry is presented and the results are discussed. Finally, in Concluding Remarks and Outlook, the paper is concluded, and future research points are discussed by the authors.

State of the Art

Machine Learning

Among the latest trends in the modern manufacturing world, is the so-called AI. An also well-known subset of the above-mentioned concept is Machine Learning (ML). Concretely ML algorithms are defined as computer-based algorithms that improve their efficiency through experience, i.e. data processing (Mitchell, 1995). Globally the AI adoption is surging at enormous rates, as it becomes apparent in the report presented in (Hupfer, 2019), from where it can be concluded that AI adoption marked a surprising 270% increase in a timespan of 4 years along with an increase in global spending of around 80 billion dollars Figure 1.

FIGURE 1

FIGURE 1. Machine Learning Types & algorithms (Adapted from Ahmed and Khan, 2019).

Additionally, Deep Learning (DL) techniques have been applied for the integration of systems in edge computing, setting edge nodes in edge services and terminal devices, using DL architectures for predictive analysis with quick preprocessing and accurate performance classification to assess the life expectancy of components. A classification of the most common DL frameworks is as follows:

Neural Networks (NN) (Chryssolouris, 2006; Chen et al., 2019)

Deep Neural Networks (DNN) (Zhao et al., 2017)

Convolutional Neural Networks (CNN) (Li et al., 2018; Mourtzis et al., 2020a)

Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) (Zhao et al., 2017)

Gated Recurrent Units (GRU) (Chen et al., 2019)

Recurrent Neural Network (CNN-RNN) (Banerjee et al., 2019)

Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) (Kong et al., 2019)

Convolutional Neural Network and Gated Recurrent Unit (CNN-GRU) (Lei et al., 2018)

As Industry 4.0 continues to evolve, many companies are struggling with the realities of AI implementation. Indeed, the benefits of PdM such as helping determine the condition of equipment and predicting when maintenance should be performed, are extremely strategic. The implementation of ML frameworks can lead to major cost savings, higher predictability, and increased availability of the systems. Therefore engineers have focused their efforts on the development of new technologies and techniques for facilitating the prediction of manufacturing equipment malfunctions and therefore to further optimize existing maintenance policies as well as to introduce more adaptive maintenance policies. PdM can be defined as a series of processes, where data is collected over time in order to monitor the state of equipment, in a manufacturing system. Ultimately, the goal is to identify/recognize patterns that in turn will facilitate engineers to predict and ultimately prevent failures (Rezaeianjouybari and Shang, 2020). Some of the most common problems that can be addressed with PdM include, the calculation of Remaining Useful Life (RUL), which aims at the suitable scheduling of future Maintenance and Repair Operations (MRO), Flagging Irregular Behavior, which is based on anomaly detection by the utilization of time series analysis, and Failure Diagnosis and Recommendation of Mitigation after failure (Lei et al., 2018; Mourtzis et al., 2020a). While certain Facility Managers perform PdM, this has been done traditionally by using Supervisory Control and Data Acquisition (SCADA) systems set up with human-coded/hard-coded thresholds, alert rules, and configurations. However, this is a semi-manual approach that does not take into consideration the more complex dynamic behavioral patterns of the machinery, or the contextual data relating to the manufacturing process, thus lacking adaptability relative to the current status of the industrial equipment (Nicholson et al., 2012). What is more, in recent research works SCADA systems are integrated with ML algorithms, in order to extend their usability as well as to shift towards prognostics (Pang et al., 2020; Ruiming et al., 2020; Zhang and Lang, 2020). In the research work of Wang et al. (2020), the authors have developed a framework based on Convolution Auto Encoder and Long-Short Term Memory (LSTM) in an attempt to estimate RUL more accurately in comparison to conventional methods. For the recognition of patterns, which facilitates the process of building the predictive model, data exploration techniques must be utilized so that the engineer can determine whether the data includes degradation or failure patterns (Erfani et al., 2016; Li et al., 2019).

Remaining Useful Life

As the name indicates, Remaining Useful Life, also referred to as RUL, describes a wide variety of algorithms which aim to predict the remaining life of assets and/or their components, ultimately developed under a predictive maintenance framework. According to Baru (2018) there can be identified three basic techniques regarding the calculation of RUL based on the data that are available, namely lifetime data, run-to-failure data, and known threshold data. An interesting approach in presented in (Loutas et al., 2013) for the calculation of RUL for rolling bearings based on the utilization of ε-Support Vector Regression (ε-SVR), concluding that linear models cannot provide accurate results since there is non-linear between the features extracted by the data spectral analysis and the RUL prediction. Another aspect of the usefulness of RUL estimation is presented by (Sun et al., 2020). The authors have implemented a framework for the RUL calculation of cutting tools, thus managing to increase the environmental sustainability of the cutting tools by 8.39% per flute. From the investigation of the available literature it can be concluded that the estimation of RUL is a challenging topic, requiring exhaustive data processing. Further to that in the majority of the publications it is implied that linear approximations regarding the degradation of the physical system are not sufficient in terms of accuracy, as presented in (Yang et al., 2021), where the authors investigated the prediction of RUL in induction motors. The authors in (Wen et al., 2021) has proposed a generalized methodology for the prediction of RUL based on the fusion of multiple signals. It is worth noting that they achieved an increase in terms of accuracy of approximately 10 percent. Kozjek et al. (2020) have also presented an interesting research work on the prediction of RUL with the utilization reinforcement learning, which is compared with two other algorithms, indicating promising results. In the research work of (Liu et al., 2019), a RUL prediction framework is proposed based on Health Index comparison, making it suitable for cases where there is limited amount historical data. It is stressed out that the topic of RUL prediction is still challenging for engineers and by extension there is plenty of room for improvement. In addition to that, the use of Digital Twin could compensate the lack of raw data from machines, with the generation of fault datasets.

Extended Reality

Among the latest developments of the current industrial revolution, advances in high-end digital technologies are entailed, including Extended Reality (XR). In its essence, XR is an umbrella term, often used by engineers and researchers around the world, in order to describe technologies such as Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) (Mourtzis et al., 2020b). The two former technologies are very close, since they are based on the partial immersion of the user to a virtual environment, while the latter, implies the total user immersion in a virtual, computer generated environment. In addition to that, what differentiates AR from MR is the fact that MR is based on the user interaction with the digital information, also known as holograms in that case (Fast-Berglund et al., 2018). The use of AR in maintenance is an important aspect that has to be further researched under the Industry 4.0 framework. Since new technologies are constantly becoming available, existing techniques could be leveraged so as to increase the efficiency of maintenance tasks, minimize the errors and the risks imposed in such operations. As presented in the research work of Vorraber et al. (2020), both maintenance technicians and experts are keen on integrating AR and MR solutions in their line of job, in order to achieve better communication and most importantly to limit the complexity of the maintenance procedures. Although the maturity level of AR applications has increased during the last decades (Mourtzis et al., 2020c), there are constantly arising new challenges, such as the integration of Predictive Maintenance and AR/MR so that digitalization of the manufacturing processes becomes a reality (Wolfartsberger et al., 2020). Further to that in two recent systematic literature reviews, presented by Palmarini et al. (2018) and Egger and Masood (2020) the current implementations of AR are based on manual solutions and the use of Predictive Analytics/Prognostics has not been yet faced, thus indicating that there is fertile ground for further research in that field.

Proposed System Architecture

In the following paragraphs the proposed system architecture will be discussed in detail. The key aspects of the proposed methodology are the DAQ device, which conforms to the latest IoT standards. However, in order to efficiently monitor the status of professional refrigeration systems, they have to be analyzed into two subsystems, namely the cooling chamber of the refrigerator and the compressor compartment. These two subsystems often are not located in the same room/building, thus require different DAQ devices to be installed. By the virtue of the diversity of installed sensors, crosschecking the measured values, is enabled and therefore more accurate predictive models can be trained. The general architecture of the proposed is depicted in Figure 2.

FIGURE 2

FIGURE 2. Proposed System Architecture.

Data Acquisition Device

In this section the architecture of the framework for the DAQ device will be discussed. For the DAQ module, two main aspects will have to be investigated, namely the DAQ device and the communication interfaces as well. The development of the DAQ device is based on the design of a custom circuit board in combination with an Arduino micro-controller which incorporates all the required modules for the data acquisition from the sensors attached to the board, the pre-processing of the data, an interface for user interaction and a wireless network module for the data transmission to the Cloud Database. In order to make the DAQ device adaptive to the customer needs, and subsequently to support a wide variety of configurations, the sensor modules are not hardwired/soldered on the main PCB of the DAQ device. More specifically, the PCB supports wired connectivity, through 3.5 mm jack ports for the sensors. However, in order to enable the communication between the DAQ device (Fig. 3A) and the Cloud Database an RF-based Wireless Sensor Network (WSN) is utilized. For the setup of the WSN, XBee modules are utilized. More specifically, an XBee module (Figure 3B) is installed on each of the DAQ devices and another one is installed on a computer which acts as the network coordinator. For the correct communication of the DAQ devices to the computer, each RF module is tagged. Furthermore, during the data transmission, the data packets are also including the tag of each RF tag, so that the received data can be correlated to the corresponding machine.

FIGURE 3

FIGURE 3. Prototype DAQ device.

As far as the sampling rate is concerned, the DAQ device collects feedback from the installed sensors on a varying rate. The sampling rate for the accelerometer sensor is set to one (1) second or 1 Hz. As far as the sampling rate for the temperature sensor is set to 5 s and for the pressure sensor is set to 10 ms (milli-second) as soon as a surge event is detected. However, if the customers require a different resolution regarding the data collection and by extension the estimations made by the framework, then they adjust the sampling rates. In Figure 3, the prototype board for the DAQ device is presented.

Digital Twin Development

In order to create a suitable framework for the predictive maintenance functionality of the equipment, the design and development of a Digital Twin is required. The aim is to analyze the data gathered from the DAQ devices and based on the simulation model to predict future equipment malfunctions. Therefore, in the case of the refrigerators, the physical model is created in the Simulink programming environment. For the simulation of the model, MATLAB is also utilized for handling the imported data as well as setting up the simulation parameters. The physical model of the refrigerator is fully parametrically designed so that it can be adapted to the technical specifications of the physical system. In Figure 4, the developed model within the Simulink environment is presented.

FIGURE 4

FIGURE 4. Refrigerator model in MATLAB Simulink.

It is stressed out that the model consists of several subsystems, or else functional blocks, in an attempt to increase the resolution of the simulation model, such as the compressor, the evaporator, the condenser, and the refrigerator compartment. In the “Data Input” block, the data from the DAQ device are imported to the model. Then the standard refrigeration cycle for refrigeration is run and the results are plotted. Through the plots, crucial parameters of the refrigerator, such as temperature, power consumption and pressures within the refrigerant distribution network can be observed. For the simulation, the fluid properties of the R134a refrigerant were also imported in the model. As a result with the proposed methodology, it is possible to predict future asset malfunctions based on the simulation of the refrigeration cycle and plan accordingly their production schedule so that the equipment downtime is further minimized. In addition to that, the simulation results are also combined/fused with the data gathered from the physical machine so as to predict the RUL of specific components of the equipment. The usability of the Simulink model extends also to the generation of fault data. For instance, in the refrigerators a common failure is the loss of pressure in the refrigerant distribution network. Therefore, in the existing model the “fault” is simulated with the addition of an array of blocks, based on which the differential pressure in specific subsystems, such as the compressor pressure differential is offset to a fault value. Then after the corresponding datasets for the healthy state and the fault state have been generated, the model automatically recalls the RUL algorithm.

In its essence the RUL algorithm utilizes data from both the digital twin and the physical model for the prediction of the time, in hours, before maintenance is required. The first step in the RUL algorithm is the Fourier transformation of the signals to the frequency domain. The next step is the creation of the spectrograms for each of the under-examination parameters, e.g. pressure inlet and outlet in the compressor, vibration signal from rotational components. In this step, two spectrograms are created, one for the fault data and one for the healthy data. Based on the spectrograms of the faulty and the healthy datasets, features can be extracted and classified for future use via the use of a Support Vector Machine (SVM). Therefore, boundary conditions can be formed for the under-examination parameters. As soon as the above-mentioned model is trained, then the model is constantly running and gets updated at a regular basis, given that there are new data posted on the Cloud database. In an attempt to generate a fault dataset, modification of the Simulink model is required. The modification involved the creation of additional subsystems which are used for the simulation of faults. Ultimately, the goal of this experiment series is to generate fault datasets, i.e. datasets containing measurements of the physical model operating under malfunction. For the generation of the fault datasets, a pressure drop in the refrigerant network/piping was simulated and increased humidity within the cooling chamber. In order to process the data derived from the simulation runs, the outputs were transformed via Fourier Transformation, in order to represent the events in the frequency domain. Then, with the use of spectrograms, useful features were extracted and based on these features, with the use of a Support Vector Machine, the faults could be classified.

Augmented Reality Module

An AR module is provisioned in order to facilitate the monitoring process of the industrial equipment. This module can be realized as a multi-platform application, from which the customers can either remotely or on-site visualize crucial information about their equipment and interact with it, easily and intuitively with the use of this cutting-edge digital technology. Concretely, the current implementation of the framework supports handheld Android-based platforms, e.g. mobile phones and tablets, as well as Head Mounted Displays (HMD), such as the Microsoft HoloLens.

User tracking and pose estimation for the Android-based devices is based on the recognition of a feature-rich image target, as in the one presented in Figure 5. Further to that, in Figure 5A the physical form of the image target, whereas in Figure 5B, the features recognized by the device are overlaid on the image target.

FIGURE 5

FIGURE 5. Example of Image Target used in Android-based AR applications.

As soon as the image target is recognized by the device, through the integrated camera, then the transformation matrix, denoted as T, between the camera and the marker is Eq. 1:

$x_c=T\ast X(1)$

Where:

$x_c$ is the projection of a point in ideal image coordinates.

T is the pose matrix.

X expresses the points in world coordinates.

Therefore for the calculation of $x_c$ a 3 × 3 rotation matrix is utilized, denoted by R, as per the Eq. 2.

$x_c=\left[R\text{|}t\right]\ast X\Rightarrow \left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{cc}\begin{array}{ccc}{r}_{\mathrm{1,1}}& r_{\mathrm{1,2}}& r_{\mathrm{1,3}}\\ r_{\mathrm{2,1}}& r_{\mathrm{2,2}}& r_{\mathrm{2,3}}\\ r_{\mathrm{3,1}}& r_{\mathrm{3,2}}& r_{\mathrm{3,3}}\end{array}& \begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\end{array}\right]\left[\begin{array}{c}\begin{array}{c}X\\ Y\end{array}\\ \begin{array}{c}Z\\ 1\end{array}\end{array}\right](2)$

Afterwards, in order to translate the result in pixel coordinates, denoted by $x_{pix}$, i.e. as a 2D representation, calibration matrix is used, denoted by C. Consequently, $x_{pix}$ based on Eq. 3 becomes:

$x_{pix}=K\ast x_c\Rightarrow \left(\begin{array}{c}{x}_{pix}\\ y_{pix}\\ 1\end{array}\right)=\left[\begin{array}{cc}\begin{array}{ccc}f& 0& p_x\\ 0& f& p_y\\ 0& 0& 1\end{array}& \begin{array}{c}0\\ 0\\ 0\end{array}\end{array}\right]\left(\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right)(3)$

Based on the pose estimation steps described in the previous paragraph, for the registration of the AR content using Android-based platforms, a fiducial image target is required. As soon as it is recognized by the device camera, then by calculating the user’s position and pose, the augmentations are overlaid on the physical environment. What is more, in order to enhance the user experience, the application supports the functionality called “Extended Tracking”, based on which, the handheld device can continue overlaying the augmentations in the physical environment in the event of the camera loosing direct contact with the image target.

However, for the implementation of the AR module in the Microsoft HoloLens HMD, the user pose estimation is approached in a different way, as the HMD is integrated with four (4) greyscale tracking cameras. As a result the Depth, denoted by D, is calculated with the use of Eq. 4.

$D=\frac{R}{\sqrt{U^2+V^2+1}}(4)$

Where:

 D Is the Depth R is the Range, which is measured from the integrated  HoloLens ToF (Time-of-Flight) camera.

 U and V are the distance values of a certain point.

Therefore the depth value into real, 3D world coordinates can be derived from Eq. 5.

$\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=D\left(\begin{array}{c}U\\ V\\ 1\end{array}\right)(5)$

Consequently, for the user pose estimation in the case of Microsoft HoloLens, the developed application initially prompts the user to select/setup an initial point of reference. This reference point is then automatically translated into a 3D world anchor based on which the AR visualizations are positioned and rendered in the physical environment.

One of the main aspects of the AR module/application, is the condition monitoring of the assets. This can be done in two ways. The first solution is for remote monitoring, where the responsible engineer uses either device to visualize a scaled 3D model of the refrigerator, upon which important information are displayed. The second solution is for monitoring the condition of the asset while inspecting it physically. In the case of the refrigerators, which is presented in the following paragraphs, it is impossible for the responsible engineer to monitor the current status/health of the refrigerator group without having to physically inspect the compressor unit, which is located away from the cooling chamber. However, with the proposed framework, it is possible to recognize the refrigerator, by utilizing the image targets, discussed previously and retrieve data for the corresponding refrigerator group from the Cloud database. As a result, the equipment/asset inspection can be performed in near-real-time.

Another aspect of the AR application is the provision of a communication tool, which enables the communication between the OEM and customer, in order to inspect the equipment in real time, and in addition to that to create basic AR instructions also in real time, by performing common “drag and drop” operation in the field of view of the user. Further to that, this functionality enables both the OEM and the client to communicate via a video call session, where the OEM can visualize the field of view of the user and with the use of basic 3D tool representation, the client can perform maintenance tasks in real time. The above-mentioned functionalities are based on the adaptation of the methodology presented in the research work of Mourtzis et al. (2020b)Figure 6.

FIGURE 6

FIGURE 6. (A) AR visualization of compressor and its working parameters; (B) Cooling chamber cluster and the current working conditions.

System Implementation

The proposed architecture can be realized as a multi-sided application. The first aspect of the application is a desktop-based application, which communicates with the server in order to retrieve the data from the server and process them through the predictive algorithm. The predictive algorithm is responsible for the identification of patterns within the processed data. Each of these patterns represents a classification of the possible situations of the under-examination machine, or cluster of machines. A predictive algorithm will have to analyze the data gathered from the sensors so that a prediction of unforeseeable machine malfunctions can be identified. However, since the data are available on the server, it is of great importance to create an application for monitoring the current situation of the machines.

From a software point of view, for the setup of the DAQ device, the Arduino IDE (Arduino, 2020) was used. Moreover, for the setup of the WSN the X-CTU (X-CTU, 2020) application from Digi has been utilized. For the development of the Graphical User Interfaces (GUI), a Universal Windows Platform (UWP) (Microsoft, 2018) application has been developed. The benefits of using UWP is the multi-platform implementation, the ease of configurability, ease of implementing security protocols, serviceability of the framework and updates’ distribution. In the following paragraphs the functionalities and the GUIs designed and developed so far will be discussed in detail. In order to do so, the Unity 3D game engine is utilized (Unity, 2020). As regards the code scripts, the Microsoft Visual Studio IDE is used (Microsoft, 2020). More specifically, for the development of the main functionalities of the application, the code scripts are written in C# programming language. Since the application supports two user groups, one for the OEM supplier/service provider and one for the customers, a common login/register system is implemented. Upon installation of the application on the end-user’s desired platform, the application prompts the user to register an account which automatically saved in the Cloud database. Therefore each time the user is connected, after getting authorized, their user group is automatically retrieved by the Cloud database and the suitable GUIs are loaded. It is stressed out that although the development of a UWP application enables multi-platform support, the AR functionalities are only available for handheld devices, such as Android-based mobile phones and tablets, and Head Mounted Displays (HMD) such as Microsoft HoloLens.

As long as the login is successful, if the user is listed as a client, then the available options are to create a new cluster of refrigerators or process/monitor an existing cluster. The “Set Up New Line” functionality is targeted for new customers, or customers that acquired new equipment, i.e. new refrigerator group(s) or new DAQ device(s), as presented in Figure 7A.

FIGURE 7

FIGURE 7. (A) Settings for a new refrigerator line; (B) Settings for new/old sensors.

The core functionality of the developed framework lies within the monitoring functionalities. In the corresponding GUIs, the customer can visualize in 3D a scaled version of their refrigerator group and upon request to visualize the available information, which are automatically fetched by the Cloud database. Then, in continuation, if requested, the data can be transformed into statistical figures, so that the client can visualize the current status of their equipment. All of the above-mentioned data can also be visualized in the form of augmentations in case the customer is close to the refrigerator. In order to further notify the customer about an upcoming maintenance action or if any piece of equipment requires special attention, certain alerts have been implemented as presented in Figures 6 and 7. Regarding the communication interfaces between the DAQ device, the end-user application and the Cloud Platform, RESTful API services have been developed. As regards the communication interface between the DAQ device and the Cloud Database, the DAQ device as a client can send POST, and PUT requests to the Cloud Platform, so as to enable the data to be posted on the database if they are not existent, or updated as needed. On the contrary, the majority of the services implemented on the end-user application send only GET requests, in order to fetch data to the end-user’s device, with the exception of user registration, where a POST request is sent in order to record the user’s data on the database. However, due to the large volumes of sensitive data that are circulated within the proposed framework, a set of security measures are taken. Initially, all the HTTP requests are of type HTTPS (Hypertext Transfer Protocol Secure). Secondly, DELETE requests are not allowed for anyone trying to connect to the Cloud Platform. Therefore, in order to delete any records from the database, this process has to be undertaken manually by the authorized system administrator. In an attempt to make the proposed framework more general, a custom editor has been developed for supporting the functionalities of the framework itself. More specifically, the development team assisted by the editor can create virtually any configuration of systems and functionalities, so that the framework can be adapted to the actual needs of the corresponding company.

From a hardware point of view, a desktop PC has been utilized for the development of the application as well as the Cloud Database and its services. For the implementation of the developed AR-based application, an Android-based tablet and a Microsoft HoloLens HMD are used. As regards, the DAQ device, an Arduino Mega 2560 microcontroller is paired with the custom board presented in the previous paragraphs. In addition to that for the XBee RF modules, the Arduino XBee shield and the Adafruit explorer shield were used.

Case Study

In manufacturing systems the profit is derived by the subtraction of operating costs from total income. Therefore, the profit becomes a problem with two possible solutions, either the minimization of operating costs, or the maximization of income. As regards, operating costs, industrial equipment maintenance costs are also included. Production equipment affects operating costs with machine deterioration and failures as well.

The applicability of the developed framework has been tested and validated in a real-life industrial scenario,derived from an OEM supplier of professional refrigerator systems. The OEM is looking forward to transforming their business model based on the PSS paradigm, by providing the proposed framework as a service, in an attempt to improve their after-sales policy. What is more, it is estimated that based on the monitoring and analysis from the AI algorithms, the engineering department of the OEM will be able to gather insightful feedback, aiming at the improvement of the design process and the quality of their products. The main benefit of the proposed methodology, is that it can provide time estimations about future equipment malfunctions, which by extension can enable both the OEM and the client to act proactively and in time, in order to further minimize the equipment downtime. Further to that, with the provision of the AR application knowing beforehand the upcoming equipment failures, can facilitate maintenance engineers to prepare the AR content timely and communicate it to the client. Therefore, the need for an external maintenance technician is further minimized, thus reducing both the overall time and cost of maintenance.

In order to test and validate the proposed framework, the DAQ device was installed, on an experimental refrigerator group located at the premises of the OEM, used for test purposes. It is stressed out that the OEM has already integrated sensing systems on the majority of their products for monitoring purposes. However, the existing solution is wired and requires a computer and an engineer close to the refrigerator compartment, in order to monitor their status. Therefore, the first step was the installation of the DAQ device, presented in the previous paragraphs as well as the setup of the required WSN network. The WSN follows the star topology, meaning that the one XBee is connected to a PC and acts as the WSN coordinator. Then, each DAQ connects to the coordinator and transmits the data at the defined rate. Afterwards, in order to handle the data arrived at the WSN coordinator, the corresponding COM port is listened by the PC via a Python script and the data are uploaded to the database and saved within the corresponding CSV file. Then the data saved in the Cloud database are automatically input to the Digital Twin of the refrigerator in order to simulate its condition and calculate the RUL for the critical components. Based on this setup the monitoring and simulation runs were executed at the premises of the case study provider. For the purposes of the experiments, a scenario of compressor malfunction has been examined. For the maintenance of the refrigerator in such case, the inspection of the equipment from an OEM technician would require travelling and inspection which according to the OEM it would account for approximately 28 h, the order and acquisition of the replacement compressor would require approximately 24 h and the installation on the refrigerator group would require approximately 5 h. However, with the adoption of the proposed framework, and based on the calculation of the compressor RUL, the client is capable to order the failing part timely, thus minimizing the waiting time. Further to that, with the utilization of the AR application the inspection of the equipment can be facilitated, thus eliminating the need for an OEM maintenance technician to visit the client. The time for inspection was calculated to be approximately 5 h. In the following figure, the time estimations for the current situation as well as the corresponding times with the adoption of the proposed methodology are presented Figure 8.

FIGURE 8

FIGURE 8. Time distribution per task, as-is situation vs proposed

Concluding Remarks and Outlook

The scope of this research work was to present the latest trends regarding the fields of predictive maintenance and XR and furthermore to propose a novel framework that will facilitate engineers to constantly monitor the status of the manufacturing equipment and in advance to predict the forthcoming maintenance activities. By extension, the prediction of malfunctions will enable companies to schedule their production more efficiently, whilst it makes them more adaptive to any disturbances caused within the company limits. From the practical implementation of the developed framework in the industrial partner, it became evident that the refrigerator downtimes can be reduced by approximately 20%, since both the clients and the OEM were capable to monitor the status of their equipment and by extension, with the use of the AI algorithm, the RUL prediction for crucial components of the refrigerator system, the client got a trustworthy estimation of when their equipment should be maintained. In addition to that, since the customer, could get an estimation of the upcoming failure, they are able to schedule a maintenance session with the OEM much faster and fitted to both ends’ schedule without creating great disturbances. An equally important finding is that the maintenance costs can be reduced by approximately 10% since the OEM can order and acquire the needed components beforehand, thus eliminating overnight delivery costs. Along with that, by predicting and scheduling timely the maintenance session, the equipment is not left to run until failure, which could affect the operation of other components, thus leading to increased maintenance cost, due to additional technician labor and extra replacement parts.

Although the development and the implementation of the proposed framework have yielded promising results, there are several implications that must be addressed before such solutions reach an acceptable maturity level and by extension, become commercially available. The most important implication faced, is that the calculation of the RUL cannot be performed in real-time thus inducing a certain amount of latency in the AR visualizations. The amount of latency is affected by two major issues, first the network speed and second the computational power of the system handling the Digital Twin. Another implication is the authoring of the AR visualizations in the field of view of the end-users. Currently, this is a manual task, which in the future must be addressed, so that AR applications can become useful tools in the modern manufacturing environment rather than increasing the complexity of the systems, by increasing the time and effort to prepare the so-called “AR scenes”.

In the future, the Digital Twin will also be improved. It is estimated that following the implementation of the proposed framework in similar equipment, i.e. a fleet of assets, will enable the creation of data ensembles. The idea behind this is to utilize similar datasets in order to improve the predicting accuracy of the Digital Twin.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

MD conceived the idea and supervised the writing and the experimentation of the research work, AJ is responsible for the conceptualization of the project and the writing of the paper, PN conducted the research for similar works and contributed in the results as well as in the writing of the paper.

Funding

This work has been partially supported by the H2020 EC funded project “SPARE – Sensor-based Product monitoring system to support Augmented Reality remote maintenancE” (GA No: 80765 ΔΕΡ5-0020271).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Ahmed, M., and Khan, A.-S.P. (2019). Data analytics: concepts, techniques, and applications. Boca Raton, FL: CRC Press Taylor & Francis Group.

Google Scholar

Arduino, . (2020). Available at: https://www.arduino.cc/ (Accessed October 2,2020)

Banerjee, I., Ling, Y., Chen, C. M., Hasan, A. S., Langlotz, P. C., Moradzadeh, N., et al. (2019). Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification. Artif. Intell. Med., 97:79–88. doi:10.1016/j.artmed.2018.11.004

CrossRef Full Text | Google Scholar

Baru, A. (2018). Three ways to estimate remaining useful life for predictive maintenance. Technical Articles and Newsletters, MathWorks. Available at:https://www.mathworks.com/company/newsletters/articles/three-ways-to-estimate-remaining-useful-life-for-predictive-maintenance.html (Accessed June 20, 2020)

Google Scholar

Chen, J., Jing, H., Chang, Y., and Liu, Q. (2019). Gated recurrent unit based recurrent neural network for remaining useful life prediction of nonlinear deterioration process. Reliab. Eng. Syst. Saf. 185:372–382. doi:10.1016/j.ress.2019.01.006

CrossRef Full Text | Google Scholar

Chryssolouris, G. (2006). Manufacturing systems: theory and practice, 2nd ed. New York, NY: Springer-Verlag

Google Scholar

Deloitte, , (2017a). Predictive maintenance and the smart factory, Available at: https://www2.deloitte.com/content/dam/Deloitte/us/Documents/process-and-operations/us-cons-predictive-maintenance.pdf (Accessed June 20, 2020)

Google Scholar

Deloitte, , (2017b). Predictive Maintenance. Taking pro-active measures based on advanced data analytics to predict and avoid machine failure, Analytics InstituteAvailable at: https://www2.deloitte.com/content/dam/Deloitte/de/Documents/deloitte-analytics/Deloitte_Predictive-Maintenance_PositionPaper.pdf (Accessed June 20, 2020)

Google Scholar

Egger, J., and Masood, T. (2020). Augmented reality in support of intelligent manufacturing – a systematic literature review. Comput. Ind. Eng. 140, 106195. doi:10.1016/j.cie.2019.106195

CrossRef Full Text | Google Scholar

Erfani, S. M., Rajasegarar, S., Karunasekera, S., and Leckie, C. (2016). High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning. Pattern Recogn. 58, 121–134. doi:10.1016/j.patcog.2016.03.028

CrossRef Full Text | Google Scholar

Fantini, P., Pinzone, M., and Taisch, M. (2020). Placing the operator at the centre of Industry 4.0 design: modelling and assessing human activities within cyber-physical systems. Comput. Ind. Eng. 139, 105058. doi:10.1016/j.cie.2018.01.025

CrossRef Full Text | Google Scholar

Fast-Berglund, Å., Gong, L., and Li, D. (2018). Testing and validating Extended Reality (xR) technologies in manufacturing. Procedia Manufacturing. 25, 31–38. doi:10.1016/j.promfg.2018.06.054

CrossRef Full Text | Google Scholar

Hupfer, S. (2019). Capitalizing on the promise of artificial intelligence. Perspectives on AI adoption from around the world, Deloitte Insights. Available at: https://www2.deloitte.com/us/en/insights/focus/cognitive-technologies/global-perspectives-ai-adoption.html (Accessed June 20, 2020)

Google Scholar

Kong, Ζ., Cui, Y., Xia, Z., and He, L. (2019). Convolution and long short-term memory hybrid deep neural networks for remaining useful life prognostics. Appl. Sci. 9(19), 4156–4174. doi:10.3390/app9194156

CrossRef Full Text | Google Scholar

Kozjek, D., Malus, A., and Vrabi”, R. (2020). Multi-objective adjustment of remaining useful life predictions based on reinforcement learning. Procedia CIRP. 93, 425–430. doi:10.1016/j.procir.2020.03.051

CrossRef Full Text | Google Scholar

Lei, Y., Li, N., Guo, L., Li, N., Yan, T., and Lin, J. (2018). Machinery health prognostics: a systematic review from data acquisition to RUL prediction. Mech. Syst. Signal Process. 104, 799–834. doi:10.1016/j.ymssp.2017.11.016

CrossRef Full Text | Google Scholar

Li, X., Zhang, W., and Ding, Q. (2019). Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliab. Eng. Syst. Saf. 182, 208–218. doi:10.1016/j.ress.2018.11.011

CrossRef Full Text | Google Scholar

Li, Χ., Ding, Q., and Sun, J. Q. (2018). Remaining useful life estimation in prognostics using deep convolution neural networks. Reliab. Eng. Syst. Saf. 172, 1–11. doi:10.1016/j.ress.2017.11.021

CrossRef Full Text | Google Scholar

Liu, Y., Hu, X., and Zhang, W. (2019). Remaining useful life prediction based on health index similarity, Reliab. Eng. Syst. Saf. 185, 502–510. doi:10.1016/j.ress.2019.02.002

CrossRef Full Text | Google Scholar

Loutas, T. H., Roulias, D., and Georgoulas, G. (2013). Remaining useful life estimation in rolling bearings utilizing data-driven probabilistic E-support vectors regression. IEEE Trans. Reliab. 62(4), 821–832. doi:10.1109/TR.2013.2285318

CrossRef Full Text | Google Scholar

Masoni, R., Ferrise, F., Bordegoni, M., Gattullo, M., Uva, A. E., Fiorentino, M., et al. (2017). Supporting remote maintenance in industry 4.0 through augmented reality. Procedia Manufacturing. 11, 1296–1302. doi:10.1016/j.promfg.2017.07.257

CrossRef Full Text | Google Scholar

Microsoft, . (2020). Visual Studio. Best-in-class tools for any developer. Available at:https://visualstudio.microsoft.com/ (Accessed June 20, 2020)

Google Scholar

Microsoft, . (2018). What's a universal Windows platform (UWP) app? Available at: https://docs.microsoft.com/en-us/windows/uwp/get-started/universal-application-platform-guide (Accessed June 20, 2020)

Google Scholar

Mitchell, R. J. (1995). “How computers process data.” In: Microprocessor systems. London: Palgrave.

Google Scholar

Mourtzis, D. (2020). Simulation in the design and operation of manufacturing systems: state of the art and new trends. Int. J. Prod. Res. 58(7), 1927–1949. doi:10.1080/00207543.2019.1636321

CrossRef Full Text | Google Scholar

Mourtzis, D., Angelopoulos, J., and Panopoulos, N. (2020a). A framework for automatic generation of augmented reality maintenance & Repair instructions based on convolutional neural networks, 53rd CIRP conference on manufacturing systems (CMS 2020). Procedia CIRP. 93, 977–982. doi:10.1016/j.procir.2020.04.130

CrossRef Full Text | Google Scholar

Mourtzis, D., Siatras, V., Angelopoulos, J., and Panopoulos, N. (2020c). An augmented reality collaborative product design cloud-based platform in the context of learning factory. Procedia Manufacturing. 45, 546–551. doi:10.1016/j.promfg.2020.04.076

CrossRef Full Text | Google Scholar

Mourtzis, D., Siatras, V., and Angelopoulos, J. (2020b). Real-time remote maintenance support based on augmented reality (AR). Appl. Sci. 10(5). doi:10.3390/app10051855

CrossRef Full Text | Google Scholar

Mourtzis, D., Vlachou, A., and Zogopoulos, V. (2017). Cloud-based augmented reality remote maintenance through shop-floor monitoring: a product-service system approach. J. Manuf. Sci. Eng. 139(6), 061011. doi:10.1115/1.4035721

CrossRef Full Text | Google Scholar

Mourtzis, D., Vlachou, E., Milas, N., and Xanthopoulos, N. (2015). A cloud-based approach for maintenance of machine tools and equipment based on shop-floor monitoring. Procedia CIRP 41, 655–660. doi:10.1016/j.procir.2015.12.069

CrossRef Full Text | Google Scholar

Mourtzis, D., Zogopoulos, V., Katagis, I., and Lagios, P. (2018). Augmented reality based visualization of CAM instructions towards industry 4.0 paradigm: a CNC bending machine case study. Procedia CIRP 70, 368–373. doi:10.1016/j.procir.2018.02.045

CrossRef Full Text | Google Scholar

Nicholson, A., Webber, S., Dyer, S., Patel, T., and Janicke, H. (2012). SCADA security in the light of cyber-warfare. Comput. Secur. 31(4), 418–436. doi:10.1016/j.cose.2012.02.009

CrossRef Full Text | Google Scholar

Palmarini, R., Erkoyuncu, A. J., Roy, R., and Torabmostaedi, H. (2018). A systematic review of augmented reality applications in maintenance. Robot. Comput. Integrated Manuf. 49, 215–228. doi:10.1016/j.rcim.2017.06.002

CrossRef Full Text | Google Scholar

Pang, Y., He, Q., Jiang, G., and Xie, P. (2020). Spatio-temporal fusion neural network for multi-class fault diagnosis of wind turbines based on SCADA data. Renew. Energy. 161, 510–524. doi:10.1016/j.renene.2020.06.154

CrossRef Full Text | Google Scholar

Rezaeianjouybari, B., and Shang, Y. (2020). Deep learning for prognostics and health management: state of the art, challenges, and opportunities. Measurement. 163, 107929. doi:10.1016/j.measurement.2020.107929

CrossRef Full Text | Google Scholar

Ruiming, F., Minling, W., Xinhua, G., Rongyan, S., and Pengfei, S. (2020). Identifying early defects of wind turbine based on SCADA data and dynamical network marker. Renew. Energy 154, 625–635. doi:10.1016/j.renene.2020.03.036

CrossRef Full Text | Google Scholar

Sun, H., Liu, Y., Pan, J., Zhang, J., and Ji, W. (2020). Enhancing cutting tool sustainability based on remaining useful life prediction, J. Clean. Prod. 244, 118794. doi:10.1016/j.jclepro.2019.118794

CrossRef Full Text | Google Scholar

Tao, F., Qi, Q., Liu, A., and Kusiak, A. (2018). Data-driven smart manufacturing. J. Manuf. Syst. 48, 157–169. doi:10.1016/j.jms y.2018.01.006

CrossRef Full Text | Google Scholar

Unity, . (2020). Available at: https://unity.com/ (Accessed June 20, 2020)

Vogl, G. W., Weiss, B. A., and Helu, M. (2019). A review of diagnostic and prognostic capabilities and best practices for manufacturing. J. Intell. Manuf. 30(1), 79–95. doi:10.1007/s10845-016-1228-8

CrossRef Full Text | Google Scholar

Vorraber, W., Gasser, J., Webb, H., Neubacher, D., and Url, P. (2020). Assessing augmented reality in production: remote-assisted maintenance with HoloLens. Procedia CIRP. 88, 139–144. doi:10.1016/j.procir.2020.05.025

CrossRef Full Text | Google Scholar

Wang, H., Peng, M. J., Miao, Z., Liu, Y. K., Ayodeji, A., and Hao, C. (2020). Remaining useful life prediction techniques for electric valves based on convolution auto encoder and long short term memory. ISA (Instrum. Soc. Am.) Trans. doi:10.1016/j.isatra.2020.08.031

CrossRef Full Text | Google Scholar

Wen, P., Zhao, S., Chen, S., and Li, Y. (2021). A generalized remaining useful life prediction method for complex systems based on composite health indicator. Reliability Engineering & System Safety. 205, 107241. doi:10.1016/j.ress.2020.107241

CrossRef Full Text | Google Scholar

Wolfartsberger, J., Zenisek, J., and Wild, N. (2020). Data-driven maintenance: combining predictive maintenance and mixed reality-supported remote assistance. Procedia Manufacturing. 45, 307–312, doi:10.1016/j.promfg.2020.04.022

CrossRef Full Text | Google Scholar

X-CTU, , (2020). Available at: https://www.digi.com/products/embedded-systems/digi-xbee/digi-xbee-tools/xctu (Accessed October 2, 2020)

Yang, F., Habibullah, S. M., and Shen, Y. (2021). Remaining useful life prediction of induction motors using nonlinear degradation of health index. Mech. Syst. Signal Process. 148, 107183. doi:10.1016/j.ymssp.2020.107183

CrossRef Full Text | Google Scholar

Yang, H., Zhao, F., Jiang, G., Sun, Z., and Mei, X. (2019). A novel deep learning approach for machinery prognostics based on time Windows. Appl. Sci. 9, 4813. doi:10.3390/app 9224813

CrossRef Full Text | Google Scholar

Zhang, S., and Lang, Z. Q. (2020). SCADA-data-based wind turbine fault detection: a dynamic model sensor method. Contr. Eng. Pract. 102, 104546. doi:10.1016/j.conengprac.2020.104546

CrossRef Full Text | Google Scholar

Zhao, Ζ., Liang, B., Wang, X., and Lu, W. (2017). Remaining useful life prediction of aircraft engine based on degradation pattern learning, Reliab. Eng. Syst. Saf. 164, 74–83. doi:10.1016/j.ress.2017.02.007

CrossRef Full Text | Google Scholar