Assessing network-based earthquake early warning systems in low-seismicity areas

Earthquake early warning (EEW) technology, designed to alert the public of earthquake risks after initial P-wave detection but before the onset of strong tremors, has developed rapidly. Methodologies from various fields are combined in EEW systems to estimate earthquake locations, magnitudes, and expected intensities based on the initial P-wave data. These systems operate automatically because prompt responses are required. However, as no common evaluation framework for EEW system verification exists, potentially divergent evaluations from reviewers or countries could ensue. Moreover, evaluating EEW systems is more complicated when the target area does not experience frequent earthquakes. We aimed to establish a guidance review process for low-seismicity areas to ensure reliable and stable integrated EEW system operation. We incorporated management aspects through actual system operator surveillance and designed an EEW assessment process based on feedback from our surveys. Using this approach, we created a comprehensive and well-informed evaluation process that considers the diverse perspectives of experts involved in EEWs. Our proposed assessment method allows for a uniform and consistent evaluation process, regardless of changes in the methods or technologies used by EEW systems. The method aims to guide EEW system assessments in low-seismicity areas.

1 Introduction

For rapid mitigation against seismic hazards, automated alerting technology based on seismic networks EEWs is used, with automated event estimation employed to facilitate the issuing of timeous public alerts (Allen and Melgar, 2019; Velazquez et al., 2020). Such systems operate at national or local government levels, as their fundamental technology requires 24 h monitoring (Mayer et al., 2008). Several countries have implemented EEW systems as public services, including Japan (Hoshiba and Ozaki, 2014), Republic of Korea (Lee et al., 2022), Taiwan (Hsiao et al., 2009), and the United States (Kohler et al., 2018). Among these countries, Republic of Korea experiences the lowest occurrence frequency of medium-to large-scale earthquakes.

After the introduction of EEW in Republic of Korea in 2015 (KMA, 2018), public awareness remained low until two earthquakes occurred (local magnitude [M_L] = 5.8 on 12 September 2016, and M_L = 5.4 on 15 November 2017). The low level of public awareness could be ascribed to the occurrence and recording of only ten earthquakes exceeding M_L > 5.0 since instrumental observations started in 1980. However, the two events of 2016–2017 were a timely reminder of the importance of EEW, and EEW technology has since developed rapidly in the country. Initially, a cell broadcasting service (CBS) was intended for governmental use, but the Korea Meteorological Administration (KMA, 2017) has extracted and applied relevant parts of the service for earthquake, tsunami, and volcano warnings (KMA, 2018). The network-based EEW system was optimized further to suit domestic observation environments, estimation of magnitudes (Sheen et al., 2018), and detection of initial P-waves (Cho et al., 2022).

KMA EEW progress relies on warning time reduction, as it is crucial to warn people as quickly as possible in the case of an earthquake (Bostrom et al., 2022; Ahn et al., 2023). Warning time is a critical factor in assessing preparedness for an earthquake and a vital consideration from a risk management perspective (Allen et al., 2009). The evaluation of EEWs is often based on lead time, i.e., the time between an alert is received and the arrival of tremors or shaking (Cremen et al., 2022). A primary method to reduce EEW lead time is decreasing the number of stations (nS) used for detecting earthquakes in the initial analysis version of the event. However, the likelihood of prediction errors increases when fewer stations are involved in earthquake determination and analysis.

Although Republic of Korea has not experienced social disruption induced by false earthquake alarms for KMA, such incidents could cause substantial social issues. On 5 January 2018, a false alarm occurred when an EEW mistakenly identified two small earthquakes as one large earthquake in Japan, causing temporary public panic (BBC News, 2018). The head of the Japan Meteorological Administration was forced to apologize publicly for the error. Owing to a computer error, a warning of a large-scale earthquake off the coast of California (United States) was issued on 22 June 2017, 92 years after it had actually occurred. This warning caused widespread public confusion (BBC News, 2017). Another notable example is a false alert issued in Mexico on 28 July 2014, by Sky Alert, a popular smartphone application, prompting response from numerous people. This incident triggered a social issue in Mexico, invoking Aesop’s fable “The Boy Who Cried Wolf” (Reddy, 2020). Although SkyAlert is not a public service, the consequences of the false alarm emphasized the importance of proper EEW management.

While the fact that Republic of Korea experiences fewer earthquakes and alerts may be considered a positive aspect, it has also led to many Koreans being unaware of EEWs. According to a 2020 survey by Ahn (2021), only 84 (approximately 44%) of 192 general public respondents (i.e., non-seismologists and non-civil engineers) were aware of EEWs. As a result of the limited number of alert cases and the lack of public awareness, obtaining public evaluations of these services remains a challenge.

Owing to these limitations, EEWs are generally evaluated by experts rather than the public. Past studies have either assessed the point source or alert accuracy. Point source assessments verify EEW algorithms for magnitude and hypocenter accuracy (e.g., Cua et al., 2009; Chen et al., 2015; Massin et al., 2021). Alert assessments are performed to prevent errors during operations or simulations (e.g., Chen et al., 2019; Chung et al., 2019; Zuccolo et al., 2021). However, management aspects require a comprehensive review of the overall EEW. Cochran et al. (2018) proposed a framework for a Testing and Certification Platform (TCP) that assesses the performance of the overall system alerts issued by the decision module and those generated by individual algorithms. The TCP assessment consists of both point-source alert elements (i.e., magnitude, epicentral location, origin time, and alert latency) and ground-motion prediction accuracy. TCP is evaluated on the basis of alerts, and the more earthquakes available for evaluation, the better. Unfortunately, in countries with low earthquake frequency and relatively few strong motions, the Cochran et al. (2018) platform is limited.

Expert assessments vary based on the knowledge and interests of the individual conducting the evaluation (Binger et al., 2012; May et al., 2016). Additionally, the evaluator must make a minimum number of decisions for the assessment, which could be challenging. Experts tend to be cautious during decision-making because their decisions ultimately affect the public services provided to citizens (Meijer and Grimmelikhuijsen, 2020). For KMA EEW, the public is informed of earthquake risks for events with M_L > 3.5. However, since 2010, there have been only five events of M_L ≥ 5.0. As a result, evaluations often include earthquakes with slight magnitudes which was events of M_L ≥ 2.5 (Cho et al., 2022). These issues raise concerns regarding the appropriate magnitude target for assessment, review area, and acceptable accuracy standard. Answers from the experts could differ, i.e., it is crucial to recognize that even experts could hold different perspectives on related issues.

The fundamental goal of an EEW is achieving promptness and accuracy (Allen et al., 2009; Kamigaichi et al., 2009; Satriano et al., 2011; Finazzi, 2020) However, balancing these two aspects is challenging, and performance scores could vary depending on the criteria and the reviewer. Therefore, as regards the decision-making process in assessing EEW, it is essential to integrate expert opinion with the goal of providing an effective public service, as different people could have different values and priorities, even when working toward a common mission. Therefore, we aimed to establish an EEW review method that incorporates management aspects through actual system operator surveillance. We designed an EEW assessment process based on the feedback received from our surveys, such as criteria for analysis, pairwise comparison, and so on. Our approach was to create a comprehensive and well-informed evaluation process that considers the diverse perspectives of the experts involved in EEWs.

2 Criteria and methods

2.1 Alert criteria and EEW service

Conservative alert standards are in place in the KMA EEW service. The Korean Peninsula is an area of low seismicity; therefore, even a small tremor could unnerve citizens. Furthermore, considering that the country hosts semiconductor factories, high-speed rails, and nuclear facilities, alert services are conducted with sensitivity preparation. In the event of an earthquake of M_L ≥ 3.5, seismic information is disseminated within 5–10 s after the initial detection.

The KMA EEW uses at least four stations for optimal performance (Cho et al., 2022), aiming to make decisions on alerts about 5 s after the first observation when an earthquake occurs on the Korean Peninsula. In EEW, the theoretical minimum number of stations (nS) for hypocenter determination is three (Yamada et al., 2021), but relying on the assessment accuracy can be challenging. Low accuracy is particularly associated with earthquakes occurring outside the observation network. Based on these settings, when an earthquake with M_L > 3.5 (or > 4.0, if occurring over the sea) occurs, a warning message is automatically propagated to the public. If an earthquake with an M_L > 5.0 occur, warnings with a stronger alert sound are disseminated. These alert criteria were designed as a policy decision in Republic of Korea. Therefore, the EEW alert decisions might differ in other countries.

Additionally, the KMA operates a network-based EEW that incorporates the algorithms ElarmS-3.0 (Chung et al., 2019), RTLoc (Satriano et al., 2008), and Maxwell–Hertz electrostatic potential theory (Sheen, 2016). These algorithms have been optimized for the Korean Peninsula by KMA and are used simultaneously for monitoring earthquakes. The KMA EEW is based on the mutual interaction and automatic collaborative decision making of the algorithms, and produces an alert when two or more algorithms detect an earthquake. In principle, if two logically distinct algorithms working concurrently sound an alert, the likelihood of an actual earthquake occurrence is higher.

By combining three algorithms, we achieve analysis accuracy as well as 24 h service stability. This approach has been instrumental in overcoming occasional system failures experienced by the KMA, including bugs, delays, and network issues. However, owing to the complexity of the algorithm, which involves the integration of multiple systems, it becomes sensitive to minor enhancements.

2.2 Disaster communications to the public

The KMA EEW alerts, determined by a combination of technology and policy, are issued to citizens in the event of an earthquake through the CBS. Using CBS technology, a warning is sent to numerous users in the vicinity of base stations (Doi, 2011; Wu et al., 2022). Alarm is transmitted to mobile phones in approximately 3 s (Minson et al., 2018), and the public receives messages shortly after earthquake detection. Alerts are forcibly sent to individual mobile phones in the form of emergency text messages. Figure 1 shows an example of earthquake warning information delivered to the public. The message provides information on the epicenter, event time, and magnitude of the earthquake, along with guidelines to “drop, cover, and hold on” (DCHO) (Porter, 2016).

FIGURE 1

FIGURE 1. Warning messages delivered to end-users in Republic of Korea to “drop, cover, and hold on” (DCHO).

Following alert dissemination, evaluations from citizens regarding the system’s effectiveness could be gathered. Speediness, as perceived by the citizens, is the time difference between receiving the alert and feeling the earthquake vibrations. It is challenging for citizens to quantitatively assess promptness, as the warning and the shaking occur within a very short time frame. However, as regards source information (i.e., location, magnitude, and event time), people judge the warning accuracy by comparing it to subsequent earthquake information released through news broadcasts. Collecting feedback is vital for understanding the needs of the public with respect to the DCHO advice, which is a crucial aspect of emergency preparedness. However, intermediate and large earthquakes do not occur frequently in the Korean Peninsula, and assessing KMA EEW might be difficult for the general public. Consequently, only people who manage and develop the system can evaluate the KMA EEW.

2.3 Assessment survey and design

Operator or manager assessments may vary depending on the expertise of the reviewer (Cooke and Goossens, 2004). Here, evaluations conducted from the perspective of a small group of developers could lead to serious issues in the future. To address this concern, we gathered opinions from individuals with relevant experience and field knowledge. We surveyed 18 KMA employees who operated EEW from July 14–16, 2021. The survey research was designed and conducted in the following order 1) assessment parameter selection, 2) relational questionnaire development based on assessment parameters and pairwise analysis questionnaires for the factors, 3) operator interviews, 4) questionnaire criteria determination, analytic hierarchy process (AHP) analysis, and weight function designation, and 5) EEW operational aspect assessment.

2.3.1 Aspect-based assessment parameter selection

To select appropriate assessment parameters for EEW, two aspects were considered, namely, operations and management, which are distinct but interrelated. Operation refers to the day-to-day activities and tasks of a system (Whipple and Frankel, 2000). In EEWs, operations focus on the reliability of alert production (Medina-Cetina and Nadim, 2008). In contrast, management involves EEW planning and control to achieve high performance (Ittner and Lacker, 1997), considering alert reliability and seismic information accuracy (Ruhl et al., 2019; Esposito et al., 2022). Managers are responsible for overseeing objective reviews and implementing improvements, ensuring that a balance is maintained between safety, effectiveness, and sustainability (Too and Weaver, 2014). Figure 2 shows the EEW algorithm analysis process with both aspects. The assessment steps are shown in the EEW flow, considering algorithm analysis based on simulations.

FIGURE 2

FIGURE 2. Flowchart of assessment parameter calculation process. nS, number of stations; nT, number of trigger sensors.

As regards the operational aspects, system behavior was categorized into three steps. Step 1 is the evaluation preparation stage, comprising input of the earthquake scenario and setting the minimum targeted nS. Step 2 is the detection phase of the analysis process [i.e., short-time-average over long-time-average trigger (Allen, 1982)] or the filter picker (Lomax et al., 2012). If the nS condition is satisfied, it is verified to determine whether it becomes an associated event (Cho et al., 2022). If the alarm fails to generate an event, verification is required whether this event was alarm worthy. Verification allows the determination of the alarm being operationally a false alarm or normal operation. Step 3 comprises predicting the source and assessing the alarm based on associated event information (Weyrich et al., 2021). Alarms can be produced by EEWs based on the predicted source information, which is termed decision making of warning (DMW).

As regards the management aspect, it is essential to verify the success of the alert and the source information that influenced the decision to issue an early warning. By considering all the parameters that impacted the final alert, it could be ensured that the algorithm and system were free of issues. Figure 2 presents representative error sources that could be easily identified from an administrator viewpoint. However, various errors could occur in operations, and this procedure is only applicable to natural earthquakes.

In the review of EEW assessment, we ignored the associate event time, which depends on the nS. Additionally, waveform data used in KMA seismic stations are applied only to locations that pass strict criteria for latency, background noise, and trigger rate (KMA, 2017; Ahn et al., 2021). Consequently, data lagging or failed equipment rarely occur for KMA EEW in real-time operations. However, from a management perspective, a station detection rate review within a certain time frame could be performed separately, because of trigger checking at the observatory.

We selected four parameters for this study based on aspects. 1) The first parameter is “prevent missed alerts (PM)” in the system, i.e., confirming whether a targeted earthquake alert was missed. Alerts in normal operations are issued when the national observation network detects a warning event using n stations. 2) The second parameter is “prevent false alerts (PF)” in the system, which determines whether a targeted earthquake alert is incorrect. This involves checking alarms that are normal operations, non-earthquakes, or source errors when EEW alerts occur. 3) The third parameter is the accuracy of magnitude (AM), and 4) is the accuracy of location (AL). These four parameters were reviewed from operational and administrative perspectives and represent all the information that the public sees when an alert is triggered.

2.3.2 Assessment criteria preparation

To help respondents better understand the assessment parameters, we provided training before conducting the research. During the training, we introduced various existing analytical examples, explained the unique features of the criteria, and received approval for ethical review surveys involving human participants. Further, we discussed the EEW technical and theoretical limitations. These are 1) Inability to fully understand the underground geological structure, implying that seismic information is subject to uncertainty at source (Iervolino et al., 2009; Tidlund et al., 2022; Murray et al., 2023; Ni et al., 2023). 2) Detection accuracy varies depending on the observation network because of wave propagation attenuation (Jackson and Anderson, 1970; Vipin and Sitharam, 2011; Goda et al., 2023). 3) Accuracy limitations of source detection in coastal earthquakes (Angove et al., 2019; Lim et al., 2020; Takabatake and Kojima, 2023), and the difficulty of installing seismometers in the seabed (Podolskiy et al., 2021). The system operators understood and agreed with these EEW limitations. In addition, we emphasized understanding of the characteristics of earthquakes in the Korean Peninsula, as well as the psychological conditions of the public. Given the infrequent occurrence of seismic events in the Korean Peninsula, even weak shaking could elicit feelings of surprise and unease among the general population (Kwon et al., 2020; Yeon et al., 2020). Therefore, unlike other countries, the KMA issues warning criteria from even slight magnitudes (M_L 3.5 and over).

Considering KMA policy and technology characteristics, we designed a questionnaire for the EEW assessment. In the criteria establishment stage of the survey, we created five response items to establish the assessment criteria. We classified the criteria between inland areas and outside (i.e., ocean, outside the country) of the seismic observation network. Both groups 1) within the seismic observation network (= inside seismic network, ISN) and 2) outside the seismic observation network (= outside seismic network, OSN) were considered. Categorization by observation network was included in all five questions.

The survey questions were:

Q.1: “What is the minimum magnitude of an earthquake to be analyzed for performance assessment?” We needed to establish a baseline progression for our evaluation, considering the margin of error in our prediction scale. However, detecting initial waveforms from slight magnitude earthquakes is complicated owing to their small amplitudes and poor observation environments. The EEW performance results could, therefore, vary depending on the criteria and magnitude, with greater sensitivity for weaker events than larger ones.

Q.2: “Compared with M_L ≥ 5.0, how important is an earthquake of M_L < 5.0?” This item was included in the questionnaire, as the level of ensuing damage could differ according to the magnitude of the event. While seismic intensity is important from a damage perspective, the basic information transmitted is magnitude and location. Therefore, we investigated the relativity of magnitude. The initial standard for early warnings of earthquakes in Republic of Korea was M_L > 5.

Q.3: “How important is the comparison of earthquake locations, both inside and outside the observation network?” This question was included because it is associated with the observation network conditions, and the current criteria classify inland and outside events (i.e., teleseismic or occurring outside the country). These criteria represent the KMA strategy for considering the impact of earthquake detection relative to the observation network. Inland earthquakes can be detected rapidly and analyzed as they occur between observation networks, but oceanic analysis could likely be inaccurate in relation to the source owing to biased and sparse observation networks.

Q.4: “What is the maximum allowable error range when evaluating the accuracy of the magnitude?”

Q.5: “What is the maximum allowable error range when evaluating the accuracy of the epicenter?” The fourth and fifth questions are related to accuracy, as accuracy could affect missed and false alarms. Additionally, if the difference between the predicted information and precise information analyzed afterward is significant, an operating system check could be required.

In the criteria establishment stage of our survey, we analyzed the results based on the five questionnaires and collected the majority opinion regarding operational recommendations. Table 1 shows a summary of the established criteria from the questionnaire. The criteria in Q.1 allowed us to define the analysis object range. We established the range for earthquakes to be considered by ISN and OSN, i.e., 0.5 smaller than the magnitude of the alert criteria.

TABLE 1

TABLE 1. Performance assessment criteria based on questionnaire responses.

We set the weight for event sources based on Q.2 and Q.3. From an operational side, all earthquakes are valued equally because the evaluation pertains only to whether an alert should be issued. For instance, in a point-source assessment by Cochran et al. (2018), all earthquakes were also valued equally. However, the damage caused by an earthquake depends on the location and magnitude of the event. Therefore, we proposed weighing of the source effects. The score for each earthquake event was:

$\text{Event\hspace{0.17em}Value\hspace{0.17em}}\left(EV\right)=\delta M\times \delta L(1)$

where EV is event value, which is the judgment of earthquakes from a management perspective. δM is the weight for magnitude, and δL is the weight for the source location (i.e., ISN, OSN) of the earthquake. For example, if an earthquake of M_L 5.5 occurred inland, the event value would be 1. Conversely, if an earthquake with magnitude M_L 3.2 occurred inland near the coast, the event value would be 0.18.

The criteria for an EEW false alarm was selected based on answers to Q.4 and Q.5. The criteria set for a false alarm were assessed based on the estimated source information produced by EEWs. In practice, the results of both source location and magnitude of EEWs depend on the nS (Lim et al., 2020). The more stations are involved in collecting information, the better would be the accuracy of epicenter analysis. From a management perspective, the criteria for a false alarm could be a crucial factor for planning system improvements.

2.3.3 Assessment parameter scoring

Assessments of EEWs must consider both the appropriateness of the alerts and the accuracy of earthquake source estimates. The source estimates in EEWs are updated rapidly as they increase because of a higher number of nS. However, we do not recommend reviewing all the steps that are updated over time, because the public responds to the first warning message (Lassa, 2008; Vihalemm et al., 2012); therefore, the information at the moment the alarm occurs is the only concern. Alert time depends on nS, which is the most important factor in EEW alarm decision making. Our proposed assessment is designed to evaluate performance according to the target nS.

The first step was to score alert appropriateness (AA). We designed the scoring to include only events where the alert was a normal operation under the target nS conditions. If an earthquake required an alert but resulted in a false or missed alarm, we scored it as zero. This allowed us to determine if the system has effective controls for false or non-alarms. Consequently, our assessment could result in a low score if the system failed to control alarms for multiple cases. The formula for this is:

$\text{Prevent\hspace{0.17em}false\hspace{0.17em}alerts\hspace{0.17em}}\left(PF\right)=EV\times AA(2)$

$\text{Prevent\hspace{0.17em}Miss\hspace{0.17em}alerts\hspace{0.17em}}\left(PM\right)=EV\times AA(3)$

where EV is the event value by Eq. 1, AA is scoring of alert appropriateness (only 1 or 0). Therefore, a normal alert is 1, and false and missed alarms are 0.

The second step is to score the accuracy of an estimated source. By default, we designed the accuracy calculation to be based only on positive cases (normal alerts). The basic framework is similar to that of the Cochran et al. (2018) model. The expression is:

$\text{Accurate\hspace{0.17em}of\hspace{0.17em}magnitude\hspace{0.17em}}\left(AM\right)=EV\times \left(1-\frac{M_{No}-M_{EEW}}{\Delta M}\right)\times AA(4)$

$\text{Accurate\hspace{0.17em}of\hspace{0.17em}location\hspace{0.17em}}\left(AL\right)=\text{EV}\times \left(1-\frac{L_{No}-L_{EEW}}{\Delta L}\right)\times AA(5)$

where M_NO and L_NO are the notified magnitude and epicenter of the earthquake according to the KMA. The notified earthquake information is calculated through manual analysis by seismic analysts at the KMA. The M_EEW and L_EEW are the calculated source information (i.e., magnitude and epicenter) according to the target nS in the EEW. The maximum allowable error values, ΔM and ΔL, are based on the answers to Q.4 and Q.5. The accuracy value is close to 1 for a small error and close to 0 for a large error.

2.3.4 Pairwise analysis and assessment parameter weight

We assumed that different experts would place different values on the four parameters; therefore, we performed a relative comparison of the four parameters to obtain their values. For the weights of the four assessment parameters, we used the Analytic Hierarchy Process (AHP) that involves pairwise comparison, weight calculation, and consistency verification, as described by Saaty et al. (1990). The AHP process could be used as a rational decision-making tool for risk assessment to stratify, simplify, and systematize multiple criteria (Wen, 2015; Thaker et al., 2018; Providakis et al., 2022). This survey aimed to identify the most important parameters for operators by employing relative comparability.

During the pairwise comparison process, the study assessed the relative importance of the four assessment parameters using a nine-point scale, as shown in Figure 3. Following this step, the results were subjected to consistency verification to ensure their reliability. The consistency index (CI) used to assess the reliability of the responses was calculated through the maximum Eigenvalue operation of the comparison matrix. A lower CI value indicated higher reliability. For this study, an average weight of 11 questionnaires with a consistency index of 0.2 or less was used for the analysis. A summary of the results of these weights is presented in Table 2.

FIGURE 3

FIGURE 3. Questionnaire example for pairwise assessment parameter comparison. EEW, earthquake early warning.

TABLE 2

TABLE 2. Weighting of four assessment parameters based on questionnaire answers received from Korean Metrological Administration (KMA).

In pairwise analysis, the most important factor from the perspective of managers and operators is controlling false alarms. This factor is crucial because the issuing of a warning and the level of the warning depend on the accuracy of magnitude and location. Although the algorithm itself does not reveal this, operators consider the activation of a warning without an actual earthquake a significant problem. Therefore, false alarm control was allocated the highest weight, followed by non-alarm control, location accuracy, and magnitude accuracy.

To determine the assessment of the overall score (OS), we used the weights and assessment points of the four parameters. The final score was obtained by multiplying the score of each element by its weight, as follows:

$\text{Overall\hspace{0.17em}Score\hspace{0.17em}}\left(OS\right)=\alpha \bullet PM+\beta \bullet PF+\gamma \bullet AM+\kappa \bullet AL(6)$

where PM, PF, AM, and AL refer to parameters such as prevent missed alerts, prevent false alerts, accuracy of magnitude, and accuracy of location, with α, β, γ, and κ as their respective weights (Table 2).

3 Results

3.1 Assessment preparation

We verified the proposed assessment of results in EEWs using past earthquake data. We analyzed 58 domestic earthquakes that occurred between January 2015 and August 2021, with M_L ≥ 3.0 and ≥ 3.5 for the ISN and OSN, respectively. The details of the events are shown in Figure 4 and summarized in Table 3. In this analysis, the operational decisions for checking the appropriateness of the alerts are shown in Table 4 using the criteria of Cho et al. (2022). However, differing from the Cho et al. (2022) study, the current assessment imposed stricter criteria for matching source information based on the survey result.

FIGURE 4

FIGURE 4. Distribution of events based on performance assessment. ISN, inside seismic network; OSN, outside seismic network.

TABLE 3

TABLE 3. Assessment target event list.

TABLE 4

TABLE 4. Performance assessment criteria comparison.

We assessed the performance of the initial P-wave trigger criteria in the ElarmS-3 algorithm, which was proposed by Chung et al. (2019) and include minPa, Range-Post Trigger (RP), NEtoZ, and Zero-Crossing (ZC). MinPa is the criterion for the initial P-wave, representing the minimum amplitude of acceleration during 4 s. Noise signals (i.e., boxcars, spikes) and block S-waves are removed using RP and NEtoZ, respectively. The validity of the signal is determined by ZC by counting the number of samples based on zero. Ultimately, we designed five simulation matrices based on the study by Cho et al. (2022). The configuration of each module is summarized in Table 5.

TABLE 5

TABLE 5. Assessment configurations of five conditions in early earthquake warning (EEW) algorithms.

We assessed the effectiveness of the EEWs by simulating past earthquake events. Although past earthquake simulations cannot completely replicate real-time data delays, simulation analyses based on recorded data could produce reliable results similar to those of real-time operations (Lim et al., 2020). The simulation method used in this study does not allow for the delay of telecommunication time, and time-related errors were not surveyed, which is a limitation of the study. We assumed that the processing of observation data was conducted without delay.

Figure 5 shows an instance of simulation results for an earthquake. Here, we observed that the analysis results differed each time the simulation was executed, despite using identical simulation environments. The recording values of several observatories were transmitted simultaneously to the algorithm, and the buffer order changed owing to differences in micro-small system time. These issues were also found in a couple of pairs of a real-time system. However, as they do not significantly affect the overall analysis, we decided to tolerate the problem and considered it in this study. We conducted 20 repetitive simulations for all events to consider any unspecified deviations that could arise during simulation reproduction.

FIGURE 5

FIGURE 5. Differences between simulation results performed with the same system environment parameters of a magnitude 3.4 earthquake on 27 October 2019, in the Korean Peninsula.

3.2 Performance assessment case

We conducted a performance evaluation of the actual EEW algorithm using the OS. The accuracy of the network-based EEWs increased along with an increase in the nS (Ruhl et al., 2019; Cho et al., 2022), as shown in Figure 5. Accordingly, the OS should receive higher scores as the nS increases. To verify this finding, we examined the evaluation results for condition 1, as shown in Figure 6. The figure shows that as the nS increased, the OS also increased, and the deviations in the 20 simulation cases declined.

FIGURE 6

FIGURE 6. Box and whisker plot for overall score (OS) of algorithmic condition 1 for earthquake early warnings (EEW).

We conducted performance assessment of the actual EEW algorithm by applying the assessment process developed for OS. This was intended to review the suitability of the assessment process and evaluate the algorithmic condition algorithm performance. The simulation analysis results for five algorithmic conditions and 58 earthquakes were obtained for this performance assessment. In this study, the five algorithmic conditions were termed A1–A5.

As four is the minimum nS for the KMA EEW, we compared the OS under five conditions with the evaluation results shown in Figure 7. As the proposed OS could calculate a score based on a single earthquake, the overall results are presented in a box and whisker plot to show the mean and median values. The results in Figure 7 indicate that A1–2 without the RP condition was more stable than A3–5.

FIGURE 7

FIGURE 7. Box and whisker plot for overall score (OS) of algorithmic condition at number of stations, (nS) = 4.

We had to determine the reason for the addition of the RP condition resulting in relatively low scores; therefore, we determined the number of points each parameter earned. Figure 8 shows the average scores obtained in the parameter domain. Compared with A3–5, A1–2 exhibited a slight increase in the AM and a significant increase in the PF. We expected that using the RP condition would be effective for controlling false alarms; however, the results were not satisfactory.

FIGURE 8

FIGURE 8. Mean score of four parameters for algorithmic condition at number of stations (nS) = 4. PM, prevent missed alerts; PF, prevent false alerts; AM, accuracy of magnitude; AL, accuracy of location.

In the case of A1–2 without the RP condition, we found that the analysis of A2 was more stable than that of A1. The criterion for minPa in A2 is the optimization proposed by Cho et al. (2022). Ultimately, the optimized amplitude criterion could secure EEW stability. In addition, the optimization criterion proposed in the study by Cho et al. (2022) was A5. However, in the current study, A2 was found to ensure EEW stability. This finding indicates that using minPa alone was more effective than using both conditions.

4 Limitation

To address earthquake hazards and establish strategies in various countries, EEWs are currently subjected to various research initiatives (e.g., Minson et al., 2018; Santos-Reyes, 2019; McBride et al., 2020; Ahn, 2021; Bostrom et al., 2022; Dallo et al., 2022; Sumy et al., 2022; Sutton et al., 2022). Similar strategies for earthquake responses are evolving to suit the characteristics of each country. Accordingly, the KMA needed to develop network-based EEW assessment strategies for Republic of Korea, a low-seismicity area.

The KMA evaluation method proposed in this study could potentially be adopted by other countries to assess stability during minor earthquakes. However, its applicability might vary depending on the disparities in the network-based EEW algorithms employed by each country and the perspectives of the network managers.

The EEWs are not only network based but also include other types, such as performance-based EEW (Convertito et al., 2008), OnSite EEW, and hybrid methods (Iervolino et al., 2006). The performance-based EEWs have focused on Intensity Measures (IM) and damage estimation for specific structures (Iervolino et al., 2009; Iervolino, 2011). The OnSite EEW issues an alert based on IM in a small area. The IM-based EEW could be tied to a strategic alerting strategy to consider damage. We believe that developing a damage-based strategic evaluation technique holds practical significance (Iervolino et al., 2011; Papadopoulos et al., 2023). The KMA is also engaged in the development and experimentation of on-site EEW systems based on IM (Ahn et al., 2023). However, these methods differ from network-based EEWs.

The current study concentrated on alerts disseminated in the public domain. Given the widespread use of smartphones, CBS stands out as the most efficient means of conveying alerts to individuals. However, to guarantee the reliability of alerts in the public sphere, this study defined successful, false, and missed alarms from an administrator standpoint. However, we note that this viewpoint could contrast with the that of the public receiving the service.

5 Conclusion

We developed an assessment for EEWs based on management aspects. While promptness is an integral factor in EEW assessment, we considered that reducing warning time based on nS could result in less accurate information. Therefore, the proposed performance assessment could comprehensively review EEW services from multiple aspects.

Surveys involving experts were conducted to establish standards for assessment criteria consistent with KMA EEW operational management. The proposed weights sufficiently addressed the preferences of operators with EEW technical system knowledge and the managers of the systems. These experts selected operational impact factors according to their operational principles, with most operators expressing a desire to minimize EEW false alerts.

The comprehensive review was scored by assigning weights to each element based on the AHP analysis. Accordingly, a quantitative assessment method was developed that could replace the conventional method, which only considers warnings. The established assessment equation reflected performance improvement based on accuracy. Performance assessments can be conducted with changes in algorithms or parameters.

Our method could be appropriate for regions that are new to EEW in low-seismicity areas. However, our study is limited in that it only surveyed operators who were familiar with EEW technology. They responded with answers characterized by 1) imagining end-user satisfaction, 2) considering only technical aspects, and 3) summarizing all perspectives. Therefore, we note that the criteria could change depending on the survey responses, as individuals have various perspectives, backgrounds, and types of thinking.

The proposed EEW system assessment offers a quantitative approach that takes into account operational aspects. This method would be also expected the quantitative values of factors during actual operational periods, such as delays and anomalies arising from issues at the observation stations. Given the range of potential scenarios, it is imperative that this method undergoes further refinements for future operations.

Data availability statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.

Author contributions

J-KA: Conceptualization, Data curation, Formal Analysis, Methodology, Supervision, Writing–original draft, Writing–review and editing. SC: Formal Analysis, Visualization, Writing–original draft. E-HH: Funding acquisition, Project administration, Supervision, Writing–original draft. W-HB: Conceptualization, Methodology, Writing–original draft, Writing–review and editing.

Funding

The authors declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by the (grant number KMA 2022-02121: Development of Earthquake Information Production Technology).

Acknowledgments

The authors gratefully acknowledge the assistance received from the staff of the KMA Earthquake and Volcano Research Division.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Ahn, J. K., Kwak, D. Y., and Kim, H. S. (2021). Estimating VS30 at Korean peninsular seismic observatory stations using HVSR of event records. Soil Dyn. Earthq. Eng. 146, 106650. doi:10.1016/j.soildyn.2021.106650

CrossRef Full Text | Google Scholar

Ahn, J. K., Park, E., Kim, B., Hwang, E. H., and Hong, S. (2023). Stable operation process of earthquake early warning system based on machine learning: trial test and management perspective. Front. Earth Sci. 11, 1157742. doi:10.3389/feart.2023.1157742

CrossRef Full Text | Google Scholar

Ahn, J. K. (2021). Study on the preparation of disaster communication strategy: Focusing on the dissemination of early earthquake warnings, Master’s thesis. Seoul: Joongang University

Google Scholar

Allen, R. M. (1982). Automatic phase pickers: their present use and future prospects. BSSA 72 (6B), S225–S242. doi:10.1785/bssa07206b0225

CrossRef Full Text | Google Scholar

Allen, R. M., Gasparini, P., Kamigaichi, O., and Bose, M. (2009). The status of earthquake early warning around the world: an introductory overview. Seismol. Res. Lett. 80, 682–693. doi:10.1785/gssrl.80.5.682

CrossRef Full Text | Google Scholar

Allen, R. M., and Melgar, D. (2019). Earthquake early warning: advances, scientific challenges, and societal needs. Annu. Rev. Earth Planet. Sci. 47, 361–388. doi:10.1146/annurev-earth-053018-060457

CrossRef Full Text | Google Scholar

Angove, M., Arcas, D., Bailey, R., Carrasco, P., Coetzee, D., Fry, B., et al. (2019). Ocean observations required to minimize uncertainty in global tsunami forecasts, warnings, and emergency response. Front. Mar. Sci. 6, 350. doi:10.3389/fmars.2019.00350

CrossRef Full Text | Google Scholar

BBC News (2017). California earthquake alarm sounded - 92 Years late. Available at: https://www.bbc.com/news/technology-40366816 (Accessed June 22, 2017).

Google Scholar

BBC News (2018). False earthquake warning panics Japan. Available at: https://www.bbc.com/news/world-asia-42582113 (Accessed January 5, 2018).

Google Scholar

Binger, C., Ball, L., Dietz, A., Kent-Walsh, J., Lasker, J., Lund, S., et al. (2012). Personnel roles in the AAC assessment process. AAC 28, 278–288. doi:10.3109/07434618.2012.716079

PubMed Abstract | CrossRef Full Text | Google Scholar

Bostrom, A., McBride, S. K., Becker, J., Goltz, J. D., de Groot, R. M., Peek, L., et al. (2022). Great expectations for earthquake early warnings on the United States West Coast. Int. J. Disaster Risk Reduct. 82, 103296. doi:10.1016/j.ijdrr.2022.103296

CrossRef Full Text | Google Scholar

Chen, D. Y., Hsiao, N. C., and Wu, Y. M. (2015). The earthworm based earthquake alarm reporting system in Taiwan. Bull. Seismol. 105 (2A), 568–579. doi:10.1785/0120140147

CrossRef Full Text | Google Scholar

Chen, D. Y., Lin, T. L., Hsu, H. C., Hsu, Y. C., and Hsiao, N. C. (2019). An approach to improve the performance of the earthquake early warning system for the 2018 Hualien earthquake in Taiwan. Terr. Atmos. Ocean. Sci. 30, 423–433. doi:10.3319/TAO.2018.12.23.02

CrossRef Full Text | Google Scholar

Cho, S., Ahn, J. K., and Hwang, E. H. (2022). Optimization of network-based earthquake early warning systems on the Korean Peninsula. IEEE Access 10, 83931–83939. doi:10.1109/access.2022.3197661

CrossRef Full Text | Google Scholar

Chung, A. I., Henson, I., and Allen, R. M. (2019). Optimizing earthquake early warning performance: elarmS-3. Seismol. Res. Lett. 90 (2A), 727–743. doi:10.1785/0220180192

CrossRef Full Text | Google Scholar

Cochran, E. S., Kohler, M. D., Given, D. D., Guiwits, S., Andrews, J., Meier, M. A., et al. (2018). Earthquake early warning ShakeAlert system: testing and certification platform. Seismol. Res. Lett. 89 (1), 108–117. doi:10.1785/0220170138

CrossRef Full Text | Google Scholar

Convertito, V., Iervolino, I., Manfredi, G., and Zollo, A. (2008). Prediction of response spectra via real-time earthquake measurements. Soil Dyn. Earthq. Eng. 28, 492–505. doi:10.1016/j.soildyn.2007.07.006

CrossRef Full Text | Google Scholar

Cooke, R. M., and Goossens, L. H. (2004). Expert judgement elicitation for risk assessments of critical infrastructures. J. Risk Res. 7, 643–656. doi:10.1080/1366987042000192237

CrossRef Full Text | Google Scholar

Cremen, G., Galasso, C., and Zuccolo, E. (2022). Investigating the potential effectiveness of earthquake early warning across Europe. Nat. Commun. 13, 639. doi:10.1038/s41467-021-27807-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Cua, G., Fischer, M., Heaton, T., and Wiemer, S. (2009). Real-time performance of the virtual seismologist earthquake early warning algorithm in Southern California. Seismol. Res. Lett. 80, 740–747. doi:10.1785/gssrl.80.5.740

CrossRef Full Text | Google Scholar

Dallo, I., Marti, M., Clinton, J., Böse, M., Massin, F., and Zaugg, S. (2022). Earthquake early warning in countries where damaging earthquakes only occur every 50 to 150 years – the societal perspective. Int. J. Disaster Risk Reduct. 83, 103441. doi:10.1016/j.ijdrr.2022.103441

CrossRef Full Text | Google Scholar

Doi, K. (2011). The operation and performance of earthquake early warnings by the Japan Meteorological Agency. Soil Dyn. Earthq. Eng. 31, 119–126. doi:10.1016/j.soildyn.2010.06.009

CrossRef Full Text | Google Scholar

Esposito, M., Palma, L., Belli, A., Sabbatini, L., and Pierleoni, P. (2022). Recent advances in internet of things solutions for early warning systems: A review. Sensors 22, 2124. doi:10.3390/s22062124

PubMed Abstract | CrossRef Full Text | Google Scholar

Finazzi, F. (2020). The earthquake network project: A platform for earthquake early warning, rapid impact assessment, and search and rescue. Front. Earth Sci. 8, 243. doi:10.3389/feart.2020.00243

CrossRef Full Text | Google Scholar

Goda, K., Rimando, J., Peace, A. L., Sirous, N., Rosset, P., and Chouinard, L. (2023). Regional seismic loss estimation and critical earthquake scenarios for the western Quebec seismic zone. Georisk, 1–20. doi:10.1080/17499518.2023.2201246

CrossRef Full Text | Google Scholar

Hoshiba, M., and Ozaki, T. (2014). Earthquake early warning and tsunami warning of the Japan Meteorological Agency, and their performance in the 2011 off the Pacific Coast of Tohoku earthquake (M 9.0). Early Warning for Geological Disasters. Adv. Technol. Earth Sci., 1–28. doi:10.1007/978-3-642-12233-0_1

CrossRef Full Text | Google Scholar

Hsiao, N. C., Wu, Y. M., Shin, T. C., Zhao, L., and Teng, T. L. (2009). Development of earthquake early warning system in Taiwan. Geophys. Res. Lett. 36, L00B02. doi:10.1029/2008gl036596

CrossRef Full Text | Google Scholar

Iervolino, I., Convertito, V., Giorgio, M., Manfredi, G., and Zollo, A. (2006). Real time risk analysis for hybrid earthquake early warning systems. J. Earthq. Eng. 10, 867–885. doi:10.1142/S1363246906002955

CrossRef Full Text | Google Scholar

Iervolino, I., Giorgio, M., Galasso, C., and Manfredi, G. (2009). Uncertainty in early warning predictions of engineering ground motion parameters: what really matters? Geophys. Res. Lett. 36, L00B06. doi:10.1029/2008gl036644

CrossRef Full Text | Google Scholar

Iervolino, I. (2011). Performance-based earthquake early warning. Soil Dyn. Earthq. Eng. 31, 209–222. doi:10.1016/j.soildyn.2010.07.010

CrossRef Full Text | Google Scholar

Iervolino, I., Zollo, A., and Erdik, M. (2011). Foreword to: prospects and applications of Earthquake Early Warning for real-time earthquake engineering, risk management and loss mitigation. Soil Dyn. Earthq. Eng. 31, 105. doi:10.1016/j.soildyn.2010.11.001

CrossRef Full Text | Google Scholar

Ittner, C. D., and Larcker, D. F. (1997). Quality strategy, strategic control systems, and organizational performance. Acc. Organ. Soc. 22, 293–314. doi:10.1016/s0361-3682(96)00035-9

CrossRef Full Text | Google Scholar

Jackson, D. D., and Anderson, D. L. (1970). Physical mechanisms of seismic-wave attenuation. Rev. Geophys. 8, 1–63. doi:10.1029/rg008i001p00001

CrossRef Full Text | Google Scholar

Kamigaichi, O., Saito, M., Doi, K., Matsumori, T., Tsukada, S. Y., Takeda, K., et al. (2009). Earthquake early warning in Japan: warning the general public and future prospects. Seismol. Res. Lett. 80, 717–726. doi:10.1785/gssrl.80.5.717

CrossRef Full Text | Google Scholar

Kohler, M. D., Cochran, E. S., Given, D., Guiwits, S., Neuhauser, D., Henson, I., et al. (2018). Earthquake early warning ShakeAlert system: west Coast wide production prototype. Seismol. Res. Lett. 89, 99–107. doi:10.1785/0220170140

CrossRef Full Text | Google Scholar

Korea Meteorological Administration (2018). Government publications registration. Avaialble at: https://www.kma.go.kr/download_01/Annual_Report_2018.pdf.

Google Scholar

Korea Meteorological Administration (2017). Seismic observation data quality report. KMA, Seoul. Korea Rep., 279–288 (in Korean). Avaialble at: http://lod.nl.go.kr/resource/CNTS-00107278365.

Google Scholar

Kwon, H. R., Eoh, Y., and Park, S. H. (2020). The mediating role of catastrophizing in the relationship between emotional clarity and posttraumatic stress symptoms among earthquake survivors in Korea: A cross-sectional study. Front. Psychol. 11, 1114. doi:10.3389/fpsyg.2020.01114

PubMed Abstract | CrossRef Full Text | Google Scholar

Lassa, J. (2008). Measuring the sustainability of tsunami early warning systems: an interdisciplinary research agenda. Georisk 2, 187–194. doi:10.1080/17499510802369140

CrossRef Full Text | Google Scholar

Lee, J., Lee, D. K., and Ahn, J. K. (2022). Automatic fault plane solution for the provision of rapid earthquake information in South Korea. Sustainability 15, 520. doi:10.3390/su15010520

CrossRef Full Text | Google Scholar

Lim, D., Ahn, J. K., and Lee, J. (2020). Comparative analysis of past earthquake simulation and test operation. J. Korean Soc. Hazard Mitig. 20, 281–290. doi:10.9798/kosham.2020.20.5.281

CrossRef Full Text | Google Scholar

Lomax, A., Satriano, C., and Vassallo, M. (2012). Automatic picker developments and optimization: filterPicker—a robust, broadband picker for real-time seismic monitoring and earthquake early warning. Seismol. Res. Lett. 83, 531–540. doi:10.1785/gssrl.83.3.531

CrossRef Full Text | Google Scholar

Massin, F., Clinton, J., and Böse, M. (2021). Status of earthquake early warning in Switzerland. Front. Earth Sci. 9, 707654. doi:10.3389/feart.2021.707654

CrossRef Full Text | Google Scholar

May, P. J., Koski, C., and Stramp, N. (2016). Issue expertise in policymaking. J. Public Policy. 36, 195–218. doi:10.1017/s0143814x14000233

CrossRef Full Text | Google Scholar

Mayer, R., Plank, C., Bohner, A., Kollarits, S., Corsini, A., Ronchetti, F., et al. (2008). Monitor: hazard monitoring for risk assessment and risk communication. Georisk 2, 195–222. doi:10.1080/17499510802506139

CrossRef Full Text | Google Scholar

McBride, S. K., Bostrom, A., Sutton, J., de Groot, R. M., Baltay, A. S., Terbush, B., et al. (2020). Developing post-alert messaging for ShakeAlert, the earthquake early warning system for the West Coast of the United States of America. Int. J. Disaster Risk Reduct. 50, 101713. doi:10.1016/j.ijdrr.2020.101713

CrossRef Full Text | Google Scholar

Medina-Cetina, Z., and Nadim, F. (2008). Stochastic design of an early warning system. Georisk 2, 223–236. doi:10.1080/17499510802086777

CrossRef Full Text | Google Scholar

Meijer, A., and Grimmelikhuijsen, S. (2020). Responsible and accountable algorithmization: How to generate citizen trust in governmental usage of algorithms. LONDON: The Algorithmic Society.

Google Scholar

Minson, S. E., Meier, M. A., Baltay, A. S., Hanks, T. C., and Cochran, E. S. (2018). The limits of earthquake early warning: timeliness of ground motion estimates. Sci. Adv. 4, eaaq0504. doi:10.1126/sciadv.aaq0504

PubMed Abstract | CrossRef Full Text | Google Scholar

Murray, J. R., Crowell, B. W., Murray, M. H., Ulberg, C. W., McGuire, J. J., Aranha, M. A., et al. (2023). Incorporation of real-time earthquake magnitudes estimated via peak ground displacement scaling in the ShakeAlert Earthquake Early Warning System. Bull. Seismol. Soc. Am. 113, 1286–1310. doi:10.1785/0120220181

CrossRef Full Text | Google Scholar

Ni, C. F., Chung, C. C., Zhang, L., Wang, Y., and Dong, J. J. (2023). Preface of the special issue on ‘geological uncertainty and its impact on geohazards and water resources assessments and infrastructure design. ’ Eng. Geol. 313, 106981. doi:10.1016/j.enggeo.2022.106981

CrossRef Full Text | Google Scholar

Papadopoulos, A. N., Böse, M., Danciu, L., Clinton, J., and Wiemer, S. (2023). A framework to quantify the effectiveness of earthquake early warning in mitigating seismic risk. Earthq. Spectra 39, 938–961. doi:10.1177/87552930231153424

CrossRef Full Text | Google Scholar

Podolskiy, E. A., Murai, Y., Kanna, N., and Sugiyama, S. (2021). Ocean-bottom and surface seismometers reveal continuous glacial tremor and slip. Nat. Commun. 12, 3929. doi:10.1038/s41467-021-24142-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Porter, K. A. (2016). How many injuries can be avoided through earthquake early warning and drop, cover, and hold on? Boulder, Colorado: Structural Engineering and Structural Mechanics Program.

Google Scholar

Providakis, S., Rogers, C. D., and Chapman, D. N. (2022). 3D Spatiotemporal risk assessment analysis of the tunnelling-induced settlement in an urban area using analytical hierarchy process and BIM. Georisk 16, 251–266. doi:10.1080/17499518.2021.1952607

CrossRef Full Text | Google Scholar

Reddy, E. A. (2020). Crying ‘crying wolf’: how misfires and Mexican engineering expertise are made meaningful. Ethnos 85, 335–350. doi:10.1080/00141844.2018.1561489

CrossRef Full Text | Google Scholar

Ruhl, C. J., Melgar, D., Chung, A. I., Grapenthin, R., and Allen, R. M. (2019). Quantifying the value of real-time geodetic constraints for earthquake early warning using a global seismic and geodetic data set. J. Geophys. Res. Solid Earth. 124, 3819–3837. doi:10.1029/2018jb016935

CrossRef Full Text | Google Scholar

Saaty, T. L. (1990). How to make a decision: the analytic hierarchy process. Eur. J. Oper. Res. 48, 9–26. doi:10.1016/0377-2217(90)90057-i

CrossRef Full Text | Google Scholar

Santos-Reyes, J. (2019). How useful are earthquake early warnings? The case of the 2017 earthquakes in Mexico city. Int. J. Disaster Risk Reduct. 40, 101148. doi:10.1016/j.ijdrr.2019.101148

CrossRef Full Text | Google Scholar

Satriano, C., Elia, L., Martino, C., Lancieri, M., Zollo, A., and Iannaccone, G. (2011). PRESTo, the earthquake early warning system for southern Italy: concepts, capabilities and future perspectives. Soil Dyn. Earthq. Eng. 31, 137–153. doi:10.1016/j.soildyn.2010.06.008

CrossRef Full Text | Google Scholar

Satriano, C., Lomax, A., and Zollo, A. (2008). Real-time evolutionary earthquake location for seismic early warning. Bull. Seismol. Soc. Am. 98, 1482–1494. doi:10.1785/0120060159

CrossRef Full Text | Google Scholar

Sheen, D. H. (2016). Determination method for location and origin time of earthquake using arrival time of primary wave: KR patents KR101635791B1. Avaialble at: https://patents.google.com/patent/KR101635791B1/en. Accessed February 27, 2015.

Google Scholar

Sheen, D. H., Kang, T. S., and Rhie, J. (2018). A local magnitude scale for South Korea. Bull. Seismol. Soc. Am. 108, 2748–2755. doi:10.1785/0120180112

CrossRef Full Text | Google Scholar

Sumy, D. F., Jenkins, M. R., McBride, S. K., and de Groot, R. M. (2022). Typology development of earthquake displays in free-choice learning environments, to inform earthquake early warning education in the United States. Int. J. Disaster Risk Reduct. 73, 102802. doi:10.1016/j.ijdrr.2022.102802

CrossRef Full Text | Google Scholar

Sutton, J., Wood, M. M., Crouch, S., and Waugh, N. (2022). Public perceptions of U.S. Earthquake early warning post-alert messages: findings from focus groups and interviews. Int. J. Disaster Risk Reduct. 84, 103488. doi:10.1016/j.ijdrr.2022.103488

CrossRef Full Text | Google Scholar

Takabatake, T., and Kojima, T. (2023). Impact of rising sea levels on future nankai–tonankai earthquake tsunamis: A case study of osaka, Japan. Georisk 17, 595–611. doi:10.1080/17499518.2022.2119256

CrossRef Full Text | Google Scholar

Thaker, T. P., Savaliya, P. J., Patel, M. D., and Patel, K. (2018). “GIS based seismic risk analysis of Ahmedabad City, India,” in Proceedings of the GeoShanghai 2018 international conference: Geoenvironment and geohazard. GSIC 2018 (Singapore: Springer). doi:10.1007/978-981-13-0128-5_13

CrossRef Full Text | Google Scholar

Tidlund, M., Spross, J., and Larsson, S. (2022). Observational method as risk management tool: the hvalfjörður tunnel project, Iceland. Icel. Georisk. 17, 346–360. doi:10.1080/17499518.2022.2046784

CrossRef Full Text | Google Scholar

Too, E. G., and Weaver, P. (2014). The management of project management: A conceptual framework for project governance. Int. J. Proj. Manag. 32, 1382–1394. doi:10.1016/j.ijproman.2013.07.006

CrossRef Full Text | Google Scholar

Velazquez, O., Pescaroli, G., Cremen, G., and Galasso, C. (2020). A review of the technical and socio-organizational components of earthquake early warning systems. Front. Earth Sci. 8, 533498. doi:10.3389/feart.2020.533498

CrossRef Full Text | Google Scholar

Vihalemm, T., Kiisel, M., and Harro-Loit, H. (2012). Citizens’ response patterns to warning messages. J. Contingencies Crisis Manag. 20, 13–25. doi:10.1111/j.1468-5973.2011.00655.x

CrossRef Full Text | Google Scholar

Vipin, K. S., and Sitharam, T. G. (2011). Multiple source and attenuation relationships for evaluation of deterministic seismic hazard: logic tree approach considering local site effects. Georisk 5, 173–185. doi:10.1080/17499518.2010.532015

CrossRef Full Text | Google Scholar

Wen, H. (2015). A susceptibility mapping model of earthquake-triggered slope geohazards based on geo-spatial data in mountainous regions. Georisk 9, 25–36. doi:10.1080/17499518.2015.1005634

CrossRef Full Text | Google Scholar

Weyrich, P., Ruin, I., Terti, G., and Scolobig, A. (2021). Using serious games to evaluate the potential of social media information in early warning disaster management. Int. J. Disaster Risk Reduct. 56, 102053. doi:10.1016/j.ijdrr.2021.102053

CrossRef Full Text | Google Scholar

Whipple, J. M., and Frankel, R. (2000). Strategic alliance success factors. J. Supply Chain Manag. 36, 21–28. doi:10.1111/j.1745-493x.2000.tb00248.x

CrossRef Full Text | Google Scholar

Wu, W. N., Chang, S. M., and Wang, Y. W. (2022). Assessment of taiwan public warning cell broadcast service: insights from an expert focus group and citizens. Public Adm. Q. 46, 406–428. doi:10.37808/paq.46.4.6

CrossRef Full Text | Google Scholar

Yamada, M., Tamaribuchi, K., and Wu, S. (2021). The extended integrated particle filter method (IPFx) as a high-performance earthquake early warning system. Bull. Seismol. Soc. Am. 111, 1263–1272. doi:10.1785/0120210008

CrossRef Full Text | Google Scholar

Yeon, D. H., Chung, J. B., and Im, D. H. (2020). The effects of earthquake experience on disaster education for children and teens. Int. J. Environ. Res. Public Health. 17, 5347. doi:10.3390/ijerph17155347

PubMed Abstract | CrossRef Full Text | Google Scholar

Zuccolo, E., Cremen, G., and Galasso, C. (2021). Comparing the performance of regional earthquake early warning algorithms in Europe. Front. Earth Sci. 9, 686272. doi:10.3389/feart.2021.686272

CrossRef Full Text | Google Scholar