Article citation information:

Okoro, O.C., Zaliskyi, M., Serhii, D., Abule, I. An approach to reliability analysis of aircraft systems for a small dataset. Scientific Journal of Silesian University of Technology. Series Transport. 2023, 118, 207-217. ISSN: 0209-3324. DOI: https://doi.org/10.20858/sjsutst.2023.118.14.

Onyedikachi Chioma OKORO[1], Maksym ZALISKYI[2], Dmytriiev SERHII[3], Ibinabo ABULE[4]

AN APPROACH TO RELIABILITY ANALYSIS OF AIRCRAFT SYSTEMS FOR A SMALL DATASET

Summary. Data-driven predictive aircraft maintenance approach typically results in lower maintenance costs, avoiding unnecessary preventive maintenance actions and reducing unexpected failures. Information provided by a reliability analysis of aircraft components and systems can improve an existing maintenance strategy and ensure an optimal maintenance task interval. For reliability work, the exponential distribution is typically used; however, this approach requires substantial amounts of data, which often may not be generated by aviation operations. Therefore, this study proposes a method for reliability analysis given a small dataset. Real-life historical data of an aircraft operating in Nigeria validate the proposed approach and prove its applicability.

Keywords: aircraft maintenance, reliability, small dataset, aircraft systems

1. INTRODUCTION

Aircraft maintenance, an integral aspect of aircraft operations, is a general term for aircraft checks that assess aircraft and the condition of their component parts and systems. It ensures the airworthiness of the fleet and includes short pre-flight checks or detailed checks of the aircraft components and systems. Effective aircraft maintenance focuses on ensuring that the required levels of flight safety and reliability are met, and also, in the case of failure, maintenance restores the safety and reliability levels to the required standards [1-4]. The most widely applied aircraft maintenance strategies are corrective and preventive maintenance actions. Corrective maintenance tasks are connected to run-to-failure maintenance strategies, while preventive maintenance work is performed as part of a fixed interval to replace, repair, or restore. It encompasses work done under a fixed-interval restoration/repair strategy and conducted based on a time or machine-run-based schedule that detects, precludes, or mitigates degradation [5]. These traditional aircraft maintenance strategies lack predictive capabilities and often lead to maintenance being performed too early, that is, before the end of a machine’s useful life, or too late, .that is, after a costly failure [6]. Therefore, a data-driven predictive and condition-based aircraft maintenance approach will result in lower maintenance costs, avoiding unnecessary preventive maintenance actions and reducing unexpected failures. A combination of preventive and predictive maintenance results in 18.5% less unplanned downtime and 87.3% fewer defects for more reliance on predictive than preventive maintenance [7].

Predictive Maintenance is one of the core pillars of Industry 4.0, and in comparison to corrective and preventive maintenance, it allows for more cost-effective operations. It is performed as part of a condition-based strategy which involves measuring the condition of equipment and assessing whether it will fail during some future period. Early approaches to predictive maintenance focused on hand-crafted, physical models and heuristics and lately, data-driven methods, are on the rise because they can be scaled to multiple systems without the need for specific domain knowledge [8, 9]. Cloud computing, wider availability of data and models, and other Industry 4.0 developments are creating a paradigm shift in how maintenance work is planned and executed. In the nearest future, aircraft maintenance will be initiated once a potential failure is detected and completed before the occurrence of functional failure . Predictive maintenance tasks are determined by the Original Equipment Manufacturer’s (OEM) recommendations and strategy development decision trees such as Reliability-Centred Maintenance (RCM) that considers failure behaviour and consequence [5].

1.1 An overview of data-driven maintenance

Data-driven maintenance methods originate from statistics and machine learning techniques. To use data-driven methods purposefully, a structural understanding of the behaviour being modelled is not needed, but run-to-failure data for each fault mode of the system should be made available [10]. Van Staden et al. investigated how historical machine failures and maintenance records can be used to determine future estimates of machine failure and, consecutively, prescribe improvements of scheduled preventive maintenance interventions. The authors modelled the problem using a finite horizon Markov decision process with a variable order Markov chain, in which the chain length varies based on the time since the last preventive maintenance action was conducted. The prescriptive optimization model captures the dependency of a machine’s failures on both recent failures in addition to preventive maintenance actions. To improve predictions for machine failure behaviour, the authors pooled datasets over different machine classes using a Poisson generalized linear model [6].

Operational data such as past aircraft faults/failures and maintenance actions can be used to estimate the probability of aircraft component failure and plan maintenance actions accordingly. However, the downside lies in the fact that sufficient run-to-failure datasets are crucial to the successful realization of predictive maintenance, and existing models show the worst performance for small datasets [11]. Furthermore, small datasets are a bad approximation of true randomness, and the variance of the decoding accuracy is high [12]. Therefore, the implementation of predictive maintenance strategies may pose challenges to aviation operations which generate a small dataset. Considering that RCM is a key component of predictive maintenance, we proposed a model for calculating reliability parameters based on failure probability. The proposed method is described in the method section and summarized in Figure 1. Real-life historical dataset of pilot and maintenance records of faults/failures from an aircraft operating in Nigeria was used for this research. The dataset was transformed into a more usable form to be used as input data . The results of the proposed model can provide insights into future faults/failures of aircraft components, sub-systems and systems and can be used to supplement existing aircraft maintenance strategies. This approach is expected to reduce waste arising from early maintenance and failure costs connected with late maintenance actions [6].

1.2 Small dataset problems

Small datasets reduce statistical significance and pose limitations [13], thereby making it difficult to reach any general conclusions [14]. A small dataset causes the estimation performance of a developed model to be poor. When there are many independent variables, a model becomes complicated, and a small dataset further invalidates the estimation method. At high total flight hours, small datasets produce large confidence intervals, which imply lower statistical reliability – a key disadvantage of using a small dataset is the lack of statistical stability [15, 16]. In specific cases of testing predictive models, small datasets are tougher because they are not offset with large effect sizes, and they undermine accurate tests with predictive models [17].

For small datasets, the model selected by the Akaike information criterion appears to be anti-conservative even with regard to the maximum Type I error rate of the maximal model [18]. A possible solution to the small dataset problem is the use of pre-trained networks, also referred to as transfer learning. This is achieved by initializing the neural network with the weights trained in the related domains and finetuning the model with in-domain data. This approach speeds up training and has gained popularity in various industries for handling the lack of significant samples in a dataset [19]. Additionally, exact non-parametric tests can be used to overcome problems associated with small datasets in hypothesis testing. The p-values in non-parametric tests calculate the exact probability of obtaining observed or extreme results under the null hypothesis [20]. Deep convolutional neural networks can be used to fit small datasets with simple and proper modification without the need to redesign specific small networks [21]. Proportion distribution of outliers and small datasets narrows the performance difference between models in a test set because the advantages and disadvantages of the model are not fully discovered [22]. For a small dataset with existing outliers, Liang et al. [23] proposed a generalized mean distance-based K-nearest neighbour by introducing multi-generalized mean distances and the nested generalized mean distance, which are based on the characteristic of the generalized mean.

In comparison to conventional analysis, the Bayesian approach to inference has the advantage of handling uncertainty for small datasets in aircraft fleet-wide prognostics [24]. The Bayesian Markov chain Monte Carlo approach allows for accurate reliability evaluation using a numerical simulation method given non-informative prior information but only works when the sample size is at least 10 [25]. A combination of variable importance in the projection analysis method and regression models can be used to tackle the problem of small dataset studies of cost estimation for general aviation aircraft [26]. Decoding performance is shown by how much classification results depart from the rate obtained by purely random classification. In a 2-class or 4-class classification problem, the chance levels are 50% or 25%, respectively, but these thresholds do not hold for small datasets [12].

2. METHOD

Reliability-Centred Maintenance is a vital component of predictive maintenance strategy. During the operation and support phase, the reliability of the aircraft and its components is of paramount importance to flight safety and availability. The reliability process allows aircraft operators to analyse the data of aircraft component parts and systems. An operator can compare the reliability of the entire fleet to understand the cost of schedule interruptions, analyse solutions, and prioritize service bulletins based on impact on the fleet.

Reliability is generally measured by a failure probability, and optimization ensures that the latter remains lower than the given threshold [27]. The relationship between the reliability and failure probability of an aircraft component or system j is given by

(1)

where – failure probability and – reliability

Over the last two decades, reliability analysis methods have been developed – stimulating interest in the probabilistic treatment of structures [28]. Reliability analysis involves the evaluation of the level of safety of a system. In engineering, the exponential distribution is the most used probability distribution, particularly in reliability work. However, statistical simulation results show that a minimum sample size of 35 is required to use an exponential distribution for reliability analysis, thus making this approach unsuitable for small datasets typically generated by small-scale aviation operations. Therefore, this study proposes a method for determining reliability with a small dataset using failure probability.

2.1 Method for reliability analysis of aircraft systems given a small dataset

The proposed model calculates reliability based on the failure probability of an aircraft component or system. The input data is a continuous statistical data x_iwith sample size n, which is extracted from the pilot and maintenance reports of faults/failures of the observed interval. The method for finding the failure probability is shown in Figure 1.

The steps in the method are as follows:

Step 1. Determine the number of observations for tails approximation j= . For the approximation, Chauvenet’s criterion is used with the transformation of the following type:

(2)

where is an approximated variable, Med is the median value of the sample, F is the basis function, K_i is a quantile of normal distribution with zero expectation and standard deviation of 1, and V is the variation coefficient .

Step 2. To determine the values of the lower (y_i_lower) and upper tail (y_i_upper), the transformed sample (order) is obtained as follows:

(3)

where is the order statistics for input data .

Fig. 1. Flow chart of the method for reliability analysis of
aircraft systems given a small dataset

Step 3. Calculate the sums of the first ( ₁) and last ( ₂) random variables using the transformed order statistic:

; (4)

where j depends on the sample size.

Step 4. Corresponding quantiles of the standard normal distribution after the transformation are calculated according to the Kazakyavicius equation:

(5)

where is the empirical probabilities of each observation of order statistic ,
i = 0…n.

Step 5. The products of the variation coefficient and the sum of corresponding quantiles is calculated thus:

; (6)

Step 6. The transformation basis for the minimum (β₁) and maximum (β₂) is determined using:

; (7)

Step 7. Calculation of the basis function using the following formulas:

, (8)

, (9)

(10)

where K_swis the quantile value that corresponds to the switching point, b is a coefficient determined by the formula:

(11)

Step 8. Computing the values of the variables Q₁, Q₂ and Q₃and plotting graphs:

(12)

(13)

(14)

where Q₁ and Q₂ define the faults/failures using the proposed method for small datasets, while Q₃ is in accordance with exponential distribution.

3. ANALYSIS AND RESULTS

Real-life historical datasets of pilot and maintenance reports of faults/failures from an aircraft operating in Nigeria were used for this study. To further reduce the sample size, one system was selected from a basic sample of the statistical data, and the dataset was transformed into a more usable form to be used as input data for the proposed algorithm. The number of faults/failures n_T is given in Table 1.

Tab. 1

Faults/failures information of the aircraft system

*x_i*	*n_T*	*x_i*	*n_T*	*x_i*	*n_T*	*x_i*	*n_T*	*x_i*	*n_T*
x₀	3	x₃	1	x₆	3	x₉	5	x₁₂	4
x₁	1	x₄	8	x₇	3	x₁₀	7	x₁₃	5
x₂	1	x₅	2	x₈	5	x₁₁	10	x₁₄	9

There are no outliers; therefore, Chauvenet’s criterion is not applied.

j = 3.615 ; 2.273;

Corresponding quantiles of the standard normal distribution are shown in Table 2.

Tab. 2

Quantiles of the standard normal distribution

*K_i*		*K_i*		*K_i*		*K_i*		*K_i*
K₀	-3.111	K₃	-2.252	K₆	-1.735	K₉	1.897	K₁₂	2.464
K₁	-2.727	K₄	-2.066	K₇	0	K₁₀	2.066	K₁₃	2.727
K₂	-2.464	K₅	-1.897	K₈	1.735	K₁₁	2.252	K₁₄	3.111

3.693; 3.693; ; 1.596

The values of the basis function are given in Tables 3 and 4.

Tab. 3

Values of basis function F₁ (K_i)

*F₁ (K_i)*		*F₁ (K_i)*		*F₁ (K_i)*		*F₁ (K_i)*		*F₁ (K_i)*
F₁ (K₀)	2.118	F₁ (K₃)	2.113	F₁ (K₆)	2.103	F₁ (K₉)	1.608	F₁ (K₁₂)	1.600
F₁ (K₁)	2.117	F₁ (K₄)	2.111	F₁ (K₇)	1.858	F₁ (K₁₀)	1.605	F₁ (K₁₃)	1.599
F₁ (K₂)	2.115	F₁ (K₅)	2.107	F₁ (K₈)	1.612	F₁ (K₁₁)	1.602	F₁ (K₁₄)	1.597

Tab. 4

Values of basis function F₂ (K_i).

*F₂ (K_i)*		*F₂(K_i)*		*F₂(K_i)*		*F₂ (K_i)*		*F₂ (K_i)*
F₂ (K₀)	2.119	F₂ (K₃)	2.707	F₂ (K₆)	2.572	F₂ (K₉)	1.623	F₂ (K₁₂)	1.475
F₂ (K₁)	2.831	F₂ (K₄)	2.659	F₂ (K₇)	2.119	F₂ (K₁₀)	1.579	F₂ (K₁₃)	1.407
F₂ (K₂)	2.762	F₂ (K₅)	2.614	F₂ (K₈)	1.666	F₂ (K₁₁)	1.531	F₂ (K₁₄)	1.596

The prognostic variables, Q₁ and Q₂, are calculated according to step 8. Q₁ and Q₂ are based on the proposed method for reliability analysis given a small dataset. The graph in Figure 2 shows the quantiles of normal distribution according to Kazakyavicius equation. An additional graph (Figure 3), referred to as the failure probability graph is also plotted in accordance with formula 15.

(15)

To determine the reliability of an aircraft component, subsystem or system over a given period, the first step is to determine the quantile, after which the failure probability is determined using Figure 3. For example, the forecast of five faults/failures, according to Figure 2, is in the 0.7 quantile; this corresponds to a failure probability of 0.25 (25%) and reliability of 0.75 (75%).

Fig. 2. Quantiles of normal distribution

Fig. 3. Failure probability graph

4. CONCLUSION

Traditional aircraft maintenance strategy involves corrective and preventive maintenance actions, which lead to maintenance being carried out too late, that is, when the aircraft component or system has failed or too early before the end of the useful life of the aircraft component or system. These maintenance strategies lack predictive capability, hence predictive and condition-based maintenance strategies, based on observed condition information and historical trends, promise significant cost savings and effectiveness.

Data-driven predictive aircraft maintenance usually results in lower maintenance costs, avoiding unnecessary preventive maintenance actions and reducing unexpected failures. Without condition monitoring information (sensors), operational data such as past aircraft faults/failures and maintenance actions can instead be used for the reliability analysis of aircraft components and systems. Sufficient data is, however, needed for the analysis because of the problems posed by a small dataset. Small datasets are statistically unreliable, have a bad approximation of true randomness, and their variance of decoding accuracy is high. Therefore, this study proposes a method for the reliability analysis of small datasets. The simulation results, which are based on real-life operational aircraft data, prove the applicability of the proposed approach.

The proposed method can be used to improve an aircraft’s maintenance strategy by providing insights into the reliability of aircraft components, sub-systems, and systems. This information can supplement an existing aircraft maintenance strategy to reduce waste caused by early maintenance and failure costs connected with late maintenance actions

References

1. US 7050894. System and method for diagnosing aircraft components for maintenance purposes. EADS Deutschland GmbH and Airbus Operations GmbH. (Halm J., Hechtenberg K.V. Kolander W.). 2006.

2. US 0073419 A1. Platform for Aircraft Maintenance Services. Airbus Operations GmbH. (Marwedel S., Reitmann J., Poupard M.). 2013.

3. US 9767413 B2. Maintenance and Computer Program for the Maintenance Aid of Aircraft Equipment. Airbus Operations SAS. (Huet J., Besseau S., Maillard B., Michaud F.). 2017.

4. Hinsch Martin. 2018. Industrial Aviation Management: A Primer in European Design, Production and Maintenance Organisations. Springer-Verlag GmbH Germany. ISBN 978-3-662-54739-7.

5. Hodkiewicz Melinda, Sarah Lukens, Michael P. Brundage, Thurston Sexton. 2021. “Rethinking maintenance terminology for an industry 4.0 future”. International Journal of Prognostics and Health Management 12(1). DOI: https://doi.org/10.36001/ijphm.2021.v12i1.2932.

6. van Staden Heletjé E., Laurens Deprez, Robert N. Boute. 2022. “A dynamic “predict, then optimize” preventive maintenance approach using operational intervention data”. European Journal of Operational Research 302(3). DOI: https://doi.org/10.1016/j.ejor.2022.01.037.

7. Thomas Douglas S., Brian Weiss. 2021. “Maintenance Costs and Advanced Maintenance Techniques: Survey and Analysis”. International Journal of Prognostics and Health Management 12(1). DOI: https://doi.org/10.36001/ijphm.2021.v12i1.2883.

8. Krokotsch Tilman, Mirko Knaak, Clemens Gühmann. 2021. “Improving Semi-Supervised Learning for Remaining Useful Lifetime Estimation Through Self-Supervision”. International Journal of Prognostics And Health Management. DOI: https://doi.org/10.36001/ijphm.2022.v13i1.3096.

9. Oikonomou Athanasios, Nick Eleftheroglou, Floris Freeman, Theodoros Loutas, Dimitrios Zarouchas. 2022. “Remaining Useful Life Prognosis of Aircraft Brakes”. International Journal of Prognostics and Health Management. DOI: https://doi.org/10.36001/ijphm.2022.v13i1.3072.

10. Hagmeyer, Simon, Fabian Mauthe, Peter Zeiler. 2021. “Creation of Publicly Available Data Sets for Prognostics and Diagnostics Addressing Data Scenarios Relevant to Industrial Applications”. International Journal of Prognostics and Health Management. DOI: https://doi.org/10.36001/ijphm.2021.v12i2.3087.

11. Liem Rhea P., Charles A. Mader, Joaquim RRA Martins. 2015. “Surrogate models and mixtures of experts in aerodynamic performance prediction for aircraft mission analysis”. Aerospace Science and Technology 43: 126-151. DOI: https://doi.org/10.1016/j.ast.2015.02.019.

12. Combrisson Etienne, Karim Jerbi. 2015. “Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy”. Journal of neuroscience methods 250: 126-136. DOI: https://doi.org/10.1016/j.jneumeth.2015.01.010.

13. Sinha Raj K., Suken A. Shah, Eric L. Hume, Rocky S. Tuan. 2002. “The effect of a 5-day space flight on the immature rat spine”. The Spine Journal 2(4): 239-243. DOI: https://doi.org/10.1016/S1529-9430(02)00197-3.

14. Yang Ji Hyun, Quinn Kennedy, Joseph Sullivan, and Ronald D. Fricker. 2013. “Pilot performance: assessing how scan patterns & navigational assessments vary by flight expertise”. Aviation, Space, And Environmental Medicine 84(2): 116-124. DOI: https://doi.org/10.3357/ASEM.3372.2013.

15. Knecht William R. 2013. “The “killing zone” revisited: Serial nonlinearities predict general aviation accident rates from pilot total flight hours”. Accident Analysis & Prevention 60: 50-56. DOI: https://doi.org/10.1016/j.aap.2013.08.012.

16. English J. Morley, Gerard L. Kernan. 1976. “The prediction of air travel and aircraft technology to the year 2000 using the Delphi method”. Transportation Research 10(01): 1-8. DOI: https://doi.org/10.1016/0041-1647(76)90094-0.

17. Varoquaux Gaël. 2018. “Cross-validation failure: Small sample sizes lead to large error bars”. Neuroimage 180: 68-77. DOI: https://doi.org/10.1016/j.neuroimage.2017.06.061.

18. Matuschek Hannes, Reinhold Kliegl, Shravan Vasishth, Harald Baayen, Douglas Bates. 2017. “Balancing Type I error and power in linear mixed models”. Journal Of Memory And Language 94: 305-315. DOI: https://doi.org/10.1016/j.jml.2017.01.001.

19. D’souza Rhett N., Po-Yao Huang, Fang-Cheng Yeh. 2020. “Structural analysis and optimization of convolutional neural networks with a small sample size”. Scientific Reports 10(1): 1-13.

20. Dwivedi Alok Kumar, Indika Mallawaarachchi, Luis A. Alvarado. 2017. “Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method”. Statistics in Medicine 36(14): 2187-2205. DOI: https://doi.org/10.1002/sim.7263.

21. Liu Shuying, Weihong Deng. 2015. “Very deep convolutional neural network based image classification using small training sample size”. 2015. In: 3rd IAPR Asian conference on pattern recognition (ACPR): 730-734. IEEE. DOI: https://doi.org/10.1109/ACPR.2015.7486599.

22. Han Liang, Guijun Yang, Huayang Dai, Bo Xu, Hao Yang, Haikuan Feng, Zhenhai Li, Xiaodong Yang. 2019. “Modeling maize above-ground biomass based on machine learning approaches using UAV remote-sensing data”. Plant Methods 15(1): 1-19.

23. Gou Jianping, Hongxing Ma, Weihua Ou, Shaoning Zeng, Yunbo Rao, Hebiao Yang. 2019. “A generalized mean distance-based k-nearest neighbor classifier”. Expert Systems with Applications 115: 356-372. DOI: https://doi.org/10.1016/j.eswa.2018.08.021.

24. Zaidan Martha A., Robert F. Harrison, Andrew R. Mills, Peter J. Fleming. 2015. “Bayesian hierarchical models for aerospace gas turbine engine prognostics”. Expert Systems with Applications 42(1): 539-553. DOI: https://doi.org/10.1016/j.eswa.2014.08.007.

25. Wang Chao, Jilian Guo, Anwei Shen. 2020. “Sensitivity analysis of censoring data from component failure analysis and reliability evaluation for the aviation internet of things”. Computer Communications 157: 28-37. DOI: https://doi.org/10.1016/j.comcom.2020.04.003.

26. Chen Xiaonan, Jun Huang, Mingxu Yi. 2020. “Cost estimation for general aviation aircrafts using regression models and variable importance in projection analysis”. Journal of Cleaner Production 256. DOI: https://doi.org/10.1016/j.jclepro.2020.120648.

27. Dubourg Vincent. 2011. “Adaptive surrogate models for reliability analysis and reliability-based design optimization”. PhD dissertation. Université Blaise Pascal-Clermont-Ferrand II.

28. Papadopoulos Vissarion, Dimitris G. Giovanis, Nikos D. Lagaros, Manolis Papadrakakis. 2012. “Accelerated subset simulation with neural networks for reliability analysis”. Computer Methods in Applied Mechanics and Engineering 223: 70-80. DOI: https://doi.org/10.1016/j.cma.2012.02.013.

Received 30.10.2022; accepted in revised form 10.01.2022

Scientific Journal of Silesian University of Technology. Series Transport is licensed under a Creative Commons Attribution 4.0 International License

[1] Department of Continuing Airworthiness, National Aviation University, Liubomyra Huzara Ave, 1, Kyiv, Ukraine. Email: okorokachi7@gmail.com. ORCID: https://orcid.org/0000-0001-5968-0424

[2] Department of Telecommunication and Radioelectronic Systems, National Aviation University, Liubomyra Huzara Ave, 1, Kyiv, Ukraine. Email maximus2812@ukr.net. ORCID: https://orcid.org/0000-0002-1535-4384

[3] Department of Continuing Airworthiness, National Aviation University, Liubomyra Huzara Ave, 1, Kyiv, Ukraine. Email: sad@nau.edu.ua. ORCID: https://orcid.org/0000-0002-4461-1837

[4] Bristow Helicopters (Nigeria) Ltd, General Aviation Area, Murtala Muhammed Airport, Ikeja, Nigeria. Email: Email: iabule@yahoo.co.uk. ORCID: https://orcid.org/0000-0003-0558-1150