Share It On

1. Introduction

While in general ECA endorses EBT and the principle of ‘less checking more training’ a cautious approach to the implementation of EBT is crucial, while keeping in mind several essential safeguards. Implementation of EBT means a paradigm shift and cannot be understood simply as replacement of a sometimes-outdated set of critical events with a new set. Crucially, EBT implementation must have as a goal more effective training programs with associated improvements in operational safety. 


In order to achieve that, it is crucial that EBT programs are closely linked to the respective operator’s environment and are not generic. Also, before a competent authority approves an EBT program for a respective operator, it must assess the operator’s capability to support the implementation of such program. Where a competent authority grants an approval for EBT programs, inspectors must receive qualification and training in EBT principles, application, approval processes and continuing oversight.


The availability of data covering both flight operations and training activity has improved substantially over the past years. ECA therefore supports the implementation of EBT as a logical step to update the current training practices in the light of evidence from these data sources. It is however of utmost importance that data collection and the protection of data are at an adequate level, and only deindividualized data is being used for the training purposes.


This paper outlines ECA’s views and concerns on selected proposals as introduced via NPA 2018-07, that - if approved and implemented - could have negative consequences on the quality of future pilot training.

2.   ECA’s key concerns as identified in NPA 2018-07

To deliver on the anticipated improvements in pilot training quality and associated operational safety, the following concerns need to be addressed: 

  • Too loose link of the EBT program to the operator
  • Insufficient involvement of the authorities in the approval of the operator’s competency framework 
  • Insufficient requirements for EBT inspectors (need for qualification & training in EBT principles, application, approval processes and continuing oversight)
  • License revalidation by an instructor who has not performed the final competency assessment
  • Limited or no line flying practice for instructors 
  • Practical assessment in competencies by an SFI [1]
  • Training provided by pilots not being FCL instructors
  • Possibility for a renewal of type rating within an EBT program
  • Lack of provisions for adequate data collection and protection (use of deindividualized data for training purposes).
3.    Detailed comments

  • Operational characteristics of the operator and a key role to play for the national aviation authorities

EBT programs cannot be generic but must be linked to the respective operator’s environment. When enhancing a baseline EBT training program it is important to first analyse the specific operational characteristics of the operator. This includes aircraft types, route structure and typical sector lengths, special operations, destinations requiring special attention, pilot experience levels and company/safety culture. It is very important to focus on the most critical operational risks identified and the training that can demonstrably mitigate these. Hence, there must be a close correlation between training and operations
Before approving an EBT programme for a respective operator - the competent authority must assess the capability of an operator to support the implementation of the EBT programme. The criteria of maturity of operator’s safety management system must be clearly defined in EASA’s governing regulation, and detailed guidance must be provided to enable the authority to assess what is meant by maturity. When capability is simply linked to the availability of resources, important aspects of such maturity might not be captured.
Moreover, where a competent authority grants an approval for EBT programmes, inspectors must have received qualification and training in EBT principles, application, approval processes and continuing oversight. The competent authority shall assess and oversee the EBT programme, together with the processes that support the implementation of the EBT programme. For proper approval and oversight of evidence-based training programmes - it is therefore important that the inspectors are trained in EBT assessment of competence in the same way as any EBT instructor to be able to perform efficient supervision. 
Due to the lack of experience and the newness of the EBT concept, we are going to observe a disparity of criteria amongst competent authorities, which may affect crews, operators, and therefore might create an unfair competition by different requirements. National authorities will need to be trained and guided, to ensure that their assessment and oversight is standardised and aligned with the EASA principles. 
ECA proposes EASA to provide and be responsible for the training, guidance, oversight and final approval of EBT programs, at least in the initial stages of mixed and base line implementation.

  • Competency framework & involvement of the authorities

Just like for quality processes and instructors’ standardization, the EBT concept offers the possibility for an operator to assess competencies with different frameworks. It is therefore the authority’s responsibility to approve the competency framework used by the operator’s program.
Article 30 of the ICAO Convention puts in place the need of mutual recognition of licenses. Moving from a prescriptive to a competency-based system should be done with an equivalent level of responsibility of ICAO states.
An adapted competency framework is the DNA of an EBT program. The complexity of the task – i.e. of setting up such a competency framework and adapting it to the operational environment of the respective operator / ATO – is high enough to justify the authorities’ commitment, starting by defining what such framework means and entails.
Licenses validities are based on delivery and revalidation processes. As EBT will introduce a new way to revalidate licenses and Class or Type rating – with the use of CBT – the revised ICAO PANS training [2] should be the base line for EASA’s implementation and taking into account ICAO’s work on CBT.

  • EBT program and license validation

Only instructors who are enrolled and thoroughly trained in the operators’ EBT program can be at the right level of knowledge for both the operational environment and the ATO specific competency model, and with the relevant experience [3].
ECA opposes the proposed concept of license revalidation [4] allowing for a delegation of signature to someone who has not done the final competency assessment (or has not even been involved in the assessment at all). In other words: The examiner (TRE) conducting the respective EBT module must be the person signing the revalidation.

  • Instructors and examiners 

Integral and key part of any operator’s training corpse are the instructors and examiners. 
One of the rationales as introduced in the NPA is that EASA foresees monitoring only for examiners [5] however wants to include qualified commanders and SFI in crucial parts of the program without any regulatory quality monitoring being required.
An EBT instructor must be enrolled in the operators’ EBT program and have successfully completed the Operator's recurrent program [6]. This will ensure that the instructors providing EBT are flying the line for this designated EBT operator which is essential as this is a recurrent training scheme, not a qualification program.
Air operations are evolving very quickly, as technology is in permanent evolution. Therefore, only regular exposure to normal operation will allow instructors to maintain a strong link between line experience and training – which is a fundamental pillar of the EBT concept. 
Line operations are under the privileges of TRI/TRE [7] as it is required to hold a valid licence to train or check. Hence, only TRI/TRE are relevant for operational assessment in an EBT program.

o   SFI and the issue of competence and currency

ECA opposes the use of SFI without adequate training & experience in the context of EBT. No practical assessment in competencies can be delegated to SFI in the context of EBT. 
According to NPA 2018-07, EBT is only considered for recurrent training for pilots already type qualified, and it is also used to validate the operational proficiency check. SFIs do not have any requirement to have line experience (1500 hours in a multi-pilot airplane) and even less to have recent experience of line flying in the airline. Therefore, they do not possess the competence to assess a pilot during a Line Orientated Exercise (LOE) or Scenario Based Training (SBT). This assessment can only be done by a TRE-EBT or a TRI-EBT.

The above comes from the definition of competency [8] and shows that to be able to observe a competency the instructor must have a practical knowledge of the line flying activity.

In addition to the above, instructors must be trained in, or hold, CRM-I in order to become an EBT instructor. Without a deep understanding of human factors and CRM, it is impossible for an instructor (regardless of the type and/or experience) to be able to identify, train and assess all competencies.
The minimum level of instructor for EBT is TRI. As the validation of a module is giving credits against ICAO Annex 6 on one hand and allowing less frequent line checks on the other hand, the validation of one module cannot be done by less than a trained, standardised TRI.
Moreover, having the ability to accurately apply the principles of fault analysis should be a major determinant in the selection process of an instructor who will be expected to conduct a competency-based training program such as EBT.
Finally, ECA cannot support that FCL training is provided by pilots not being FCL instructors at all. This would not only create a legal loophole as those EBT instructors could potentially be trained by persons not proficient, but also not entitled to deliver FCL assessment.

  • Renewal of type-rating

ECA is opposed to the possibility of renewal of type rating within EBT programme. As EBT is a new way of training for recurrent training, it is not mature enough to deliver or renew a licence, class or type rating. Renewal should remain under the existing training scheme. 

  • Data collection and training data monitoring programme

EBT is data driven. It is therefore of utmost importance that data collection and the protection of data are at an adequate level. This is not only relevant for protection of the system but also of the individual personal data.
The operator must therefore establish and maintain a training data monitoring programme, of a non-punitive nature, containing adequate safeguards to protect the source(s) of the data.
Flight data (as well as any other data collected) must be deindividualized, before it can be used for training purposes. 

  • Assessment of non-technical skills & non-verbal communication

Assessment of Non-Technical Skills requires a high level of training and standardization amongst the instructor corpse. Even for highly trained human factors specialists and psychologists it is almost impossible to correctly assess non-verbal communication, therefore behavioral indicators like “correctly interprets body language" should not be used towards a final competency assessment.

  • Mind change for the examiners & a need for a change of thinking

Especially for scenario-based evaluation and training the range and variety of possible scenarios will increase and possibly differ from current LOFT scenarios through including other operational aspects than purely technical ones (e.g. including incidents in the cabin, on ground, Dangerous Goods issues and other external threats). This will require different (e.g. role play) qualities amongst the instructor corpse, which will need to be addressed in instructor training/recurrent training.

Appendix

Definitions - CBT vs EBT

There seems to be, many times, confusion about the terms CBT and EBT, with some experts even using these terms interchangeability. We have therefore identified a need for a simple terminology clarification:
Competency based training (CBT) is an underpinning concept, concentrating on the output of training. Teaching and learning using this approach aims at developing and strengthening concrete skills rather than learning abstract concepts. In CBTA - training and assessment are oriented at performance, with an emphasis on the standards of performance and their measurement. That on the other hand requires properly defined competencies and competence standards. 
Moreover, in CBTA focus is placed on the training and developing of competencies rather than on the pure assessment. Some competencies will have to develop over time and cannot be taught as such. The assessment as such must focus on the enhancement of the competency rather than just following a fail/pass concept. CBTA can exist independently from EBT. One example where that is the case – is MPL which shifted focus from prescriptive requirements to competency training and assessment.
Evidence based training (EBT), on the contrary, relies on the concept of CBTA. According to the EBT methodology, as the name suggests, the curriculum/program for recurrent training is built on evidence, using the underpinning concept of CBT. In EBT, the development of the curriculum is data driven, synthesizing information from e.g. flight data records, training data, audit observations, accidents and incidents [9].
The approval of the program should be focused on the process and validation of developing the curriculum rather than on the contents itself. [10]


New approach to pilot training

Implementation of Evidence Based Training (EBT) by the operators means a paradigm shift. It cannot be simply understood as replacement of sometimes-outdated set of critical events with a new set. Rather, the scenario-based events should be used as a vehicle and a means to develop and assess crew performance across a range of required defined competencies.
There are many elements that are crucial for the right EBT development. With EBT being based on ‘evidence’, one of these preconditions is a thorough data analysis capability within an operator, in order to evaluate all the available data. Such analysis must be conducted by a knowledgeable team of instructors and must rely on the data from multiple training sessions.
In EBT programs – a ‘refocusing’ of the instructor is necessary. That includes putting focus on the root causes analysis of unsuccessfully flown maneuvers rather than simply asking the pilot to repeat a maneuver with no real understanding as to why it was not successfully flown in the first place. Within such an EBT session - the instructor needs to analyze where and why the maneuver was unsuccessful and come up with appropriate mitigating measures. 
E.g. an unstable approach is quite often caused by a bad descent planning in the first place, therefore a reposition on 6 miles final will not address the root cause of the problem. However, that analysis is not related to data as such, but is rather an observation.

[1] Synthetic Flight Instructor

[2] Amendment 5 to be published

[3] As specified in ARO.OPS.226 (c) (2) (iv)

[4] Page 207 of the NPA 2018-07, referring to Annex 1 (Part FCL) to Regulation (EU) No 1178/2011

[5] AMC2 ARA.FCL.205 and GM to AMC2 ARA.FCL205(b)

[6] as per ORO.FCL.231

[7] TRE = Type Rating Examiner / TRI = Type Rating Instructor

[8] NPA page 16

[9] ICAO Doc 9555

[10] As stated in ICAO Doc 9555 3.4.1.

It is necessary to collect real world data from accidents, incidents, flight operations and training to feed and validate course development. Data collection as described in this manual has been used to construct the baseline EBT programme and will be reviewed and updated on a continual basis. The enhanced EBT programme described in Chapter 5 of this Part is intended to create an improvement to the baseline programme, utilizing operator-specific data.

Evidence Based Training, Position Paper ECA 2019