Testing of digital operational resilience

Article 24 of the Regulation sets out the general requirements for the performance of digital operational resilience testing. For the purpose of assessing the resilience of ICT systems and services, firms need to establish, maintain and regularly review a testing programme. This testing programme has to be part of the ICT risk management framework and has to include the tests, practices, methodologies and tools regularly applied to assess the organisation’s ICT systems, tools and processes.

The assessment considers the processes designed to timely detect and solve any ICT-related incidents. Additionally, firms have to perform their own assessment of their ability to detect any vulnerabilities and deficiencies in their digital resilience. Finally, the tests need to provide insight into the extent to which organisations can timely implement remediation measures that minimise the length and impact of a disruption. When conducting testing programmes, firms should consider the evolving landscape of ICT risk, any specific risks to which the financial firms are or might be exposed, and the criticality of ICT systems and services provided.

Organisations are expected to conduct, at least yearly, tests on their ICT systems and applications supporting critical or important functions. These tests can be undertaken by both internal and external testers. It is important to ensure that any potential or actual conflicts of interest are avoided throughout the design and execution phases of the test. Where tests are undertaken by internal testers, financial firms need to take appropriate measures to ensure that the testers have no vested interest in the results of the tests. Financial firms will furthermore need to establish procedures and policies to prioritise, classify and remedy all issues revealed throughout the performance of the tests, and ascertain that all identified vulnerabilities and deficiencies are fully addressed.

Article 25 of the Regulation describes the tests that firms can conduct to test their ICT systems and applications. Firms have to determine which tests are relevant for this purpose in accordance with the proportionality principle. Examples of tests include vulnerability scans, gap analyses, end-to-end testing and penetration testing.

Abovementioned requirements do not apply to microenterprises. Microenterprises are expected to combine a risk-based approach, by considering the need to maintain a balanced approach between the scale of resources and the time to be allocated to the ICT testing, on the one hand, and the urgency, type of risk, criticality of the ICT system as well as any other relevant factors, on the other hand.

In addition to the general requirements for testing ICT systems and tools, a number of firms are also subject to additional requirements with regard to testing the digital operational resilience. These firms are required to conduct advanced testing by means of a threat-led penetration test (TLPT) once every three years.

TLPT involves extensive testing that mimics tactics, techniques and procedures that are exploited in real life by threat actors such as hackers . It checks the cyber resilience of financial firms in a controlled, firm-specific and intelligence-led manner. Such a test is a more comprehensive type of red teaming, partly owing to the involvement of the supervisory authority.

For the performance of TLPT to be successful, it is important that firms consider this type of testing as an opportunity to learn and to expose any vulnerabilities. In addition, TLPT is resource intensive, which means that it is important that firms dedicate sufficient resources and staff throughout the different phases of the test.

Each TLPT has to cover the critical or important functions of a financial firm, and it has to be performed on live production systems supporting such business functions. For this, firms first need to identify the ICT systems, processes, tools, and services, including those which have been outsourced, that support the critical or important functions. Financial firms then assess which critical or important functions need to be covered by the test.

Before the precise scope of the test may be made final, it will need to be validated by the TLPT authority supervising the test. In the Netherlands, the validation will either be done by the AFM or DNB, depending on which supervisory authority issues the licence.

Where ICT third-party service providers are included in the scope of a TLPT, firms must take appropriate measures to ensure the participation of such ICT third-party service providers in the TLPT. If the participation in the TLPT will have an adverse impact on the quality of services delivered by the ICT third-party service provider to organisations falling outside the scope of DORA, the external service provider may be excluded from the scope of the TLPT of the firm. In that case, the ICT third-party service provider designates an external tester for the purpose of conducting a pooled TLPT involving several financial firms to which the ICT third-party service provider provides ICT services. This pooled testing must cover all ICT services supporting critical or important functions contracted to the third-party service provider by the different financial entities.

A number of requirements apply to the testers (or red team) to ensure that the TLPT is carried out correctly. Firms must make use of testers that are certified by an accreditation body in a Member State and/or adhere to formal codes of conduct or ethical frameworks. Testers also need to possess adequate technical and organisational capabilities and demonstrate specific expertise in threat intelligence, penetration testing and red teaming. Finally, testers must be able to ensure the independent performance of the test and the confidentiality of information, such as of test results.

When using internal testers, firms must have sufficient dedicated resources to avoid any conflicts of interest and the use of internal testers has to be approved by the TLPT authority supervising the test. The requirements for the use of internal testers are further elaborated in Chapter IV of the RTS.

For the designation of firms that are required to perform TLPT, the TLPT authority decides which financial entities will be designated to perform TLPT. Depending on the regulator granting the license, either the AFM or DNB will be responsible for the designation (and oversight) of TLPT.

Some financial institutions may be designated by both the AFM and DNB. In such cases, it may be decided that a joint test will be conducted, involving both regulators. The criteria for identifying firms for TLPT are described in Chapter II of the RTS. The TLPT authority takes proportionality into account. Firms can be identified for TLPT based on ‘hard or quantitative criteria’. This for example includes trading venues having a certain market share at national or European level. Article 2(1) of the RTS describes these hard criteria.

Any firms not identified based on the quantitative criteria may still be required to perform TLPT, taking into account their ICT risk profile, systemic character and impact on the stability of the financial sector. More specifically, firms can be identified based on the following (mostly qualitative) factors:

• Systemic character and impact-related factors:
  o The size of the firm;
  o The extent and nature of the interconnectedness of the firm with other financial entities in the financial sector;
  o The importance of the services provided;
  o The substitutability of the services provided;
  o The complexity of the business model of the financial firm;
  o Whether the firm is part of a group using common ICT systems.
• ICT risk-related factors:
  o The risk profile of the firm;
  o The degree of dependence of critical or important functions or their supporting business functions on ICT systems and processes;
  o The complexity of the ICT architecture of the firm;
  o Outcomes of any supervisory reviews relevant for the assessment of the ICT maturity of the financial entity;
  o The maturity of ICT business continuity plans and ICT response and recovery plans;
  o The maturity of the operational ICT security detection and mitigation measures;
  o Whether the firm is part of a group using common ICT systems.

Microenterprises and firms for which the simplified ICT risk management framework applies, are not identified for TLPT.

Chapter III of the RTS outlines a number of requirements related to the test methodology and the TLPT process to ensure that each TLPT is performed correctly. To begin with, firms have to assess the risks associated with the performance of TLPT. Appropriate measures must be taken to prevent these risks from materialising as a result of performing the testing activities. These involve risks relating to:

• granting external parties access to sensitive data;
• failing to meet the requirements related to TLPT;
• crisis/incident management;
• disruptions in critical activities and processes;
• loss of data due to testing activities;
• failing to fully recover any systems affected by the test.

Before any of the testing activities can be carried out, it needs to be clear to everyone involved in the performance of TLPT what their roles are. The authority supervising the TLPT, i.e. the AFM or DNB, must assign a test manager who is to coordinate the testing activities and who has to ensure that all requirements are being met throughout the performance of the test. Additionally, at least one alternate test manager has to be designated who can take over the test manager’s tasks if needed. The authority supervising the TLPT is expected to participate in all phases of the testing activities and to provide feedback, validations or approvals where needed.

Firms are responsible for having a team of employees involved in the day-to-day management of the test (the control team or white team), with one person designated as team lead. The control team is kept informed of all detections resulting from the TLPT. This relates to any detection resulting from the TLPT by both staff members of the organisation itself or of its third-party service providers. Any follow-up actions for incidents resulting from the test should be taken up by the team itself, and information on the progress of the testing activities should be shared with test managers when requested.

Finally, appropriate measures should be taken to ensure the confidentiality of the TLPT. Thus, the access to information about the TLPT needs to be limited to the control team, the managing body of the firm, the TLPT authority, the providers of threat intelligence and the testers. The threat intelligence providers are external specialists who collect data and analyse actual threats upon which basis they develop realistic scenarios.

The testers consist of external or internal ethical hackers (or red team) attempting to gain access to a firm’s production systems. The blue team consists of staff members of the firm trying to protect the network and ICT systems against external attack actions. The blue team will not be involved in the test and is thus not informed about the test.

Preparation Phase

As soon as a firm receives a notification from the TLPT authority, it can initiate the preparation phase of the test. During the preparation phase, firms conduct the risk assessment, which examines the risks involved in conducting a test on the production environment of systems that support important and critical business functions. Prior to the start of the test, the TLPT authority also must receive a project charter, the contact details of the control team lead and information on the use of internal or external testers.

Firms must furthermore share information with the TLPT authority on the communication channels to be used during the implementation of the testing activities and the code name for the test. This information must be shared with the test managers within three months from having received a notification from the TLPT authority. Firms identified by the AFM for TLPT can submit the required reports via the AFM portal . As soon as this documentation has been approved by the test managers, the firm can set up the control team that is to support the team lead in the preparation of the test.

Once the TLPT authority validates the composition of the control team, firms need to determine which critical or important functions are to be included in the scope of the testing activities. In doing so, firms need to consider the importance of the function for the firm and the stability of the financial sector, the exchangeability of the function, the interconnectedness with other functions and the geographical location of the function, among other things. Once the scope of the test has been determined, it has to be approved by the managing body and submitted to the test manager . This report must be submitted no later than six months from having received a notification from the TLPT authority. Once the reports submitted have been approved by the test managers, these can be shared with the testers and the threat intelligence providers. Firms must ensure that both the red team and the threat intelligence providers are contracted before the start of the testing phase.

Testing phase

The testing phase can be subdivided into two components: threat intelligence and red teaming. Following approval of the test by the TLPT authority, the threat intelligence providers have to analyse the firm and identify any threats and vulnerabilities concerning the firm. During this phase, they gather data by identifying cyber threats and existing or potential vulnerabilities that may be exploited during the test. For this step, the threat intelligence providers may consult the control team and the test managers supporting this test. Based on this analysis, the threat intelligence providers will propose a number of scenarios that can be used when conducting the test. The control team lead then selects at least three scenarios based on the threat intelligence providers’ recommendation, the input of the test managers, the feasibility of the proposed scenarios during the performance and the size, complexity and risk profile of the firm. The threat intelligence providers’ analysis has to be included in a report and submitted to the test managers .

Following approval of the threat intelligence report by the test managers, the testers, i.e. the red team, can prepare their test plan . The test plan includes the tactics, techniques, procedures, and communication channels to be used when conducting the test. Once the test plan has been drawn up, it needs to be discussed with the control team, the test managers and the threat intelligence providers before the control team and the test managers can approve the plan. Following this approval, the testers can start their testing activities.

The duration of these testing activities will be proportionate to the scope, scale, and activities, amongst others. However, the test must take at least twelve weeks. During the performance of the test plan, the testers, test managers and the control team meet on a weekly basis to discuss the progress. In case of detection of the testing activities by any staff member of the firm, the control team, in consultation with the testers, will need to take appropriate measures to ensure the secrecy of the test. These measures should then also be shared with the test managers. In case the testers reach the point that continuing the test may lead to serious disruptions in critical or important business functions, the control team lead may decide to suspend the test.

Closure phase

Following the completion of all testing activities, the staff members of the blue team need to be informed about the TLPT that has taken place. Within four weeks from the end of the test, the red team has to submit to the control team a report with information on the test and the results . Subsequently, this report has to be shared with the blue team and the test managers.

No later than ten weeks after the end of the testers’ testing activities, the blue team has to submit to the control team a report containing a list of attack actions detected during the test (including log files) . This file should then also be submitted to the test managers. In that same period, the testers and the blue team conduct a smaller test by carrying out a replay of the attack actions on the ICT systems and infrastructure, also referred to as purple teaming. They can then also conduct additional tests that could not be tested during the TLPT. After completion of this smaller test, all parties involved are given the opportunity to provide feedback to each other on the TLPT process.

Once the TLPT authority has assessed and approved the blue team test report and the red team test report, firms will have eight weeks to submit the report summarising the findings and the corrective plans to the TLPT authority . Within that same period, firms also have to provide a remediation plan describing the identified shortcomings, the proposed remediation measures and the prioritisation thereof, a root cause analysis, the staff or functions responsible for the implementation of the proposed remediation measures and the risks associated with not implementing the measures. Finally, after the receipt of the required documents, the TLPT authority provides the firm an attestation confirming that the TLPT was performed in accordance with the requirements.