Incident Response: How to Form A Response Team

Every incident is different — and the response team that is formed as a result will need to be different as well. Companies have different needs depending on things like regulatory requirements and the sector that they operate in. This means that no two incident response teams will be the same. For example, there will be incident response teams with two people and others with a dozen or more.

All effective response teams, though, will identify, stop, and successfully resolve incidents. The response team should also research, educate, and communicate security response methodology. An entire company's attitude around safety and security should be in line with the policies and procedures created by the security teams that form incident response plans.

Setting up a response team and understanding each member's responsibilities is crucial to successfully — and quickly resolving security issues. Let's discuss some best practices when forming an incident response team.

What is a Computer Security Incident Response Team?

A CSIRT can be defined in a broad sense as a team of security professionals who work as a team to prevent cyberattacks that target a company. The main reason that CSIRTs have become an integral part of modern businesses is a direct consequence of business transactions and communications migrating to online and digital mediums.

There is no set structure to a CSIRT, and each organization will have their own structures when considering roles and team members. A popular structure looks something like this:

• Managerial Role: This role is responsible for coordinating the CSIRT and relaying information to the rest of the executive suite. They understand the technical requirements of the business, and can acquire the talent, tools and training that they need to successfully run their CSIRT. They also create and update the Incident Response Plan according to changes in the business and cyber attacks that are currently occurring. Other functions include documentation, procedure creation, and even keeping track of meeting minutes and major decisions regarding the cybersecurity stance of the organization. In short, it is a stressful position with a lot of responsibility. It requires a mix of technical understanding and managerial skills.

• Technical Manager/Technical Lead: This is a hands-on role that requires continuous learning, practice and solid hands on experience. The person in this role is the technical pillar of the team and is usually a multidisciplinary champion of all things related to security. The technical lead must also have an exceptional understanding of how the environment works, and where the potential weaknesses in the system lie. This role could be filled by one person, or by many, depending on the size and budget of the organization. This role will generally report back to the CSIRT Manager, who then relates the technical ramifications of an incident back to the management team in the form of an executive summary.

• CSIRT Member: These are the frontline workers of the team. They are responsible for monitoring potential incidents and escalating them whenever a breach or security issue arises. They usually have more contact with end users and customers which means that they also are responsible for enforcing good cybersecurity practices. They can make solid recommendations about new threats and potential security issues too.

When we look at these general roles, we start to understand how the team should operate when dealing with incidents. Within these roles there are additional responsibilities that they may undertake and may even involve other departments. Examples of this are when there is a legal requirement that the legal team or appointed lawyer needs to act on. There may also be a 'floating' role that falls on the person working when an incident occurs, like Crisis Manager, Incident Commander, or Subject Matter Expert.

CSIRTs are responsible for detecting and controlling a cyber incident, and then understanding the scope of potential damage. After an incident has been contained and resolved, then the CSIRTs need to gather information and create a postmortem that will outline the causes of the issue, and a solution to prevent it from happening again.

Other key responsibilities of a CSIRT Include:

• Security incident detection and remediation

• Responding to incidents as soon as they are detected

• Generate and have access to a full catalogue of past security incidents, including detection and remediation information. This data must be highly detailed and provide the teams with a comprehensive data stack.

• Engage and undertake specialized training to keep up to date with the latest security developments.

• Security audit management

• Periodic reviews of security measures, including pen tests and various security scenarios

• The creation of a comprehensive incident response plan, as well as updates and amendments based on incidents and the latest cybersecurity developments

• Maintain the CIA triangle of Confidentiality, Integrity, and Availability

It is important to note that, like most IT based teams, CSIRTs have the potential to be understaffed. The end result is that members of these teams often have to perform multiple roles, meaning that the skills required for the role are often quite comprehensive.

Considerations When Setting Up a CSIRT

The team that you create needs to have the tools and authority to act when it is needed. The last thing that you need in your team is to have their hands tied while they wait for management to understand the scope of the threat, and the potential damage to business if the attack continues. The team needs to be empowered to take action and allowed to fail when innovating. The team needs to encourage a cooperative, collaborative space that is not afraid of failure.

The team needs to have many different layers of knowledge, and it is not always possible for a company to justify a permanent position for a niche or highly specialized skill set. For this reason, we normally find a blend or mix of the following tiers of knowledge in a CSIRT:

• Employee-sourced: Full-time staff that offer skills that are needed often.

• Partially Outsourced: A company that has a pool of professionals who deal with specific incidents quite often or have additional staff to assist the organization during an active incident. They bring skills and knowledge to the company and can work within the existing frameworks set out by the CSIRT.

• Fully Outsourced: This is normally reserved for highly specialized consultants who respond to incidents as a technical lead, or a subject matter expert. This can be a single person, or a team of people that come in to perform a specific role either in the planning phase of an incident response plan, an active incident, or a postmortem analysis of a resolved incident.

• Software Automation: Software automation is a great way of mitigating threats that come from other automated processes like bots and malware. These automated solutions keep detailed logs about the mitigations that they carry out, giving the team an excellent source of information for mapping trends of the current load on the automated systems. Automated systems can also perform failovers and play books when certain criteria are met leading up to a security incident.

Another thing to think about is the types of incidents your company might reasonably expect to experience. These are many different attack vectors to consider, too many to list in fact. But some of the main ones that you can expect to come across are:

• Email: Yes, even in 2020 there are still plenty of malicious emails that deliver malware through infected attachments, embedded malware in other files, or phishing scams. User awareness training is essential for preventing these kinds of attacks from being successful.

• Removable Storage: Items such as USB drives, CD-ROMS and even a smart phone connected via a USB cable have the potential to transmit malware. An infected device can automatically propagate across devices and cause severe damage to a computer network. Again, user training and a company User Acceptance Agreement are vital for controlling the security of your network.

• Insider Attack: It might not seem like a common occurrence, but users within the organization have been known to either willingly or unknowingly cause damage to a company network from within the organization. This can happen when malware like a crypto variant of malware is released, causing havoc with files and network shares. Other more extreme cases can involve disgruntled employees intentionally destroying virtual machines and virtual storage arrays. These kinds of severe actions have legal ramifications and a CSIRT will have to involve law enforcement in most cases.

• Data Leakage: There are many proprietary data Loss prevention systems that scan network traffic for known intellectual property, and flag them when detected. Even copying certain file types to a removable storage device can trigger a DLP response, which will normally involve the CSIRT once a DLP event occurs.

• Web-Based Attacks: These can cover things like a DDoS attack, unauthorized access to internet facing systems, user account hijacking, or any other cyber criminal activity that is launched over the internet. These attacks can sometimes be a distraction from another activity, like securing backdoor access to a corporate network. These can remain dormant for some time, or they can be active without detection for a very long time, much like the Marriott breaches that were disclosed in recent years.

• Theft and Loss of Equipment: This is not as big of an issue if all of your company laptops and computers are equipped with drive encryption. Laptops that use encryption require a password before they boot up. If a system is taken without these kinds of protections, then there is a greater chance of your company network being accessed with that equipment if passwords are extracted or cracked.

What else does the CSIRT need to handle?

A key competency that you need to be able to carry out is to manage stakeholder expectations. When an incident is ongoing and there are investigations underway you need to be able to tell the executive and management teams exactly what is happening while still working on the problem itself.

There needs to be realistic timeframes given, and the management teams need to give the CSIRT room to operate without being interrupted and disturbed too often to give details about the ongoing incident.

In order to properly carry out the roles within the CSIRT you have to understand what the company's critical components are, how they tie in with one another, and what business impact will be if there is a failure on a specific system.

This leads into the next important metrics that need to be measured: success and failure. How do you define what constitutes a win or a loss after an incident has been resolved? The truth is that it depends on who you ask. The business owners and top executives will ultimately decide on how the handling of an incident affected business operations and business turnover.

However, there are important pieces of information that come out of the postmortem, both positive and negative. How long did it take for the team to raise the alarm when the incident first appeared? How long did it take for the CSIRT to spring into action? What systems were affected? What reputational damage, if any, will the business suffer? What did we do right? What did we do wrong?

All of these questions need to be answered before you can assert how success and failure is measured. There is a process for each and every CSIRT but finding out all of those key pieces of information is a good start.

Legal Considerations

Most IT professionals don't have much legal grounding because it is not traditionally something that needs to be dealt with on a daily basis. As countries become more interconnected with one another, so too do the laws of these jurisdictions. This is especially true of regions like the EU, which has pushed through with the GDPR (General Data Protection Regulation). This is a set of laws that mandate how long a company can keep user information, what it can keep, and much more.

If a company is caught in breach of this regulation, then they will suffer hefty fines and other negative consequences too. A normal CSIRT member understands quite a lot about regulatory and legal requirements for specific markets, but when it comes to the technical legal details then you are better off consulting with the companies legal or compliance teams. CSIRT members must understand which regulations and laws apply to their industries and take appropriate steps to make sure that they do not expose the company to unnecessary risk from a legal perspective.

Preparing your CSIRT for Standby and Live Issues

Like any emergency response team, the CSIRT needs a selection of tools, procedures, contacts and schedules to ensure success. This "go-bag" must be available to everyone on the team so that no matter who gets the call, and no matter what time it is, that they have all of the things that they need to get a running start on the current incident.

The items that need to be available for each member include

• Response Plans: This is a list of different playbooks, automated responses and plans.

• On-call Schedules & Contact Lists: Each incident will focus on a different core competency, so knowing who to call for which issue is of the utmost importance for a faster response. These contacts need to be aware that they are on standby so that they are contactable when the worst comes to worst.

• Escalation steps: The top-level managers don't want to be contacted out tof the blue in the middle of the night for a false alarm. This means that when the top members get the call things need to be absolutely certain of an incident. To this end there are specific escalation plans that need to be fleshed out when dealing with incidents. Another reason to have an escalation policy is that when a standby engineer is not available and is not answering their phone, the escalation policy points out who to call next until the response is adequate for the issue.

• Links/instructions to communication tools: Everyone needs to know which communications platforms and applications are to be used for talking with one another during an incident. If your teams operate in different regions around the world, then you need to understand the data retention policies for the apps that run in these locations. More simply, people need to know where to send messages, where to make voice calls, and which ones to use for which teams when it is needed. Clear documentation must be made available to everyone that is on the CSIRT, the subject matter exports, the managers and executives.

• Access codes: If any of the resources that your team needs are locked down with passwords and access codes then they must have secure access to them so that they can use whatever they need to get the job done.

• Other tools or protocols: Anything that is specific to your particular environment must be available to everyone that needs to respond to an active incident.

The key aspect of acting as a team is transparency. Transparency protocols that are in place throughout the entire process must be made clear to both CSIRT members and managers. This ensures that there are no miscommunications and mistakes. Members of your team need to know why they are doing what they are doing almost as much as how to do what they are doing. Transparency is key in a well-functioning team.

High-Level CSIRT Roles

In a perfect world there would be a hired individual for each required role within the CSIRT. This is not the case for most companies, however. Mainly because cybersecurity incidents vary in frequency depending on the complexity, veracity and business impact of each one. As a result, companies do not need to keep full time members on their payroll if the skills that they offer, however valuable, are only needed every six months to a year. This means that most members on your team will hold dual roles and have multiple responsibilities that may even fall outside of the original job description of their role in the team.

CSIRT Manager

Take for example a CSIRT manager. They will generally have great managerial skills, pay attention to details, and follow processes and procedures that are in line with both the company's needs, and IT governance requirements. CSIRT managers wear many hats. The position is often given to team members that have worked their way up from a technical background.

There is a mix between people skills and real world, hands on knowledge. This is important because as the manager of the team you will need to guide your team in processes of the job that they will be performing, as well as guiding them through the company culture and making sure that they fit in with the rest of the department and organization.

Something that a CSIRT manager is not responsible for is being the main technical contact when there are issues. CSIRT managers are responsible for more general concepts such as the mission of the cybersecurity team. For a technical reference for the rest of the team there are other roles, such as the lead investigator, or team lead.

Lead Investigator(s)/Team Leads

A lead investigator needs to have a very long list of technical capabilities as well as managerial and leadership skills. Lead investigators are responsible for guiding the CSIRT when dealing with live incidents, and for generating and updating procedures for future ones. Skills requirements include cyber analysis, automation testing, scripting and solid knowledge of Security Operations Center policies and procedures. Other skills include threat hunting, some digital forensics skills, monitoring and detection skills as well as data loss Prevention knowledge.

The lead investigator is responsible for the team's performance during an incident, as well as the systems that are in place to monitor, detect and mitigate cyberthreats. The role requires a high stress tolerance as well as flexible working hours because incidents do not keep normal office hours. If the standby CSIRT teams are having trouble trying to control an incident, then the team lead will need to come online and assist where they can.

The position is a blend between the CSIRT manager and the technical lead, with more focus on the analytic and technical side of the team than with the overall management of it. As we mentioned earlier, it is not unusual to find that the team lead is also the manager of the CSIRT or the whole security department, but for larger organizations that can afford it there is a definite plus to having these roles divided between multiple team members.

The team lead/ lead investigator is not responsible for staff issues that relate to HR and payroll, and instead are there as the technical go to person. This means that they need to be up to speed with all of the latest developments in both cybersecurity mitigation techniques and cyberattack methods as they become available.

Incident Responders

Incident responders go by many different names such as CSIRT Engineer, a security or intrusion analyst. Whichever title they hold, they are generally cybersecurity professionals that are still in the process of studying and working towards a more specialized and technical role. Incident responders are the backbone of the CSIRT, and their roles and responsibilities vary greatly from company to company. An incident responder is responsible for shutting down an attack as soon as possible. This means working under great pressure and at odd hours in many cases. As an incident responder you are the backbone of the CSIRT, and this comes with its own set of challenges.

Incident responders understand the importance of escalation policies, and when an issue comes around that needs additional eyes, they can get the right people looking at the issue right away. They are responsible for getting the initial triage done on an incident before it becomes a much bigger problem.

Cross-Functional CSIRT Roles

Not all teams operate from within a company's Information Technology sphere, but they still offer guidance and direction when there are serious security incidents that are ongoing. Each of these different teams offer their own areas of expertise that they can bring to the table, even if they don't necessarily fix the issue. They all have their own mandates that need to be satisfied in order for the business to function correctly, and the CSIRT helps to facilitate these objectives.

Executive Management

Executive management is the highest tier of management within an organization. As such, what they have to say is normally the final word on the matter, regardless of the technical ramifications. They lead the technological drive of the company, and they control how that mission is delivered through each of the various departments.

Executive Management is responsible for all mission critical aspects of the business, so it is no surprise that the CSIRT needs to have a direct line of communication with them. If the gears of the business are not turning as a result of a security incident, then the CSIRT needs to be on speed dial for the executive management teams.

The executive management teams are not responsible for any technical aspects of the business, but they are results driven. They need to keep their shareholders happy, so they are under immense pressure all the time. They have to bear the brunt of the pressure from stakeholders and partners, so that pressure will inevitably find its way to the CSIRT if expectations are not managed correctly.

Audit and Risk Management (IT & Facilities)

The audit and risk management teams are responsible for many different aspects relating to regulatory and risk aspects of the business. There are several key differences between these teams as they have different objectives. Risk assessment is geared towards preparing for the worst-case scenarios of each potential issue that could arise from a security breach.

Audit teams take a more retrospective approach to incidents and compile reports based on past issues so that the impacts of those occurrences can be better understood. In both cases the skills that are required for these roles involves attention to detail. You can think of auditing as an assurance role that includes elements of consultation and risk management as well.

They are generally not responsible for resolving an active security incident. However, they are normally available during an incident so that they can provide the CSIRT with valuable information as it relates to procedural and risk matters.

Legal & HR

The legal and HR teams are complicated departments on their own. The legal team needs to understand the legal exposure that the company is subject to whenever a cybersecurity incidentis detected. They need to liaise with the CSIRT managers and have a full understanding of what the incident means for the legal status of the company. One example is when data breaches release client data into the public domain. The legal team needs to understand how much liability the company will need to absorb, and then make recommendations in order to minimize any damage that the company will take.

The HR department needs to make sure that the company is protected from any user information that might have been released during a breach. This protects the company from any actions that might be taken against them from the staff, and it ultimately protects the company from any labor disputes that may arise.

Both Legal and HR need to work together to ensure that they can keep the impact of a cybersecurity breach from impacting too negatively on the organization. The CSIRT then needs to keep stakeholders from both departments updates so that they know what the implications of ongoing incidents might have on their departments.

Trainers

Trainers are one of the main barriers between cyber threats and the company's users. Proper training helps to prevent common attack vectors from landing successfully in a company. Basic user awareness training should form a big part of a company's user induction and continued learning strategies. Trainers need to possess great people skills and also be aware of all the latest threats that target users. The CSIRT can help to facilitate this by passing on relevant information about some of the latest threats that could possibly impact the company.

The trainer's level of skills can range from basic user training, where basic threat avoidance training is taught to users, all the way up to training developers and security teams. The trainers are not responsible for actively mitigating threats.

Trainers are not responsible for deciding the kind of training, or the direction in which the training must go. The trainer's job is to get direction from the management teams, who in turn are informed by the technical teams in the CSIRT, Information Security team, and the IT department heads. All of this is done in consultation with the executive level members who help to forge the company's strategy not just in Information Security, but in all aspects of operations.

• What skills are needed?

• What are they responsible for?

• What are they not responsible for?

Final Thoughts

Setting up an effective CSIRT allows a company to quickly respond to security incidents such as a breach, DDoS attack, or malware outbreak. Some important considerations include good communication, transparency, clear policies, and training opportunities.

A good CSIRT will help protect the confidentiality, integrity, and availability of a company's information systems while maintaining effective communications with both the executive and management teams as well as regular users within the organization.

Your CSIRT should also have other skills across a wide variety of domains such as forensics, software development, networking, and much more. Being able to conduct thorough postmortems and understanding the implications of an incident is just as important as mitigating a threat. This helps to bolster the incident response of your team the next time something similar occurs, and can reduce incident times and impacts to the business.

There is a lot to consider when forming your own CSIRT, and we hope that this serves as a good starting point for the planning phase of your own endeavors.