SecGW for Mobile Networks Deployment Guide

Deployment Guide

SecGW for Mobile Networks

TableofContents

Change Log 5

Introduction 6

About this Guide 6

Intended audience 6

IPsec 7

IKEv2 7

IKEv2 re-authentication for Phase1 8

Extensible Authentication Protocol (EAP) 8

Global IKE attributes 9

Tighten security for configuration payload 9

Dial-up IPsec tunnels for RAN connectivity 9

Site-to-site IPsec tunnels: backhauling edge locations 9

Single-tunnel throughput considerations 10

Traffic tunnel design 10

IPsec termination local gateway: loopback interfaces 11

Loopback interface hardware acceleration 11

IPsec ciphering considerations 11

Crypto sets 12

GCM crypto sets 12

Diffie-Hellman groups 12

Rekeying and key lifetime 13

Perfect Forward Secrecy 14

Anti-replay and packet reordering 14

Extended sequence numbers (ESN) 14

IPsec and packet fragmentation 15

Certificates-based authentication 16

CMPv2 16

Accessing the CMP server 17

Deployment Guide 2

High availability 21

Intra-site redundancy 21

Intra-site redundancy with FGCP 21

Intra-site redundancy with FGSP 21

Inter-site redundancy 22

SecGW scale-out with FGSP 22

HA flexibility 23

Design considerations 23

Inter- and intra-site redundancy with FGSP 24

Design concept 24

Standalone node description 24

Physical connectivity 25

Logical connectivity 26

Routing loopback advertisement (and path preference) 27

Routing SecGW inner address advertisement 29

Other routing configuration 30

Multi-node FGSP cluster solution 32

Per-tunnel failover 32

Synchronization 32

Conditional advertisement 34

Putting it all together 35

Anycast 35

Dead peer detection 37

How many nodes? 37

Noteworthy FGSP considerations 38

Mixed firmware and/or hardware 38

Many-to-one disaster recovery option 38

FGSP race condition 39

Hardware switch to reduce session dirtying 39

IKE monitoring with FGSP in split-brain situations 40

Intra-site redundancy with FGCP 41

NSR or BFD? 41

PE to PE link 42

Intra-site FGCP summary 42

Inter-site redundancy with FGCP 43

Cluster connectivity using MNO network 43

Cluster Connectivity with VXLAN 44

Interface considerations 45

Logical connectivity for HA 46

FGCP with FGSP 48

Security functions 50

SCTP firewall 50

Firewall flow policy 51

S1-AP/NGAP message flooding protection 52

S1-AP 53

NGAP 54

Deployment Guide 3

DoS protection 54

L3 anomalies 54

L4 anomalies 55

VDOM recommendations 56

VDOM resources 56

VRF recommendations 57

Traffic prioritization and QoS 59

QoS and anti-replay 59

DSCP copy-down and inbound-dscp-copy for IPsec 59

Management, orchestration, and API 61

FortiManager 61

FortiManager JSON API 61

Fortinet Developer Network (FNDN) 62

Ansible 62

Terraform 62

FortiOS REST API 62

Admin profile, admin user, and token APIs 63

VPN configuration APIs 64

Log VPN APIs 65

Monitor VPN APIs 65

Logging 66

Fault and performance monitoring (FM/PM) 67

IKE and IPsec monitoring 68

NP7 IPsec troubleshooting commands 69

Conclusion 70

Appendices 71

IPv6 and SecGW configuration examples 71

VLAN interface 71

Loopback interface 72

BGP and referenced configuration 72

IPsec VPN 74

Firewall policy 75

FGSP synchronization 75

GTP-U payload inspection 75

Security policy 76

Logging 77

Performance considerations 77

S1-AP/NGAP message flooding configuration and testing 78

Custom IPS signatures 78

Intrusion Prevention profile 81

Firewall policy 82

Logs 83

Tuning thresholds 84

S1-AP S1 setup request message flooding 84

Interoperability with strongSwan 85

Deployment Guide 4

Change Log

Date Change Description

2023-09-25 Initial release

Deployment Guide 5

Introduction

The SecGW (security gateway) is a security component in a 3GPP wireless network, such as 4G LTE or 5G:

fundamentally, it provides a security layer between access (Radio Access Network or RAN) and core.

See the SecGW for Mobile Networks Architecture guide for more information about SecGW.

About this Guide

The deployment guide helps the reader understand how to deploy the SecGW solution in a Mobile Network. It comes

with advice on how to configure, operate, and troubleshoot the SecGW solution.

It should be noted that this guide is specific only to the SecGW use case, and does not broaden or elaborate on the many

other functions and use cases the platform can offer.

This guide is based on the latest maturity release of FortiOS that can be recommended for SecGW deployments.

Currently this is FortiOS 7.0.xM (Mature). Earlier or more recent versions of FortiOS might require different design and

configuration considerations.

FortiOS 7.0.8 introduced several changes and features relevant to SecGW and features, which

are used in this guide.

Intended audience

This guide is intended for any audience who is interested in learning how to deploy Fortinet’s SecGW solution.

Interested audiences may include Network and Security Architects, Network and Security Engineers in organizations

such as Mobile Network Operators, Professional Services, Fortinet partners, and System Integrators to name some

examples.

Readers should have a basic understanding of networking, cellular wireless, and security concepts before they begin.

Deployment Guide 6

IPsec

The following sections will discuss IPsec in relation to SecGW. This document does not serve to educate on the basics of

IPsec; it assumes that the reader has general knowledge and awareness of IPsec technologies. Further information on

FortiGate IPsec functionality can be found in the Fortinet Document Library.

IKEv2 on page 7

IKEv2 re-authentication for Phase1 on page 8

Extensible Authentication Protocol (EAP) on page 8

Global IKE attributes on page 9

Tighten security for configuration payload on page 9

Dial-up IPsec tunnels for RAN connectivity on page 9

Site-to-site IPsec tunnels: backhauling edge locations on page 9

Single-tunnel throughput considerations on page 10

Traffic tunnel design on page 10

IPsec termination local gateway: loopback interfaces on page 11

Loopback interface hardware acceleration on page 11

IPsec ciphering considerations on page 11

IPsec and packet fragmentation on page 15

Certificates-based authentication on page 16

Links:

IPsec VPNs

IKEv2

Throughout this document, all IPsec references are based on the use of IKEv2. While IKEv1 can be used, it is not

recommended primarily because the 3GPP recommendations for SecGW are to use IKEv2. Add to this the fact that IKEv1

is generally perceived as legacy, with IKEv2 providing many benefits over IKEv1. For these reasons IKEv2 should be the

only considered choice for SecGW.

FortiGate platforms support both IKEv1 and IKEv2 to ensure full coverage for legacy systems

where IKEv2 support is not available.

Deployment Guide 7

IKEV2 RE-AUTHENTICATION FOR PHASE1

IKEv2 re-authentication for Phase1

IKEv2 by default does not re-authenticate the peer after the expiry of the Phase1 keys. Rekeying will still occur, but the

peer is not re-authenticated as part of this process. This lack of re-authentication can be seen as reduced security by

some MNOs. For this reason, FortiOS provides the ability to enable re-authentication as part of the rekeying.

The command used for this is as follows. By default reauth is disabled:

config vdom

edit SecGW

config vpn ipsec phase1-interface

edit "IPsecGW4"

set reauth enable

end

The use of re-authentication will be dictated in part by the configuration of the peer device, as the initiator of the

connection. Also, be mindful that there will be a small performance and latency overhead associated with re-

authenticating for every Phase1 rekey.

This document makes no recommendation as to whether re-authentication should be enabled. This setting will be

dictated by the MNO requirements.

This setting may be needed to ensure that the SecGW can respect certificate revocation

(CRL) by forcing the eNodeB to reauthenticate rather than just re-key.

Extensible Authentication Protocol (EAP)

The FortiGate platform supports EAP in association with IKEv2. The FortiGate is able to broker EAP messages into

RADIUS messages to authenticate against a remote AAA service.

config vdom

edit SecGW

config vpn ipsec phase1-interface

edit "IPsecGW4"

set eap enable

set eap-identity [use-id-payload|send-request]

set acct-verify [enable|disable]

set authusrgrp "ExampleGroup"

end

EAP requires the configuration of a remote RADIUS AAA platform to allow onwards authentication to apply to the IPsec

tunnel.

EAP identify may be shared by the client. If the client does not offer this, the FortiGate can use the ID from IKEv2, or it

may request the EAP identity from the client.

The acct-verify setting is used to pause the completion of the IKEv2 authentication process, until a RADIUS

accounting acknowledgment is received. This is used to assure billing.

Deployment Guide 8

GLOBAL IKE ATTRIBUTES

Global IKE attributes

FortiOS 7.0 and later offers a number of global IKE settings, for example, IKE embryonic limit, DH multiprocess, DH cache,

and DH HW offload. For SecGW deployments, it is recommended to leave the options set to default values, unless there

are specific requirements.

Links:

config system ike

IPsec global IKE embryonic limit

Tighten security for configuration payload

With FortiOS 7.0.11, 7.2.5, 7.4.0 and later, a number of server side checks are performed to prevent duplicated IP

connections. The checks safeguard from malfunctioning clients connecting without sending configuration payload

requests.

If the client is based on strongSwan, it is recommended to include this vips in your connection profile on the

strongSwan client, which enforces the client to send a configuration payload request:

IPv4

connections.<conn>.vips = 0.0.0.0

IPv6

connections.<conn>.vips = ::

Links:

strongSwan documentation

Dial-up IPsec tunnels for RAN connectivity

The most common, and therefore most practical, IPsec tunnel configuration when considering RAN connectivity is to use

dial-up VPNs. There are number of reasons for this:

The remote peer is always the initiator of the tunnel setup.

The remote peer address is generally unknown or dynamic.

Dial-up tunnel scale is significantly higher on the FortiGate platforms.

Simplified configuration and management of the FortiGate, with only a single or small number of IPsec VPN

configuration(s) required.

This doesn't preclude the use of site-to-site tunnels; they can of course be used. However, be mindful of tunnel scale

when using this approach. And the use of both types of tunnels can be combined on the same FortiGate system to

provide the SecGW functions.

Site-to-site IPsec tunnels: backhauling edge locations

In distributed MNO scenarios, where SecGW elements are deployed at edge site locations, connectivity from the edge

site to core-located service functions will also require security control. If the backhaul connections from the edge

locations to the centralized sites is considered fully secure, this may be negated. However there are still security threats

Deployment Guide 9

SINGLE-TUNNEL THROUGHPUT CONSIDERATIONS

and considerations for the MNO to be mindful of. Therefore, Fortinet recommends the use of accompanying centralized

SecGW elements to secure communication between core and edge locations.

These connections between the core and edge generally utilize the site-to-site IPsec configuration approach, as the

edge site and central site are fixed points and known entities. This differs from the RAN connectivity, where the

eNB/gNB is generally considered an unknown entity, until it initiates a connection.

In general the number of concurrent tunnels from the edge site to the core site is very small and likely only one single

tunnel. All communication is therefore aggregated across this single tunnel.

Single-tunnel throughput considerations

The continued growth in data consumption, especially with the changing behavior in data content (for example, video

streaming), coupled with the increased bandwidth per subscriber available with 5G, means that the throughput of any

single tunnel is increasing. This needs to be carefully considered, especially when sizing platforms, but also when

consulting with customers to provide guidance. Fortinet can provide detailed guidance on platform considerations and

single-tunnel throughput where needed.

NP6 powered FortiGate systems can support up to 10Gbps full duplex for a single tunnel, based on 10Gbps Upload

direction and 10Gbps Download direction.

NP7 powered FortiGate systems can support up to 55Gbps full duplex for a single tunnel, based on 55Gbps Upload

direction and 55Gbps Download direction.

Virtual FortiGate systems vary based on underlying hardware and network configuration.

Traffic tunnel design

MNOs have different demands for IPsec tunnel design. These demands are specific to the MNO and driven by various

needs. This is in part driven by the need to separate OAM, CP and UP traffic if so desired. Conversely different MNO

customer may choose to use IPsec tunnels for some traffic but not all. The different tunnel designs variations can be

grouped as follows:

One tunnel for all traffic: OAM, CP and UP

One tunnel for OAM and CP. No IPsec for UP

One tunnel for OAM. No IPsec for CP and UP

Two tunnels: one for OAM and CP, and one tunnel for UP

Two tunnels: one for OAM and one for CP. No IPsec for UP

Three tunnels: a tunnel each for OAM, CP, and UP

Some of this strategy is in part driven by the capabilities of the RAN equipment (eNB/gNB). Many MNOs utilize a multi-

vendor RAN, with much of the RAN consisting of a mix of new and older equipment. In such cases it may simply not be

possible for certain vendor equipment to support multiple tunnels in some of the ways mentioned.

Generally MNOs utilize IPsec tunnels for all traffic types: OAM, CP, and UP. Some MNOs focus primarily on OAM traffic

and CP traffic, which is seen as the most critical communication. This may be through a single tunnel or a tunnel for each,

based on the points mentioned previously. Because UP traffic accounts for the vast majority of overall traffic in the

connections to and from the RAN, some MNOs choose not to secure UP traffic with IPsec, which is a decision primarily

driven by cost reduction.

Deployment Guide 10

IPSEC TERMINATION LOCAL GATEWAY: LOOPBACK INTERFACES

IPsec termination local gateway: loopback interfaces

The IPsec tunnels are terminated on an interface on the FortiGate side. Depending on the HA architecture used, the

MNO network design and the surrounding routing configuration, using a loopback interface for the local tunnel gateway

offers a highly flexible approach to define resiliency because multiple physical interfaces can reach the loopback. On top

of this, using a loopback as the local IPsec gateway allows for streamlined and segregated tunnel configuration. It allows

for simplified tunnel configuration for the peer end devices, such that all peers use the loopback address as their peer

gateway, benefiting from anycast routing to provide load sharing and additional resiliency.

Loopback interface hardware acceleration

While the recommended path is to use loopback interfaces for local IPsec gateway termination, depending on the

systems in use, there may be some restrictions on the viability. This document is focused on NP7-based FortiGate

systems primarily, but some points of note should be taken into account:

NP7 provides accelerated loopback interface support for IPsec, and can be used for local IPsec gateway

termination.

NP6 does not provide direct acceleration of loopback interfaces for any functions including IPsec. Loopback

interfaces are considered software interfaces and therefore handled by CPU. They should not be used for local

IPsec termination, unless hardware acceleration is not required.

MTU on a loopback interface in FortiOS 7.0.x cannot be configured. When terminating the IPsec tunnel on a

loopback interface, it limits the MTU on the tunnel interface to 1500bytes. In FortiOS 7.4.1 and later, MTU can be

configured on a loopback interface to support jumbo frames.

There are cases where hardware acceleration is possible for NP6-powered appliances, but

these either have resiliency or performance considerations.

IPsec ciphering considerations

This section will discuss some of the specific considerations concerning cryptographic cyphering and IPsec functions.

This is intended as a guide, and basis for design discussion, and as such will be shaped by the demands of the MNO

network.

Crypto sets on page 12

GCM crypto sets on page 12

Diffie-Hellman groups on page 12

Rekeying and key lifetime on page 13

Perfect Forward Secrecy on page 14

Anti-replay and packet reordering on page 14

Extended sequence numbers (ESN) on page 14

Deployment Guide 11

IPSEC CIPHERING CONSIDERATIONS

Crypto sets

The FortiGate solution supports many different types of cryptographic (crypto) sets. For the most part, the chosen

crypto set is driven by the support level of the peer devices (eNB/gNB) rather than the SecGW itself. The choice is also

driven to some degree by mandates from the MNO customer's network and design. However some considerations

should be made when choosing the most appropriate crypto sets, as there is a trade-off between crypto set strength or

complexity and performance.

Multiple crypto sets can be specified for both Phase1 and Phase2 negotiations on the FortiGate, allowing for various peer

devices with different levels of support to be catered for.

GCM crypto sets

AES-GCM provides significant optimization for parallel processing verses CBC-based algorithms, for example. The use

of AES-GCM will be somewhat dictated by the support of the peer nodes in the MNO's network (eNB/gNB). Importantly,

legacy eNBs may not support AES-GCM, and as such it may not be a choice when considering interoperability.

With NP7-based systems, AES-GCM is natively supported on the NP7 itself, and provides the most optimum

performance for IPsec. Furthermore, when considering FortiGate VM, the use of AES-GCM is preferred to provide the

most optimized crypto performance. Therefore AES-GCM is recommended for NP7 based FortiGates and FortiGate VM

deployments, as long as the peer devices can also support this cryptographic function.

However, with NP6-powered systems, the AES-GCM support can only be realized with systems that also include CP9

content security co-processors in E model FortiGate platforms. NP6 itself does not support AES-GCM. The use of AES-

GCM must be carefully considered with NP6 systems, because the performance limitation will be that of the CP9 itself

rather than the NP6. For this reason, AES-GCM is not recommended for use with NP6-powered systems, where high

aggregate performance is required.

GCM is configured in phase1 and phase2:

config vpn ipsec phase1-interface

edit "secgw"

set proposal aes128gcm-prfsha256 aes256gcm-prfsha384

end

config vpn ipsec phase2-interface

edit "secgw"

set phase1name "secgw"

set proposal aes128gcm aes256gcm

end

Remember the order of the proposals also dictates the priority.

Diffie-Hellman groups

The Diffie-Hellman (DH) key exchange in phase1 is used to negotiate and exchange private keys for phase2.

The security level used by the DH key exchange should be equivalent to the exchanged symmetric key. The security

level (symmetric key length in bits) is an operator decision, and this section only recommends which DH groups to match

the symmetric key length for the SA (phase2).

Elliptic curves (EC) offers better performance over security level compared to the MODP groups. If the IPsec client

supports elliptic curves, then the ECDH groups are preferred over the MODP groups.

Deployment Guide 12

IPSEC CIPHERING CONSIDERATIONS

Performance of the DH key exchange is mainly important in case of a catastrophic network failure, such as a regional

power outage. When the power is restored, many base stations would initiate IPsec tunnel setup simultaneously, putting

the SecGW under heavy load. But even under normal circumstances, where tunnel re-keying is spread out, the SecGW

would benefit from better DH key exchange performance. Fortinet has performed tunnel setup tests, where 10,000

tunnels are re-establish at the same time, whilst, in parallel, traffic ramps up to 25Gbps, which is around 50% of the

maximum performance of the device under test (FG-2600F with FortiOS 6.4.8.). The CPU load was lowest for groups 19,

31, and 32. It was also verified that DH calculations for the recommended DH groups are offloaded to CP9.

The asymmetric key lengths used in DH does not operate with the same bit length as the symmetric keys. For

comparison the symmetric key equivalence has been added to the recommendation table below. Key equivalence is

documented in NIST SP800-57 Part1 rev5.

The table only includes the recommended DH groups; it omits DH groups that are not recommended. Any DH groups <15

are not recommended due to low security level. The Brainpool curves (RFC 6954) performs poorly compared to DH

group 19, 21, 31, and 32, so they are also omitted. Recommended DH groups are listed for both 128- and 256-bits

symmetric key length in prioritized order.

DH Group Computation

Asymmetric key

length

Symmetric equi-

valence

Recommended for Phase2

encryption

31 EC 256-bits 128-bits AES-128-GCM/CBC

19 EC 256-bits 128-bits AES-128-GCM/CBC

15 MODP 3072-bits 128-bits AES-128-GCM/CBC

21 EC 521-bits 256-bits AES-256-GCM/CBC

$32$ EC 448-bits 224-bits AES-256-GCM/CBC

DH group 32 offers 224-bits security level. If that is considered enough, it can be a more performance efficient

alternative to DH group 21.

The DH groups in phase2 should be set to the same value as for phase1, and PFS is recommended, see Perfect Forward

Secrecy on page 14.

Rekeying and key lifetime

The default key lifetime for Phase1 FortiOS is 24 hours and defined in seconds (86400 seconds). This provides a

reasonable security level whilst maintaining good performance characteristics. Key lifetime can be changed to meet with

the requirements imposed by the MNO and the interoperability constraints of the connecting peer eNB/gNB devices. In

many cases the default setting can remain.

For Phase2 rekeying, the default setting is based on seconds and defined as 43200 seconds (half the key lifetime of

Phase1) or 12 hours. Reducing this rekeying time will increase the security characteristics of the IPsec tunnels; however,

consideration should be made as to performance impacts for both the FortiGate and the peer eNB/gNB devices. Setting

the rekey interval to a small window will increase the performance overhead on both endpoints and specifically for the

SecGW, which will service many peer IPsec tunnels. As with key lifetime, rekeying frequency will be dictated in part by

the MNO design and requirements.

Rekeying can be specified in time (seconds) as well as data (kilobytes) or using both metrics. Because the SecGW will be

processing generally large volumes of data and potentially large single tunnel volumes, it is recommended to use only a

time based rekeying configuration. This will alleviate high performance impact for rekeying based on large data volumes.

Deployment Guide 13

IPSEC CIPHERING CONSIDERATIONS

Perfect Forward Secrecy

Perfect Forward Secrecy (PFS) is a mode which causes a new Diffie-Hellman key exchange to occur each time a phase2

SA is established or rekeyed. The alternative is to use the phase1 SA to protect the key exchange, which means that if

the phase1 SA is broken, all of the IPsec data will also be compromised. For this reason it is generally considered good

practice to use PFS, and it is enabled by default on the FortiGate. Unless there is a specific reason to disable this, it

should remain enabled.

Anti-replay and packet reordering

Anti-replay as a function of security validation is especially important with IPsec, as it ensures that replayed packets

cannot be used to subvert the security controls of the system. But anti-replay generally can introduce packet reordering

issues, as the very nature of anti-replay is that the right packet arrives in the correct sequence for the appropriate

security control (decryption) to be applied. Where significant packet size variations can exist within a given traffic

stream, there is potential for smaller packets to be processed quicker than larger packets, and fall into an out of order

scenario. For anti-replay to be used effectively with IPsec, packet ordering must be carefully considered.

NP7 based systems: NP7 provides an integrated packet ordering engine that works in conjunction with the IPsec

functions to ensure anti-replay and packet ordering issues are mitigated. This is inherent to the functions of NP7,

and therefore anti-replay can be enabled.

NP6 based systems: NP6 does not have an integrated packet ordering function. Each NP6 operates multiple crypto

engines to provide high crypto performance, and this can cause packet ordering issues with large packet size

variations. A workaround exists whereby the number of crypto engines can be reduced, which mitigates packet

ordering issues. However, reducing the number of crypto engines also directly reduces the crypto performance of

each NP6. Therefore this should be carefully considered along with the tradeoffs imposed.

For simplicity, anti-replay can be disabled, thus mitigating the potential impacts it imposes with NP6 based systems. But

disabling anti-replay can be perceived as reducing the security strength of the solution, and may not be acceptable. The

MNO must be aware of these considerations in order to decide on the appropriate path.

In many MNO networks, anti-replay can cause adverse effects for traffic with QoS applied. For

this reason, anti-replay is often disabled to ensure QoS marked traffic is unaffected.

Extended sequence numbers (ESN)

To support high-speed IPsec processing, larger extended sequence numbers should be used. ESN is an extension to the

standard sequence number, increasing from 32-bit to 64-bit. The use of ESN is mutually agreed as part of the SA

negotiation, which is implicit with IKEv2. By default ESN (64-bit) is used, unless explicitly negotiated to use 32-bit

sequence numbers.

NP7 based systems natively support ESN with the processor and therefore provide hardware acceleration. This

further extends the performance capability for NP7 to meet the demands of high performance IPsec services.

NP6 based systems do not support the use of ESN. While FortiOS does support ESN configuration. If npu-offload

is enabled, then ESN configuration options are disabled. Thus using ESN on NP6 based systems effectively renders

the NPU offload unavailable, which drastically reduces the IPsec performance of the system.

Regarding NP7 based systems, ESN can be configured to require the peer to support and use ESN. Merely allow the use

of ESN, if mutually agreed, or force ESN to be disabled. Generally ESN should be set to allow in most cases.

config vdom

edit SecGW

config vpn ipsec {phase1|phase1-interface}

Deployment Guide 14

IPSEC AND PACKET FRAGMENTATION

edit <name>

set esn [require|allow|disable]

end

IPsec and packet fragmentation

Adding IPsec to the packet flow between cell site and mobile core adds additional packet overhead, which increases the

risk of packet fragmentation. Mobile backhaul, both when owned by the MNO itself or with RAN sharing, often consists

of multiple link types, such as dark fibers, wave lengths, radio links, and leased lines. On top of that, the network

equipment to serve the multiple link types might have different MTU capabilities. The risk of packet fragmentation

somewhere is likely, and a SecGW solution should be able to address that without being overwhelmed.

Fragmentation of encrypted packets can have quite adverse effects on IPsec. The most significant effect is that all

encrypted fragments must be collected before the entire combined reassembly can take place. This adds significant

latency to the process, as well as requiring buffer memory to hold the fragments before processing. Once received,

these fragments must be reassembled in the correct order for successful decryption to take place.

Furthermore, fragmented IPsec can easily break anti-replay, where fragments received outside the anti-replay window

are discarded. This renders the packet flow obsolete, and the originator must then retransmit. If the transmission

continues to fall outside of the anti-replay window, the communication never completes successfully, and the traffic is

dropped. If the MNO expects to experience fragmentation with the IPsec tunnels, it is recommended not to use anti-

replay.

For this reason, if fragmentation is required, it is recommended that fragmentation occurs before encryption. FortiGate

can perform this method, ensuring that the original packet is fragmented when needed whilst maintaining that the final

encrypted packet (with all ESP header additions) itself is ultimately not too big and therefore not fragmented. It

calculates this in realtime as part of the processing.

To set the fragmentation to pre-encapsulation, use the following command:

config vdom

edit SecGW

config vpn ipsec phase1-interface

edit <name>

set ip-fragmentation pre-encapsulation

end

FortiGate performs fragmentation post-encapsulation by default, which is RFC compliant. If

pre-encapsulation fragmentation is required, then it must be set on the FortiGate explicitly

using the set ip-fragmentation pre-encapsulation variable.

Hardware acceleration and fragmentation considerations:

NP6: NP6 powered systems do not support fragment reassembly on ingress. This is true for all traffic, including

IPsec. Any fragmented packets received on an ingress interface are sent to CPU to be processed, which can have

adverse affect on performance. Fragmentation on egress is supported both towards the encrypted and clear text

side.

NP7: NP7 powered systems support fragment reassembly on ingress, thus providing hardware acceleration.

Deployment Guide 15

CERTIFICATES-BASED AUTHENTICATION

The most predominant situation is that a packet is split into two fragments. NP7 supports two fragments per packet.

Additional fragments are handled by the CPU. Out-of-order reassembly, that is receiving the second fragment before

the first, is supported by NP7.

Fragmentation on egress is supported both towards the encrypted and clear text side.

According to the convention, the host does reassembly. In the SecGW use case, it means the EPC/NGC must process

the reassembly, which severely affects EPC/NGC performance. By enabling the ip-reassembly feature of NP7, this

process is offloaded from the EPC/NGC.

NP7 is very flexible and capable when it comes to IP fragmentation. For example, it is supported in hardware to

reassemble two fragments on ingress, inspect the packet, and then fragment the packet on egress, depending on the

MTU of the destination interface, and do this bidirectionally.

By default NP7 does not do reassembly. Reassembly can be enabled from CLI with this command, which is

recommended for SecGW applications:

config global

config system npu

config ip-reassembly

set status enable

end

Certificates-based authentication

When considering the SecGW, and specifically IPsec, peer devices can be authenticated in two distinct ways: pre-

shared key (PSK) and certificate.

While PSK is generally the simplest method, it is not deemed the most appropriate option for SecGW deployments. PSK

relies on the pre-configuration of the PSK on the peer devices, as well as the SecGW itself. Furthermore, if the PSK is

uncovered, then the integrity of the connection is at risk. For this reason, SecGW deployments use client certificates to

identify both peers.

3GPP Technical Specification 33.501 titled Security architecture and procedures for 5G system requires certificate-

based IPsec with the following text:

9.2 Security mechanisms for the N2 interface

...

In order to protect the N2 reference point, it is required to implement IPsec ESP and IKEv2

certificates-based authentication as specified in sub-clause 9.1.2 of the present document.

IPsec is mandatory to implement on the gNB and the ng-eNB. On the core network side, a SEG

may be used to terminate the IPsec tunnel.

CMPv2 on page 16

Accessing the CMP server on page 17

CMPv2

According to 3GPP Technical Specification 33.310 says CMPv2 shall be the supported protocol to provide certificate

lifecycle management capabilities. This applies both to SecGW and Network Elements (for example, base stations).

Deployment Guide 16

CERTIFICATES-BASED AUTHENTICATION

FortiOS supports the use of CMPv2 for certificate enrollment and renewal. FortiOS does not provide PKI services, and is

not a CA for example. In general, the MNO will operate an existing PKI solution that supports CMPv2 to which the

FortiGate is required to communicate for its certificate management services.

Accessing the CMP server

The CMP server is a sensitive application, so access to it should be limited. Naturally it should be protected by a firewall,

such as FortiGate. A number of proposals are offered to protect the CMP server and access to the CMP server. Typically

the security measures are required or audited by the MNO's security team.

The following topics include examples on how to apply security in the RAN and mobile core domain.

Access from RAN domain on page 17

Access in mobile core domain on page 17

Post tunnel establishment on page 18

Preparation on page 18

Initialization request (IR) on page 18

Key update request (KUR) on page 19

Certificate request (CR) on page 20

Key usage for CMP server in CA mode on page 20

Access from RAN domain

The RAN domain is untrusted, and care should be taken when opening up access to the CMP server. Extra care should

be taken when public internet is used as backhaul.

The 3GPP NDS/IP specification says all NDS/IP traffic shall pass through a Security Gateway before entering or leaving a

security domain.

Following are a number of security proposals about how to secure traffic passing from the RAN-domain to core-domain

by using a FortiGate:

Limit the exposed port on the CMP server to TCP port 80 (HTTP). Note, as of 3GPP TS 33.310, transport of CMPv2

messages is done over HTTP. HTTP over TLS is not mandated as CMP provides built-in integrity protection and

authentication.

Limit by only allowing specified source IP addresses. Could be using a Geo IP or by creating an address object with

certain announced prefixes ISPs/ASNs.

Use DNAT in the firewall to not expose the IP address of the CMP server.

Use SNAT in the firewall to limit which IP addresses the CMP server would be allowed to communicate with.

Apply DOS protection in the firewall to prevent overwhelming attacks.

Apply Intrusion Prevention System (IPS) in the firewall to protect the CMP server.

Use FortiWeb, Fortinet's web application firewall to protect the CMP server.

Limit access through a temporary VPN tunnel with PSK. Only exposing the CMP server to the tunnel interface and

not RAN.

Access in mobile core domain

Here the MNO is typically in full control of the network. It is recommended to limit access to the CMP server inside the

core domain. Access could be limited by segmentation that keeps the CMP server in a separate VRF. Several of the

techniques listed previously are also applicable.

Deployment Guide 17

CERTIFICATES-BASED AUTHENTICATION

Post tunnel establishment

When the authenticated IPsec tunnel is established, the base station still needs access to the CMP server for key update

requests (KUR). This can be performed inside the VPN tunnel using, for example, the management plane and the OAM IP

of the base station.

Preparation

To ensure the FortiGate can verify responses from the PKI server, a remote CA certificate is required. This is imported

within the global context such that it is available in all VDOMs. This certificate should be provided by the PKI owner.

config global

config certificate ca

edit "mno_cmp_server"

set ca "-----BEGIN CERTIFICATE-----

MIID0jCCArqgAwIBAgIUGS8W7UImNyEeKe5SgZo027oGuP0wDQYJKoZIhvcNAQEL

...

QAgVXsvG77rqifNajoGGycqBVZTckkIRPrRSf6HTtxBG6ooco3Doa6+RLpZQd/5U

ELrrVze6X6wQibBV6Ukf5L3Qhs9yw==

-----END CERTIFICATE-----"

end

Assuming the PKI solution is going to use vendor based authentication. You may also need to supply the PKI owner with

the Fortinet_CA and Fortinet_Sub_CA certificates: The Fortinet_Factory certificate is used for vendor based

authentication, and this is signed by the Fortinet_Sub_CA.

The Fortinet_Factory certificate has a Common Name (CN) of the appliance's serial number. The serial number should

thus be communicated to the PKI owner such that the PKI system can be pre-provisioned for dealing with CMPv2

requests from the FortiGate.

For an FGCP cluster, the Fortinet_Factory from the primary node will have overwritten

(synchronized) the equivalent on the secondary node, so a bit of care is needed to provide the

correct serial number.

Initialization request (IR)

The IR is initialized as such:

execute vpn certificate local generate cmp seg1 2048 pkivip.mno.com:8080

/cert/cmp2/ mno_cmp_server Fortinet_Factor

This requests a new certificate is generated through CMPv2:

Local name: seg1

Key length: 2048-bit key length

PKI listener: pki-vip.mno.com:8080

PKI path: /cert/cmp2/

Server certificate: mno_cmp_server

Authentication certificate: Fortinet_Factory

Running this command automatically introduces the following configuration:

SecGW-node0 (global) # show certificate local seg1

config certificate local

Deployment Guide 18

CERTIFICATES-BASED AUTHENTICATION

edit "seg1"

set password ENC <encrypted string>

set enroll-protocol cmpv2

set cmp-server "pkivip.mno.com:8080"

set cmp-path "/cert/cmp2/"

set cmp-server-cert "mno_cmp_server"

end

SecGW-node0 (global) # sudo SecGW show vpn certificate local seg1

config vpn certificate local

edit "seg1"

set range global

end

The certificate can now be referenced in the IPsec configuration to enable authentication through digital signature.

config vdom

edit SecGW

config user peer

edit "SecGW4"

set ca "mno_cmp_server"

end

config vpn ipsec phase1-interface

edit "IPsecGW4"

set authmethod signature

set certificate "seg1"

set peer "SecGW4"

end

Key update request (KUR)

The validity of the certificate from PKI is dependent upon the PKI solution. Before the certificate expires, a renewal is

required. The carrier may wish to keep this process manual, so it can be performed in a maintenance window. If not, a

KUR can be scheduled.

config global

config certificate local

edit "seg1"

set cmp-regeneration-method keyupate

set auto-regenerate-days 90

set auto-regenerate-days-warning 60

end

The number of days is a measure of how many days before certificate expiry. The above config could be applied to a

certificate to enable renewal 90 days before expiry, and a warning at 60 days before the expiry.

The Forticron daemon in FortiOS checks the expiry every minute against these values, and will attempt a KUR when the

conditions are met.

To see this:

config global

diagnose test application forticron 2

end

Deployment Guide 19

CERTIFICATES-BASED AUTHENTICATION

Certificate request (CR)

CR is similar to KUR (for the FortiGate at least); however, it has some implications on the PKI server configuration as the

CR requires the status of the endpoint (that is, the FortiGate) to be changed.

config global

config certificate local

edit "seg1"

set cmp-regeneration-method renewal

end

Key usage for CMP server in CA mode

When using the CMP server in CA mode, the CA server signs the responses from the CA server. This requires the CA

certificate to have the digitalSignature key usage flag, which may not be the case. In order to allow the use of CA

certificates without this, use the following CLI command:

config vdom

edit SecGW

config vpn certificate setting

set cmp-key-usage-checking disable

end

Deployment Guide 20

High availability

The role of SecGW demands maximum resiliency and availability. Any failure of the SecGW effectively cuts off the RAN

communications, and leads to major service outage for the MNOs subscribers.

Most MNO customers group their resiliency and redundancy requirements into two distinct definitions:

Intra-site redundancy:The site/location in which the SecGW is placed provides local redundancy, such that a failure

with a single SecGW system does not impact the site/locations ability to provide continued service

Inter-site redundancy:The failure of a complete site or location can be catered for by way of redundant systems in

neighboring sites or locations. This is also sometimes referred to as geographic redundancy or geo redundancy.

The following sections will discuss how the FortiGate SecGW can support these key redundancy requirements:

Intra-site redundancy on page 21

Inter-site redundancy on page 22

SecGW scale-out with FGSP on page 22

Intra-site redundancy

Intra-site redundancy can use either FGCP or FGSP clustering.

Intra-site redundancy with FGCP

Providing redundancy within a site is generally realized using FGCP. It provides the most seamless clustering and

resiliency for the local site, but can be at the expense of optimal usage of the SecGW investment. FGCP is naturally an

active or backup scenario for FortiGate units. The ability to broaden FGCP and utilize virtual clustering provides

significant value as it allows the second FortiGate in the FGCP cluster to be utilized for traffic for different VDOMs.

However, an important factor to consider is SCTP multihoming support, which is only provided using FGSP and not

FGCP.

Intra-site redundancy with FGSP

A point of note with FGSP is that it can indeed be considered for intra-site redundancy as well. While intra-site

redundancy may be well suited to FGCP clustering, FGSP can certainly be considered for this. In essence this allows for

Deployment Guide 21

INTER-SITE REDUNDANCY

localized site redundancy based on FGSP, where all local nodes operate as FGSP peers sharing IPsec SA state

information.

Inter-site redundancy

This redundancy architecture is generally realized using FGSP. The nature of inter-site requires that another location

take over from a failed site with minimal interruption, and FGSP provides the most flexible way to do this in combination

with dynamic routing protocols. However, inter-site redundancy can also be achieved by way of FGCP, but only

between two distinct sites. FGCP requires some form of L2 connectivity between the two sites, as well as peering

routers, and naturally precludes the ability to deliver intra-site redundancy.

Furthermore FGCP can be combined with FGSP to provide both inter-site (using FGSP) and intra-site (using FGCP)

redundancy across multiple locations.

SecGW scale-out with FGSP

FGSP also provides the benefit of scale-out options to meet traffic and performance demands. Essentially allowing a

given site to scale beyond a single device based on an active-active configuration, applying to both intra-site and inter-

site architectures. This provides great flexibility for MNOs, allowing them to invest and grow the SecGW solution as the

demand and forecast grows. The number of peers in FGSP is discussed in How many nodes? on page 37.

Deployment Guide 22

HA flexibility

The flexible nature of both the FGCPand FGSPFortiGate HA options ensure that the most appropriate architecture can

be realized in almost any situation. This provides huge value to the MNO, and significant differentiation for the FortiGate

solution verses its competitors.

In many cases, FGSP will be the most appropriate solution for the MNO, and is the most widely adopted in the MNO

customer base. Combined with dynamic routing protocols, FGSPprovides the best redundancy architecture and most

seamless resiliency model.

Design considerations

The next sections explore some example topologies and concepts that can be used in the MNO environments. The

example topologies are not prescriptive; nor are they complete (but they do show how flexible the platform is).

Essentially whatever the customer requires can likely be met. It's even the case that the MNO may want to mimic

existing competitor solutions and evolve as confidence is gained.

The following sections on FGSP, FGCP and FGSP+FGCP architectures should be consumed fully and in order written as

to some degree they build from each other.

Keep in mind that this documentation assumes the use of physical FortiGate appliances with NP7 security processing

units or virtual FortiGate appliances. NP6-based appliances can also adopt the same design but the throughput is limited

to the use of only one NPU for IPsec hardware offload.

Deployment Guide 23

Inter- and intra-site redundancy with

FGSP

FortiGate Session Life Support Protocol (FGSP) is clustering technology that lends itself particularly well to providing a

geo-resilient Security Gateway solution.

FGSP provides synchronization of session information between multiple standalone FortiGate units. Of particular

importance for this use case, the synchronization includes dial-up IPsec tunnels, SCTP, and UDP sessions. This allows

resilience and scale to be accommodated by routing traffic intelligently, and providing backup paths that allow the traffic

to be re-routed (quickly) in the event of a failure.

The synchronization can be performed over layer 3, so the location of the FGSP members can be based on serving the

SecGW clients and resiliency requirements. Regardless of whether the FGSP solution is across multiple sites, a single

site, or anything in between, it functions the same way. Thus FGSP, provides for both intra- and inter-site redundancy

scenarios.

FGSP also allows some elements of the configuration to be synchronized; however, this would be better achieved using

FortiManager.

Design concept

The design aims to best leverage FGSP to make full use of its ability to synchronize over layer 3, and to provide a means

to scale horizontally as well as cater for geographic resiliency.

MNOs are likely to have their own requirements around the deployment, and that may involve FGCP (with or without)

FGSP between sites. These requirements aren't wrong, but this section of the document presents a reference

architecture that Fortinet believes provides a better RoI without compromising network availability, and is based upon

FGSP alone.

Standalone node description

In order to understand the FGSP solution, first consider a standalone node. If this is understood, the FGSP is realized by

adding synchronization and routing.

Deployment Guide 24

STANDALONE NODE DESCRIPTION

Physical connectivity on page 25

Logical connectivity on page 26

Routing loopback advertisement (and path preference) on page 27

Routing SecGW inner address advertisement on page 29

Other routing configuration on page 30

Physical connectivity

The diagram depicts a typical deployment scenario: the MNO provides a resilient pair of provider edge routers to

connect to the core network. The actual detail of the connectivity will vary with each MNO. For example, the MNO may

want to run trusted and untrusted traffic on different physical fiber.

Objective Motives

The single node is powered by resilient

power feeds.

Fault tolerance: power feed and PSU failure

Maintenance of power without service impact

The single node is connected to both

MNO routers.

Fault tolerance: network connectivity

Maintenance of MNO routers without service impact

There is no layer 2 requirement.

The support teams and vendors for the router

and SecGW may be different. Layer 3 is a

safer demarcation of responsibility.

There is no requirement to have layer 2

between the MNO routers.

Each physical link is configured as an

802.3ad aggregate.

Allows for connectivity to be scaled without

significant changes to the configuration

For 802.3ad, consider LACP mode, speed, frame distribution algorithm (default is L4 but you may not get enough

variation with this algorithm to use the link effectively), and minimum link configuration. Example interface configuration:

Deployment Guide 25

STANDALONE NODE DESCRIPTION

config global

config system interface

edit "Agg1"

set vdom "root"

set type aggregate

set member "port2" "port3"

set description "SecGW Agg1 to PE1 BE1"

set lacp-speed fast

set min-links 2

set algorithm L3

end

Logical connectivity

The SecGW requires a link address for each physical connection into each logical routing function. This allows the

FortiGate to advertise routes accordingly using dynamic routing protocols. FortiOS is compliant with RFC 3021, so

address space usage can be minimized.

config global

config system interface

edit "Agg1.1001"

set vdom "SecGW"

set ip 10.0.0.1 255.255.255.254

set description "SecGW Agg1.1001 to PE1 BE1.1001 vrf: RAN-untrust"

set interface "Agg1"

set vlanid 1001

end

The link addresses can be reused elsewhere in the MNO backbone as there is no requirement

for the backbone to redistribute these connected routes. This will be a consideration later in

the document when multiple sites are discussed.

In addition to the link addresses, a common IPsec tunnel termination end point address is needed. Loopback hardware

offload is supported from NP7, and this is the reason for the earlier design consideration.

config global

config system interface

edit "Lo1-IPsecGW"

set vdom "SecGW"

set ip 100.64.0.1 255.255.255.255

set description "SecGW Lo1-IPsecGW: IPsec termination endpoint"

set type loopback

end

MEC might be a valid SecGW use case for considering the use of FortiGate-VM.

From FortiOS 7.0.6 and 7.2.0 FortiGate-VM's vSPU support acceleration of encrypted traffic with a five times

performance increase. Both IPsec CBC and GCM are supported. Some limitations remains, that is, IPv6 and no anti-

replay support.

This documentation is not generally targeting NP6-based FortiGate appliances; however, it is worth noting that with this

platform there is the possibility to use one end of an NPU VDOM link to terminate IPsec, although the IPsec throughput is

thus limited to one NPU.

Deployment Guide 26

STANDALONE NODE DESCRIPTION

Routing loopback advertisement (and path preference)

It is desirable for the operator of the SecGW to fully control how traffic is attracted to the appliance. This allows less

complex maintenance and recovery of the service as fewer parties are required to make changes. It is recommended to

use dynamic routing, specifically eBGP.

eBGP is used to advertise the loopback address into the MNO backbone network to influence how the traffic routes to

the SecGW, but the eBGP path selection algorithm must be understood.

Consider the diagram with the following concerns:

The 5G gNodeB is connected to a single MNO router.

The MNO has opted to use the links as active/backup. (This is potentially a real scenario, where the primary link is

optically tapped, and/or the operator desires the traffic path to be easily determined.)

Influencing the traffic initiated by gNodeB to go through the primary SecGW link requires specific integration to the MNO

router. The only (non-proprietary) BGP selection parameter that would trump the locally learned route (and therefore

force the traffic to the primary MNO router through the MPLS cloud) is local preference (LP). The more commonly used

multi-exit discriminator (MED) or AS path length would not help in this scenario because they are of no use for platforms

sharing the same core MNO router.

If the MNO can guarantee this specific scenario (client and SecGW sharing the same PE) is

never going to occur, or that they have no concerns about the consequences of such, then

the other mechanisms of MED or AS path length can safely be implemented.

LP is specific to the AS, but the SecGW is not in the same AS as the MNO router. We would thus need route policy on the

MNO router to interpret a flag that the SecGW sends within the BGP advertisement to apply an LP that favors the use of

the primary path across the whole backbone network.

This consideration applies also between influencing traffic flow for the nodes in an FGSP

cluster, and this topic is discussed in greater detail later in this chapter

An example of how this might be achieved between a FortiGate SecGW and a Cisco IOS-XR MNO router is using a BGP

community as the flag.

First of all, FortiGate sets a community to flag:

config vdom

edit SecGW

config router route-map

edit "set-primary-community"

Deployment Guide 27

STANDALONE NODE DESCRIPTION

config rule

edit 1

set set-community "65001:1"

set set-community-additive enable

end

edit "set-secondary-community"

config rule

edit 1

set set-community "65001:2"

set set-community-additive enable

end

config router bgp

config neighbor

edit "10.0.0.0"

set route-map-out "set-primary-community"

edit "10.0.0.2"

set route-map-out "set-secondary-community"

end

And then configure (and apply to the BGP session) a route policy, for example:

route-policy SecGW-set-LP

if community matches-every (65100:1) then

set local-preference 200

delete community in (65100:1)

endif

if community matches-every (65100:2) then

set local-preference 100

delete community in (65100:2)

endif

pass

end-policy

This routing strategy needs to be thought through and agreed with the MNO. The goal is a one-time route policy

configuration on the MNO routers that allows the SecGW configuration to influence the traffic path. That is, the SecGW

operator can now change paths by simply changing community value advertised. It might even be advisable to include

this in the initial deployment, even if not used initially, as it includes flexibility without the need for a future redesign.

Cisco IOS-XR has been used as a convenient example. This operation is a fairly basic route

policy, and should be supported by any MNO router product.

If the goal is to use only one link, then the return path also needs to be suitably manipulated.

config vdom

edit SecGW

config router route-map

edit "set-lp-high"

config rule

edit 1

set set-local-preference 200

Deployment Guide 28

STANDALONE NODE DESCRIPTION

end

edit "set-lp-low"

config rule

edit 1

set set-local-preference 100

end

config router bgp

config neighbor

edit "10.0.0.0"

set route-map-in "set-lp-high"

edit "10.0.0.2"

set route-map-in "set-lp-low"

end

It is desirable to advertise only the routes needed to provide service:

config vdom

edit SecGW

config router prefix-list

edit "IPsecGW4-loopback-addresses"

config rule

edit 1

set prefix 100.64.0.1 255.255.255.255

unset ge

unset le

end

config router bgp

config neighbor

edit "10.0.0.0"

set prefix-list-out "IPsecGW4-loopback-addresses"

edit "10.0.0.2"

set prefix-list-out "IPsecGW4-loopback-addresses"

end

config redistribute "connected"

set status enable

end

Routing SecGW inner address advertisement

The same considerations are needed for the trusted side as for the loopback advertisement. In the configuration

examples that follow, the SecGW is allocating inner addresses from the 100.64.16.0/20 IP address range. The inner

addresses present themselves as static host routes once the VPN is established.

Deployment Guide 29

STANDALONE NODE DESCRIPTION

config vdom

edit SecGW

config router prefix-list

edit "Client-Inner-Addresses"

config rule

edit 1

set prefix 100.64.16.0 255.255.240.0

unset ge

set le 32

end

config router bgp

config neighbor

edit "10.0.100.0"

set prefix-list-out "Client-Inner-Addresses"

set route-map-in "set-lp-high"

set route-map-out "set-primary-community"

edit "10.0.100.2"

set prefix-list-out "Client-Inner-Addresses"

set route-map-in "set-lp-low"

set route-map-out "set-secondary-community"

end

config redistribute "static"

set status enable

end

Other routing configuration

The other routing configuration to consider is to enable ECMP (in the case where both links run with equal routing costs),

and enable features to detect failures.

config vdom

edit SecGW

config router bgp

set ebgp-multipath enable

end

In most cases, the SecGW will be directly plugged into the MNO router, and LACP will help ensure the link status is good

through any intermediary devices. However, tuning the BGP parameters to protect from less likely failures is also

recommended:

config vdom

edit SecGW

config router bgp

set keepalive-timer 10

set holdtime-timer 30

end

Deployment Guide 30

STANDALONE NODE DESCRIPTION

BFD is an important part of the solution to ensure fast failure in case of a link failure. Without BFD the solution would rely

on BGP timers in certain failure scenarios. BFD will detect a failure in a smaller timeframe and trigger the BGP process to

update the routing table.

config vdom

edit SecGW

config system settings

set bfd enable

set bfd-desired-min-tx 300

set bfd-required-min-rx 300

end

config router bgp

config neighbor

edit "10.0.0.0"

set bfd enable

end

Additional per BGP peer configuration should also be considered. The advertisement-interval is the amount of time

between successive routing update messages. This is a balance between fast convergence and route stability in the

case of flapping routes. The soft-reconfiguration grants the ability to trigger BGP updates based on configuration

changes without having to restart the BGP process.

config vdom

edit SecGW

config router bgp

config neighbor

edit "10.0.0.0"

set advertisement-interval 1

set soft-reconfiguration enable

end

The MNOmust agree to the features and the respective timers because tuning these

parameters can affect knock-on performance and testing implications in the MNO

router/network.

In addition, to the above features, the BGP sessions should be authenticated by the peers to ensure the BGP sessions

only establish between authenticated BGP peers. The password will be stored as an MD5 hash.

config vdom

edit SecGW

config router bgp

config neighbor

edit "10.0.0.0"

set password <shared password>

end

When using IPv6, it is recommended to keep distinct IPv4 and IPv6 peers.

Deployment Guide 31

MULTI-NODE FGSP CLUSTER SOLUTION

config vdom

edit SecGW

config router bgp

config neighbor

edit "10.0.0.0"

set activate6 disable

end

Multi-node FGSP cluster solution

The Standalone node description on page 24 section described the set up of a single standalone node in a fault

tolerant configuration. This section expands upon that to describe a multi-node FGSP cluster solution.

From FortiOS 7.0.8, a number of new features have been introduced for IPsec in FGSP, such as per-tunnel failover, DPD

support in FGSP, and group-id in standalone clusters.

This section shows Fortinet's reference deployment using these new FGSP features.

Per-tunnel failover

This new feature basically switches away from per-gateway failover to per-tunnel failover. FGSP cluster members are

now aware if a peer is reachable using DPD. If a peer is deemed dead, the tunnel role is changed to role=standby, and

when ESP traffic reaches another cluster member, it can elect itself as primary setting role=sync-primaryfor that

tunnel. FortiOS does not forward traffic into a tunnel with role=standby.

The feature is enabled in phase1 config and should be combined with dpd config.

config vpn ipsec phase1-interface

edit <name>

set fgsp-sync {enable | disable}

end

The fgsp-sync feature together with DPD is applicable for building FGSP clusters with multiple members, and it

distributes the load between them using ECMP. See Anycast on page 35 for ECMP recommendations.

Link:

FGSP per-tunnel failover for IPsec

Allow IPsec DPD in FGSP members to support failovers

Synchronization

The nodes in the FGSP cluster requires a layer 3 path between them, considered resilient two paths. However, it is

deemed best to only use static routing as it is best to keep this simple, and there is no requirement for any convergence.

FGSP will source from the IP address of the egress interface it uses. Thus, you must configure the peer address and

routing with this in mind.

When using multiple HA links, FortiGate considers each peer as an individual peer. To avoid this, set a group-id and a

group-member-id.

Deployment Guide 32

MULTI-NODE FGSP CLUSTER SOLUTION

config global

config system standalone-cluster

set standalone-group-id 200

set group-member-id 1

end

config system cluster-sync

edit 1

set peerip 100.64.1.11

set syncvd "SecGW"

set down-intfs-before-sess-sync "Lo1-IPsecGW"

set secondary-add-ipsec-routes disable

edit 2

set peerip 100.64.1.13

set syncvd "SecGW"

set down-intfs-before-sess-sync "Lo1-IPsecGW"

set secondary-add-ipsec-routes disable

end

config system ha

set session-pickup enable

set session-pickup-expectation enable

set override disable

end

configure system interface

edit "Agg1.1000"

set vdom "root"

set ip 100.64.5.11 255.255.255.254

set description "SecGW Agg1.1000 to PE1 BE1.1000 vrf: FGSPSync"

set interface "Agg1"

set vlanid 1000

edit "Agg2.1000"

set vdom "root"

set ip 100.64.5.13 255.255.255.254

set description "SecGW Agg2.1000 to PE2 BE1.1000 vrf: FGSPSync"

set interface "Agg2"

set vlanid 1000

end

config vdom

edit root

config system settings

set bfd enable

set bfd-desired-min-tx 300

set bfd-required-min-rx 300

end

config router static

edit 2

set dst 100.64.1.11 255.255.255.255

set gateway 100.64.5.10

set device "Agg1.1000"

set bfd enable

edit 3

set dst 100.64.1.13 255.255.255.255

Deployment Guide 33

MULTI-NODE FGSP CLUSTER SOLUTION

set gateway 100.64.5.12

set device "Agg2.1000"

set bfd enable

end

This configuration (along with the complimentary configuration on the other FGSP nodes) ensures there are two physical

paths for the FGSP synchronization traffic to follow:

100.64.5.11 <-> 100.64.1.11

100.64.5.13 <-> 100.64.1.13

When using multiple links for FGSP sync, it is important to specify a standalone-group-id and group-member-id

because it avoids sending duplicate sync messages and handles parsing duplicated messages. FortiGate would know its

peers, and only send the sync message on one of the links/paths. It is up to the MNO to design these two paths to be as

diverse as desired. At a minimum, it is recommended to ensure that the paths are to two different upstream routers.

The IKE process on an FGSP node labels itself as IKE primary if it terminates the dial-up VPN traffic, and this is

communicated to the other FGSP peers that mark themselves as IKE secondary.

set ipsec-tunnel-sync enable ensures the VPN tunnels are synchronized on all FGSP nodes, but also ensures the

inner client IP address is not installed into the FIB, unless the particular node is IKE primary (which ensures the return

path to the client is symmetrical through the IKE primary node).

With set secondary-add-ipsec-routes disable, the IKE termination endpoint remains down (therefore, not

advertised into the MPLS backbone network) until synchronization is complete.

Conditional advertisement

As you will see in Synchronization on page 32, the FGSP IKE process controls the advertisements of the inner client IP

addresses; however, if the eBGP connection into the trusted VRF is not up, then the SecGW should not advertise the

IPsec termination endpoint at all. It does not have the onward routing and would thus black-hole traffic. To workaround

this issue, we use conditional advertisement configuration to prevent the IPsec termination endpoint advertisement,

unless the route to the trusted network is present in the SecGW routing table.

config vdom

edit SecGW

config router prefix-list

edit "LTE-IPv4-Addresses"

config rule

edit 1

set prefix 172.22.255.0 255.255.255.0

unset ge

set le 32

end

config router route-map

edit "RAN_TRUST-eBGP-Check"

config rule

edit 1

set match-ip-address "LTE-IPv4-Addresses"

end

edit "IPsecGW-loopback-addresses"

config rule

Deployment Guide 34

PUTTING IT ALL TOGETHER

edit 1

set match-ip-address "IPsecGW4-loopback-addresses"

end

config router bgp

config neighbor

edit "10.0.11.1"

config conditional-advertise

edit "IPsecGW-loopback-addresses"

set condition-routemap "RAN_TRUST-eBGP-Check"

end

This is applied to all relevant neighbors. The prefix-list must contain information unique enough to ensure the route is

from the trusted network. (It is no use to match a route from the untrusted side of the network!)

Putting it all together

Using the same loopback address across all the nodes within the FGSP solution, with IKE controlling FIB entries and well

thought through dynamic routing, a geographically resilient solution can be found that best utilizes the hardware for the

end customer.

Anycast on page 35

Dead peer detection on page 37

How many nodes? on page 37

Anycast

Using the same address on multiple nodes is referred to as anycast. Although a really useful feature, it requires the MNO

to implement the network with due consideration.

If we consider the normal BGP path selection algorithm, the IGP link cost is considered if all parameters considered

before are equal. When this is used well, the path to the less costly endpoint is taken, even though the same IPsec

endpoint is advertised in multiple geographic places.

Deployment Guide 35

PUTTING IT ALL TOGETHER

The diagram shows an example of an underlying network with the IGP costs marked on each link. The two SecGWs are

advertising the IPsec loopback termination endpoint with no discernible difference in BGP metric, until the IGP costs are

considered.

The following path would be used:

eNodeB -> S04PE01 -> S04PE02 -> S03PP01 -> S02PE01 -> SecGW

As long as the MNO has considered the costs and related them to geography, then the eNodeB uses the closest SecGW.

Note the following:

If S02PE01 fails, the route would change but still terminate on the same SecGW. However, if the costs were

different, then the flow could also have shifted to the other SecGW. Therefore, the failover may not always be to the

same site in a more complex deployment. This could be considered a little anti-intuitive without this knowledge.

When we consider FGCP in the later chapter, bear in mind that this failure would cause a cluster failover, and if the

traffic went to the other site, this could also cause some confusion. Anycast is about getting the traffic to the least

cost (or closest) service endpoint rather than maintaining a resilient failover in a more tradition deployment.

In the event that the two SecGWs have equal IGP costs, that is there is no clear route preference, then there is a

potential problem. The IPsec traffic uses IKE and ESP, and these both must go to the same SecGW to ensure

stability. The core network nodes will most likely balance (ECMP) traffic based on both layer-3 and layer-4

parameters, which means it is very possible the IKE and ESP will not be sent to the same SecGW. The symptoms will

be that the IPsec tunnel established on one SecGW (the tunnel is synchronized), and the ESP is passed by the

second SecGW. At some point, the second SecGW will send an IKE message to the eNodeB, but the answer will be

sent to the other SecGW, which considers the response unsolicited, and eventually the tunnel will be torn down.

This potential separation problem can be avoided by using NAT-T, where both IKE and ESP use the same UDP session,

and thus would follow the same path to the same SecGW. This can be forced on the SecGW by adding the following:

Deployment Guide 36

PUTTING IT ALL TOGETHER

config vdom

edit SecGW

config vpn ipsec phase1-interface

edit "IPsecGW4"

set nattraversal forced

end

The client would attempt to connect first by IKE on 500/udp, but the message back from the SecGW will force the

attempt to NAT-T, and thus to use 4500/udp for both IKE and ESP encapsulation.

ECMP features vary from vendor to vendor. If possible an alternative is to configure ECMP to distribute load based only

on source IP, and then both IKEv2 and ESP traffic is forwarded to the same cluster member.

NAT-T is not supported by NP6-powered appliances for hardware acceleration. In fact if the

session matches a source or destination port of 4500, the NP6 is completely bypassed. NP7

accelerates NAT-T traffic so it is not a problem for newer platforms.

Dead peer detection

FortiOS 7.0.8 introduced IPsec DPD for FGSP cluster members. DPD is in IKEv2 RFC 7296 called liveness detection as it

is implemented by sending empty INFORMATIONAL requests. As of RFC 7296, all IKEv2 requests requires a response.

When no response after dpd-retryinterval happened for dpd-retrycount times, the peer is concluded dead.

DPD is recommend to be used in conjunction with the per-tunnel failover, which was also introduced in FortiOS 7.0.8.

Per-tunnel failover in FGSP is described in the FortiOS 7.0 Administration Guide. Per-tunnel failover enables the cluster

members to detect if a tunnel peer is reachable. If a peer is no longer reachable, FortiGate changes the IKE role from

sync-primary to standby.

FortiOS 7.0.7 and earlier did not support active DPD probing in FGSP clusters, but limited

FortiOS to only respond to DPD requests as specified in the RFC.

How many nodes?

FGSP supports synchronization of up to 16 cluster members. A cluster member in this context can be a single FortiGate

or an FGCP cluster.

The number of nodes impacts performance since SAs and sessions are synched between all cluster members. With

increased number of cluster members, comes an increase in the number of sync messages. Hence this number should

be used with due caution.

Several deployments are in production with four SecGW nodes in an FGSP cluster.

Links:

Synchronizing sessions between FGCP clusters

Deployment Guide 37

NOTEWORTHY FGSP CONSIDERATIONS

Noteworthy FGSP considerations

This section explains the following noteworthy FGSP considerations:

Mixed firmware and/or hardware on page 38

Many-to-one disaster recovery option on page 38

FGSP race condition on page 39

Hardware switch to reduce session dirtying on page 39

IKE monitoring with FGSP in split-brain situations on page 40

Mixed firmware and/or hardware

FGSP supports members of different firmware versions and/or hardware models, but testing is necessary before

implementing this in production. The performance criteria of the hardware should also be carefully considered,

particularly when deploying new firmware or hardware.

This approach is especially useful when deploying new appliances or VMs, where a gradual deployment and stability

period between sites can be easily employed.

From docs: "FortiOS 7.0.8 added group-id into the protocol header. This means that FortiGates running 7.0.8 and

later cannot perform session synchronization with FortiGates running 7.0.7 or earlier.".

Upgrading an FGSP cluster crossing FortiOS 7.0.8 causes the cluster to loose synchronization. It is generally

recommended to deploy SecGW solution with FortiOS latest 7.0.x.M release (7.0.8 or higher).

Links:

FGSP session synchronization between different FortiGate models or firmware versions

IKE monitor for FGSP

Many-to-one disaster recovery option

An MNO might have the requirement to deploy a disaster recovery capability, where one of the FGSP members backs up

multiple live sites. The following diagram shows how this could be deployed, giving due consideration to how the MNO

would apply the routing changes in the event of a site failure.

Deployment Guide 38

NOTEWORTHY FGSP CONSIDERATIONS

FGSP race condition

A potential race condition can exist with the use of FGSP in anycast deployment scenarios.

If two or more FGSP nodes are creating IPsec tunnels at a rate faster than they can synchronize, the synchronization

could cause sessions to be overwritten, for example:

2021-02-12 11:13:07.091516 ike 1:SECGW_1: deleting 10.0.0.1->31.1.0.10:500 as name will be

used for 10.0.0.1->31.1.0.3:500

In this case, the FGSP node reports that the tunnel from an FGSP synchronization identified as SECGW_1 does not match

the details of its local session with the same name, and the tunnel is deleted.

This has yet to be seen in a production environment and it is considered an unlikely scenario that should self heal.

Hardware switch to reduce session dirtying

The deployment shown in the Physical connectivity on page 25 topic ( referred to as X deployment) can cause

sessions to be flagged as dirty in the situation where the MNO is unable to maintain the use of the same physical link for

the lifetime of a session. Essentially, the session is flagged dirty if an associated packet is observed to come from an

interface that is not referenced in the session information table. When a session is flagged dirty, it is re-evaluated by the

CPU. If the packets for a session keep changing, the benefit of hardware acceleration is undone because the CPUis

constantly re-evaluating the session.

For FortiGates with a single NP7, the hardware switch can be used to combine the two interfaces into a single logical

interface to avoid this situation. For FortiGates with more than one NP7, consider asymroute and auxiliary-

sessions.

Deployment Guide 39

NOTEWORTHY FGSP CONSIDERATIONS

asymroute does not support hardware offload. auxiliary-sessions creates additional

sessions, and it impacts maximum CPS and CCS on FortiGate.

Links:

Technical Tip : Difference between asymmetric routing and auxiliary sessions.

Technical Note: How the FortiGate behaves when asymmetric routing is enabled

IKE monitoring with FGSP in split-brain situations

The IKE monitor feature was implemented to enhance resiliency for FGSP clusters with two members. With the

introduction of FGSP per-tunnel failover, FGSP group/member ID and DPD support, the IKEmonitor feature is no longer

included in our reference design. The feature can be useful in cases with two cluster members and a less reliable HA link,

but generally it is recommended to update the setup in accordance with this guide.

IKEmonitor is not available from firmware FortiOS 7.2.0.

Links:

IKE monitor for FGSP

Deployment Guide 40

Intra-site redundancy with FGCP

FGCP should be well understood by the audience of this document, so this section only includes details relevant to the

SecGW deployment.

This diagram shows a typical deployment, but includes a typical (albeit simplified) core network layout to help highlight

some of the points around the deployment requirements.

The MPLS cloud provides a high level of resiliency. Deployments are expected to connect across the two PE routers in

the site, but this leads to a less than ideal deployment scenario:

NSR or BFD? on page 41

PE to PE link on page 42

Intra-site FGCP summary on page 42

NSR or BFD?

When reviewing the diagram in Intra-site redundancy with FGCP on page 41, note that:

Deployment Guide 41

PE TO PE LINK

BGP is performed from the SecGW-Region1 VDOM, which means on a cluster re-election event, and the BGP

session needs to be re-established if the VDOM mastership changes.

To work with this, use NSR (non-stop routing) on the PE routers and BGP graceful restart and HA routes on the

FortiGate. This means the routing remains in place after a failover whilst the new BGP relationship is being

established, allowing service to continue (post successful GARP).

Unfortunately, this excludes the use of BFD for fast detection of issues as this technology is specifically for quickly

establishing issues and triggering BGP to withdraw routes (the opposite of NSR). This implies the triggering of

failover comes down to BGP timers for some failures scenarios.

There is therefore a need to establish and agree on more aggressive BGP timers more than maybe other deployments.

This puts more overhead on the PEs and even then will not meet the capabilities of BFD. The MNO will define the

minimum BGP timers that can be used and therefore the possible service impact to be anticipated in the event of a

failure.

config global

config system ha

set route-ttl 60

end

config vdom

edit SecGW

config router bgp

set keepalive-timer 3

set holdtime-timer 9

set graceful-restart enable

config neighbor

edit "10.0.0.0"

end

PE to PE link

For resiliency purposes, there is a need of the connection (highlighted in red in the diagram on Intra-site redundancy

with FGCP on page 41) between the two local PE routers.

The FGCP will BGP peer with the two local PE routers such that a loss of a single BGP peer does not impact service.

This implies the connection between the PEs is operated at layer 2 for this deployment.

This connection must be sized appropriately. For example, if the link between Site A PE1 and Site X P1 is broken, the

link between the two local PE routers must deal with all traffic to/from the SecGW (and other services on top). The

SecGW is a transit for this traffic, so in fact this link must cater to twice the traffic (in addition to any other services).

This requirement could lead to quite an expensive investment in router hardware (in addition to the standby FortiGate

hardware) to ensure failure conditions can be handled with minimal service impact.

Intra-site FGCP summary

FGCP offers a deployment scenario which is typical to MNOs and matching that of competing vendors. FGSP (seen

earlier in this document) provides for a possibly superior model, and is a USP for the Fortinet SecGW solution.

Deployment Guide 42

Inter-site redundancy with FGCP

This section includes the following topics about inter-site redundancy with FGCP:

Cluster connectivity using MNO network on page 43

Cluster Connectivity with VXLAN on page 44

FGCP with FGSP on page 48

Cluster connectivity using MNO network

The MNO will have technologies to present layer 2 connectivity between sites. This could use multiple techniques, but a

common example is Virtual Private Wire Service (VPWS). This enables the possibility of plugging the HA connections

directly into the core network, and as far as the FGCP nodes are concerned, they are plugged directly into each other.

The connectivity between sites must be of sufficient quality (low-latency, error free, and so

on), but these requirements should not pose too difficult a task for a Telco!

Deployment Guide 43

CLUSTER CONNECTIVITY WITH VXLAN

The benefit of the geographically distanced FGCP architecture is derived when using vclusters. It allows the cluster to

act within two geographical regions. Under normal operation, the SecGW clients can be served from the local node

(reducing latency) with a resilient backup capability.

Of note here is that the failover happens at layer 3. This means NSR is not used, allowing BFD to be employed to

facilitate fast failover.

Cluster Connectivity with VXLAN

Some MNOs will have concerns about providing layer 2 connectivity between sites, which is needed for the FGCP HA

links. If this is the case, there is a mechanism using a standalone-mgmt-vdom and VXLAN for the FortiGate appliances

to provide layer 2 for themselves.

This isn't specific for SecGW and could be used in other FGCP scenarios where this level of

flexibility is required. What is currently lost is the ability to manage the cluster by a floating

management address, which prohibits the use of FortiManager for example.

Here the FortiGate 4000F family will be used to describe how to achieve this architecture.

The initial configuration activity is preferred to be performed prior to the cluster units being in separate locations. The

following topics describe this process:

Interface considerations on page 45

Logical connectivity for HA on page 46

Deployment Guide 44

CLUSTER CONNECTIVITY WITH VXLAN

Interface considerations

Choose appropriate ports for HA as per normal considerations. On the FG-4200F we have ports labeled HA1 and HA2.

These ports will be physically connected back into the appliance, so we need another 2 ports for this. These ports will

use VXLAN, and in the case of an NP7 platform, there is benefit using the hardware acceleration for the VXLAN. For FG-

4200F we shall connect:

HA1 <-> AUX1

HA2 <-> AUX2

The original intention of the AUX ports are for hardware session synchronization for the

CGNAT use case, but in this case, they are being used as standard traffic ports.

Connectivity to the MNO's MPLS backbone routers is also required. In the FG-4200F case, assume 200Gbps aggregate

links to resilient routers:

Agg1 (port17 + port18) to PE1

Agg2 (port21 + port22) to PE2

Management port connectivity is handled as usual; however, note there is no floating management IP to consider.

The interfaces must be managed into VDOMs.

config system global

set vdom-mode multi-vdom

end

config vdom

edit intersite

end

config global

config system global

set management-vdom "intersite"

end

config system interface

edit "mgmt1"

set status down

edit "mgmt2"

set vdom "intersite"

set ip 10.40.100.1 255.255.255.0

set allow access ping https ssh snmp

edit "ha1"

set alias "HA1 root VDOM"

edit "ha2"

set alias "HA2 root VDOM"

edit "aux1"

set vdom "intersite"

set alias "HA1 intersite VDOM"

edit "aux2"

set vdom "intersite"

set alias "HA2 intersite VDOM"

Deployment Guide 45

CLUSTER CONNECTIVITY WITH VXLAN

edit "Agg1"

set vdom "root"

set type aggregate

set member "port17" "port18"

set description "SecGW Agg1 to PE1 BE1"

set lacp-speed fast

set min-links 2

set algorithm L3

edit "Agg2"

set vdom "root"

set type aggregate

set member "port21" "port22"

set description "SecGW Agg1 to PE2 BE1"

set lacp-speed fast

set min-links 2

set algorithm L3

end

Logical connectivity for HA

On top of each of the aggregate interfaces towards the MNO MPLS network, a VLAN interface is created. The simplest

deployment is to index the connected addresses into a dedicated FGCP VRF, which redistributes connected routes.

This then can be used with VXLAN to provide a layer 2 overlay between the appliance HA ports to produce an

architecture as shown in the following diagram:

Deployment Guide 46

CLUSTER CONNECTIVITY WITH VXLAN

config vdom

edit intersite

config system interface

edit "Agg1.1000"

set vdom "intersite"

(node0) set ip 100.0.1.2 255.255.255.252

(node1) set ip 100.0.5.2 255.255.255.252

set interface "Agg1"

set vlanid 1000

edit "Agg2.1000"

set vdom "intersite"

(node0) set ip 100.0.1.6 255.255.255.252

(node1) set ip 100.0.5.6 255.255.255.252

set interface "Agg2"

set vlanid 1000

end

config system vxlan

edit "vxlan-ha1"

set interface "aux1"

set vni 1600001

(node0) set remote-ip "100.0.5.2"

(node1) set remote-ip "100.0.1.2"

edit "vxlan-ha2"

set interface "aux2"

set vni 1600002

(node0) set remote-ip "100.0.5.6"

(mode1) set remote-ip "100.0.1.6"

end

config router static

edit 2

(node0) set dst 100.0.5.2 255.255.255.255

(node1) set dst 100.0.1.2 255.255.255.255

(node0) set gateway 100.0.1.1

(node1) set gateway 100.0.5.1

set device "Agg1.1000"

set bfd enable

edit 3

(node0) set dst 100.0.5.6 255.255.255.255

(node1) set dst 100.0.1.6 255.255.255.255

(node0) set gateway 100.0.1.5

(node1) set gateway 100.0.5.5

set device "Agg2.1000"

set bfd enable

end

The VXLAN configuration effectively connects AUX1 to AUX1 and AUX2 to AUX2 over the layer 3 network, which is as

good as connecting HA1 to HA1 and HA2 to HA2.

Assuming the MNO has deployed layer 3 routing between the sites, the FGCP cluster can now be built in the same way

as if the MNO has provided layer 2 connectivity between sites, with one difference:

config global

config system ha

set standalone-mgmt-vdom enable

Deployment Guide 47

FGCP WITH FGSP

end

This instructs FGCP that the "intersite" VDOM is not to be synchronized as per knowledge base item.

FGCP with FGSP

This architecture provides a means to have centralized locations, which provide a backup for each other should the

whole site fail, or as some MNOs might say mated sites.

It is also possible to use this in an anycast scenario, but this could be anti-intuitive in a MNO

router failure, which causes the firewall to failover, but the IGP cost sends traffic to a different

cluster.

This is a belt-and-braces approach that allows for multiple failures. It is however also an expensive arrangement as the

design must allow for a site failure such that all nodes within the FGCP+FGSP solution can take up the additional traffic.

MNOs may find this approach their preferred option, as it will give them a similar solution to those previously explained,

but with an additional benefit of a seamless site failover to the mated site.

This diagram shows that sites A and C are mated, and the two FGCP clusters swap session details through FGSP. In the

case of a site failure (or maintenance window), traffic is catered for without significant service impact.

To configure the actual FGSP on top of FGCP is very simple once you have FGCP sorted.

config global

config system cluster-sync

edit 1

set peerip 100.64.1.11

set syncvd "SecGW-Region1" "SecGW-Region2"

Deployment Guide 48

FGCP WITH FGSP

set down-intfs-before-sess-sync "Agg1.1001" "Agg1.2001"

set secondary-add-ike-routes disable

end

config system standalone-cluster

set standalone-group-id 200

set group-member-id 1

end

In this case, there are clustered interfaces (and FGCP IP addresses) and thus the IPsec would not terminate on a

loopback. There is also the need to have one FGSP peer as the cluster failure will provide for resiliency in this

connectivity.

As with standalone FGSP, the number of FGSP peers is limited and must be considered carefully with performance

impacts.

Deployment Guide 49

Security functions

The SecGW provides a critical security function in the MNO network, protecting the core services and functions from

attack originating in the RAN. Simply using IPsec alone is not sufficient in ensuring RAN-born threats and attackers are

mitigated.

Fortinet's security-first approach to SecGW sets the foundation for the SecGW use case with the FortiGate platform.

SCTP firewall

The use of an SCTP firewall is highly recommended for the SecGW use case. SCTP itself provides no inherent security

functions and is at risk to attack and vulnerability, such as described in RFC 5062. SCTP is critical to the service

operation as it is used to transport the control plane communications from the RAN to the core elements. An attack

directed at SCTP will have a direct impact on the control plane messages it carries.

The FortiGate platform will perform SCTP firewall inspection for SCTP traffic (which will carry the NG-AP/S1-AP/F1-AP

protocol messages). When configured, the FortiGate will inspect the SCTP traffic and perform the following checks:

Source and destination port

Verification tag

Chunk type, chunk flags and chunk length

Verify that association exists

Sequence of chunk types (INIT, INIT ACK, etc)

Timer checking

Four-way handshake checking

Heartbeat mechanism

NAT over SCTP

DoS protection against INIT/ACK flood DoS attacks and long-INIT flooding

DoS protection against association hijacking

The FortiGate does not have any predefined SCTP service definitions. Therefore you must configure your security policy

to use the ALL services option to enforce SCTP firewall inspection in a broad manor. The recommended solution is to

define an SCTP custom service and utilize this in the relevant security policy to ensure granularity and to deliver a strict

security check.

The following example shows how to define custom SCTP services that should be used to cover NG-AP, F1-AP, and S1-

AP control plane communications. Depending on the MNO architecture, SecGW placement, and service architecture,

only some of these will be applicable:

Deployment Guide 50

FIREWALL FLOW POLICY

config vdom

edit SecGW

config firewall service custom

edit "F1-AP"

set sctp-portrange 38472

edit "NG-AP"

set sctp-portrange 38412

edit "S1-AP"

set sctp-portrange 36412

end

These SCTP definitions are based on the IANA allocated destination port numbers.

Firewall flow policy

A firewall policy will be required to allow the traffic to flow from or to the IPsec tunnels that are established. The FortiGate

should enforce security control of flow traffic from the RAN to the core services. This can, and should, be as specific as

possible to ensure the most complete security control. An example of a strict Firewall Security policy is defined below:

The following example assumes that an administrator has already predefined firewall address

objects or groups for the source and destination elements and networks.

config vdom

edit SecGW

config firewall policy

edit 1

set name "AMF-Flows"

set srcintf "IPsecGW4"

set dstintf "SZ_RAN-TRUST"

set srcaddr "RAN-Nets"

set dstaddr "AMF-Group"

set action accept

set schedule "always"

set service "NG-AP"

set logtraffic all

set nat disable

edit 2

set name "UPF-Flows"

set srcintf "IPsecGW4"

set dstintf "SZ_RAN-TRUST"

set srcaddr "RAN-Nets"

set dstaddr "UPF-Group"

set action accept

set schedule "always"

set service "GTP-U"

set logtraffic all

set nat disable

end

Deployment Guide 51

S1-AP/NGAP MESSAGE FLOODING PROTECTION

end

Custom services should be defined that cover the specific protocols in use for these connections:

NG-AP using SCTP, destination port 38412. Used for control plane traffic from the gNB to the AMF.

GTP-U using UDP, destination port 2152. Used for user plane traffic from the gNB to the UPF.

These services may need to be defined as they are not part of the default services in FortiOS.

config vdom

edit SecGW

config firewall service custom

edit "NG-AP"

set sctp-portrange 38412

edit "GTP-U"

set udp-portrange 2152

end

With such specific security control, the SecGW will only allow SCTP and GTP-U to traverse to the core, and then only

once the IPsec tunnel has been successfully authenticated, authorized, and set up with the relevant integrity protection

and confidentiality required. And as such, the firewall will block any other types of IP communication from the RAN and

protect the Core elements from threats or attacks originating from the RAN.

The nature of a firewall is to block all connections, except those expressly defined as being

permitted. Therefore all other connectivity is by default blocked, ensuring a strong security

control function.

S1-AP/NGAP message flooding protection

In some scenarios, MNO requires that the mobile core infrastructure is protected from control plane message flooding.

FortiGate has built in features for SCTP security, DoS protection, and SCTP PPID filtering.

The FortiGate uses custom IPS rate-based signatures to provide S1-AP and NGAP message flooding protection.

Implementing this on the SecGW enables threats to be handled as far out in the network as possible, avoiding

transporting malicious traffic.

The custom signatures in this guide inspect the SCTP chunks matching on PPID values for S1-AP/NGAP and calculate

rates. When the specified threshold over the specified duration is reached, the specified action is taken. The action is

configured in the Intrusion Prevention profile.

Applying IPS inspection to a firewall policy affects the performance of FortiGate, so it is recommended for multiple

reasons, not covered here, to have separate firewall policies for control plane and user plane. This section only deals

with recommendations on S1-AP and NGAP in control plane.

S1-AP/NGAP message flooding configuration and testing on page 78 includes a step by step guide on how to

implement the custom signatures.

Deployment Guide 52

S1-AP/NGAP MESSAGE FLOODING PROTECTION

From FortiOS 7.0.1, PPID filtering is supported. This can be used to take action on specified PPIDs. For example, allow

PPID=18(S1-AP) in a firewall policy towards the MME, but block PPID=60(NGAP). This feature lacks rate limitation, so it

cannot be used to protect against message flooding.

S1-AP

The SCTP protocol allows multiple chunks in the same SCTP packet. Multiple signatures are created to cover scenarios

where the first chunk is a DATA chunk, but also when the first chunk is a Selective Acknowledgment (SACK)

subsequently followed by a DATA chunk.

The custom signatures use pattern matching on chunk type and PPID.

This signature match when the first chunk is type DATA=0 and PPID=18(S1-AP).

F-SBID( --name "SCTP.Client.Chunk.Data.PPID.S1AP.Rate.Custom"; --severity medium; --

protocol 132; --flow from_client; --pattern "|00|"; --context packet; --within 1,packet; --

pattern "|00 00 00 12|"; --distance 11; --within 4; --rate <COUNT>,<DURATION>;

Option-value pairs are explained in the table below.

Option Value

name Name of the signature and also displayed in logs

severity

Medium = S1-AP is a valid PPID, so no need to set it to high or critical. Medium matches well

with other SCTP signatures.

protocol 132 = SCTP

flow from_client = Match packets sent from the client to the server

pattern "|00|" = chunk type DATA

context packet = Searches for the pattern in the whole packet

within 1,packet = match one byte from the beginning of the packet

pattern

"|00 00 00 12|" = PPID field is four bytes. 12hex = 18dec = S1-AP. Network byte order for IPS

engine and SCTP is big-endian.

distance 11 = start pattern matching 11 bytes in the packet relatively from the previous match.

within 4 = match within four bytes. Same size at the pattern.

rate

<COUNT>,<DURATION>. Number of matches(COUNT) that must be seen over a period

(DURATION), in seconds, before a log entry is generated. Actual values has to be set

depending on the size of the network, similar to DoS thresholds. The DoS polices are seconds

based, so setting duration to 1 makes them comparable.

To match SCTP packets where the first chunk is a SACK and the subsequent chunk is DATA with PPID=18(S1-AP), the

following signature is created:

F-SBID( --name "SCTP.Client.Chunk.SACK.Chunk.Data.PPID.S1AP.Rate.Custom"; --severity

medium; --protocol 132; --flow from_client; --pattern "|03|"; --context packet; --within

1,packet; --byte_jump 2,1,relative; --pattern "|00|"; --distance -4; --within 1; --pattern

"|00 00 00 12|"; --distance 11; --within 4; --rate <COUNT>,<DURATION>;)

When there is a match for chunk type SACK, the counter jumps the length of the SACK chunk by reading the length field

using the byte_jump option. Notice how the next distance is negative, since the length field is two bytes long, and two

bytes into the SACK chunk the counter has to go 4 bytes backwards to find the second chunk type field.

Deployment Guide 53

DOS PROTECTION

NGAP

5G uses NGAP protocol on the N2 reference point with a PPID=60 (3C in hexadecimal).

To be able to count NGAP separately, a new custom signature is needed. The structure is the same as for the S1-AP

signatures, except the PPID to match:

F-SBID( --name "SCTP.Client.Chunk.Data.PPID.NGAP.Rate.Custom"; --severity medium; --

protocol 132; --flow from_client; --pattern "|00|"; --context packet; --within 1,packet; --

pattern "|00 00 00 3c|"; --distance 11; --within 4; --rate <COUNT>,<DURATION>;)

The same applies for the signature where the first chunk is a SACK, the only difference compared the to S1-AP signature

is the PPID value.

F-SBID( --name "SCTP.Client.Chunk.SACK.Chunk.Data.PPID.NGAP.Rate.Custom"; --severity

medium; --protocol 132; --flow from_client; --pattern "|03|"; --context packet; --within

1,packet; --byte_jump 2,1,relative; --pattern "|00|"; --distance -4; --within 1; --pattern

"|00 00 00 3c|"; --distance 11; --within 4; --rate <COUNT>,<DURATION>;)

DoS protection

The use of Denial of Service (DoS) filters is highly recommended. Importantly, DoS filters are applied on ingress before

decryption, and therefore do not inspect the encrypted traffic from the RAN endpoints.

However, the RAN presents a security threat towards the core. As such, DoS filters should be applied to any other traffic

attempting to traverse the SecGW towards the core. DoS filters are not a substitute for firewall policies. The DoS filters

will augment the firewall functions and will be hardware offloaded in FortiGate physical platforms, which is important.

DoS filters and the thresholds they observe will require careful planning, and will likely require tuning over time. The

SecGW provides a critical function in the network; therefore, consistent monitoring during the initial deployment as well

as part of any major network change is important. The following are some guidance areas for DoS Filters:

L3 anomalies on page 54

L4 anomalies on page 55

L3 anomalies

The following L3 anomalies will be applicable to any L3 IP connectivity from the RAN, including

the IPsec connections from the eNB/gNB peers.

ip_src_session:This DoS filter focuses on the number of concurrent IP connections from the same source address.

This has significant relevance as the SecGW should not be seeing large numbers of IP connections from the same

source address during normal operations. Each eNB/gNB will create a number of concurrent IP connections, but the

number overall should generally be small in normal. A large number of IP connections from the same source address

is a good indicator of an attacker, or mis-configured endpoint. An initial threshold of 1000 should be used here and

monitored for relevance and tuning, adjusting up or down as needed.

ip_dst_session: Should be enabled for monitoring only, as the SecGW is expected to see many IP connections to

the same destination address (the local tunnel gateway IP). The threshold value should match the number of

connecting peers (eNB/gNB) to a factor of 10 as a starting point. For example, if the number of connecting peers is

expected to be 5000, this threshold should be set at 50000. This should be monitored for relevance and tuning,

adjusting up or down as needed.

Deployment Guide 54

DOS PROTECTION

L4 anomalies

Traffic from the RAN endpoints will be encapsulated in ESP (with IPsec) or be UDP

connections used for IKEv2 communication. Any other L4 traffic seen by the FortiGate would

be considered anomalous here.

tcp_syn_flood, tcp_port_scan, tcp_src_session, tcp_dst_session: There should be no significant TCP-based

connectivity seen by the SecGW in general. As such all of these DoS filters should be enabled and set to block. The

same threshold value can be applied to all, and the recommendation is to set this at 1000. This should be monitored

for relevance and tuning, adjusting up or down as needed.

udp_flood, udp_scan, udp_src_session, udp_dst_session: Because IKEv2 uses UDP, we must be mindful of large-

scale reconnection attempts from the RAN that may occur in the event of a network service issue, network reroute,

or redundancy scenario. Therefore these parameters should be enabled for monitoring only, and the threshold value

set at the number of potential connecting peers (in the event of a failure scenario) to a factor of 10. It is important to

avoid that this DoS filter negatively impacts the ability for the SecGW to serve large numbers of connections from

the RAN when such a failure scenario occurs. This should be monitored for relevance and tuning, adjusting up or

down as needed.

icmp_flood, icmp_sweep, icmp_src_session, icmp_dst_session:ICMP traffic in general should not be experienced

in large volumes. Such activity is an indication of an attack. These DoS filters should be enabled for blocking, initially

using the default thresholds. This should be monitored for relevance and tuning, adjusting up or down as needed.

sctp_flood, sctp_scan, sctp_src_session, sctp_dst_session: In general operation, the SecGW should not see any

SCTP communications coming from the RAN that is not encapsulated in ESP (IPsec). For this reason, any SCTP

connections seen by the FortiGate are expected to be anomalous. These DoS filters should be enabled and set to

block, with a threshold of 1000 applied to all. This should be monitored for relevance and tuning, adjusting up or

down as needed.

Links:

DoS Protection

Deployment Guide 55

VDOM recommendations

It is recommended that VDOMs be used to separate management or logging functions from the traffic processing

functions of the SecGW. This essentially means defining two or more VDOMs on the system:

1. root VDOM for management and administration access to the system (CLI/SSH/Web-UI/REST API), as well as

logging (SYSLOG) and monitoring (SNMP)

2. traffic VDOM(s) for serving the main SecGW IPsec termination, firewall inspection, and routing functions

Additional VDOMs can be configured in scenarios where the MNO needs to logically split the platform into multiple

functions, for example providing SecGW for macrocell in one VDOM and another VDOM for microcell termination.

Generally, if the MNO has no specific need for a multi-VDOM capability, then only a single traffic processing VDOM is

used for all SecGW functions (plus the root VDOM for management), which provides the most simplistic solution whilst

retaining the management and traffic processing separation.

VDOM resources

VDOM resource controls for IPsec VPNs (Phase 1 tunnels, Phase 2 tunnels, and tunnel interfaces) can be used to limit the

resource allocation for a given VDOM. This can be useful in situations where multiple VDOMs are used for different

specific traffic processing tasks, such as the example explained previously. The VDOM resource controls can be used to

ensure that each VDOM has a guaranteed amount of IPsec VPN resources, being mindful of the overall system limits.

This document makes no specific recommendations on VDOM resource controls. As such configuration will be specific

to a given MNO customer scenario and requirement.

Deployment Guide 56

VRF recommendations

FortiOS supports virtual routing and forwarding (VRF). For FortiOS 7.0.x, up to 32 VRFs are supported in each VDOM and

more in Feature releases. For the scenarios in this section, only a few VRFs would be needed.

VRFs allow separation between multiple routing or forwarding domains. VRFs only have local presence and are not

exchanged between routers like route targets and route distinguishers.

In FortiOS, VRFs are configured by applying VRF ID to the interfaces participating in the desired VRF.

If a dynamic routing protocol, such as BGP, is used to exchange routes with the mobile core, it is recommended to hand

off traffic on multiple VLANs and use route-maps to limit the advertised and received routes to and from BGP neighbors.

One example where multiple VRFs can be relevant is the case where traffic is split into multiple tunnels, for example, for

management-plane, control-plane, and user-plane. In that case the tunnel interface and the associated clear text

interface(s) can be separated into their own VRFs.

Deployment Guide 57

Both the single-VRF and multi-VRF topology supports NPU offload.

Having a single tunnel interface with route leaking towards multiple VRFs before forwarding to mobile core is supported,

but not part of the reference design.

Route leaking in FortiGate uses inter-VDOM or NPU links to bridge traffic between VRFs. While it is hardware-offloaded

with NPU links, the performance between the two VRFs is then constrained by the NPU link performance.

Links:

Virtual routing and forwarding

Route leaking between VRFs with BGP

Deployment Guide 58

Traffic prioritization and QoS

This document does not intend to provide guidance on specific QoS configuration. Such information can only be derived

through the design phases of the project with the traffic prioritization capabilities of the FortiGate in mind. However,

relevance of QoS generally to the SecGW use case will be discussed in the following sections:

QoS and anti-replay on page 59

DSCP copy-down and inbound-dscp-copy for IPsec on page 59

QoS and anti-replay

Where traffic is marked for traffic prioritization, the use of anti-replay can be in conflict with the differing priorities of

packet flows and communications. For this reason, many MNOs must disable the use of anti-replay to ensure consistent

and unaffected traffic flows with QoS. Such requirements will be dictated by the MNO's network architecture and QoS

requirements.

DSCP copy-down and inbound-dscp-copy for IPsec

The term copy-down is widely used when describing the copying of any DSCP marking from an inner payload packet to

the outer ESP header used in IPsec. This allows the marking of the packet to be honored by the transporting network,

even when the payload is encrypted. The FortiGate will perform this action by default, as specified by the relevant RFC.

Traffic prioritization is generally applied on egress, copying of DSCP to the outer ESP header applies to the encrypt

direction (from the SecGW to the eNB/gNB endpoint). Traffic on the reverse path (decrypt path) will not need to have a

reverse copy from the outer ESP header to the inner payload, because the marking is already in place in the payload

packet. Once the ESP header is removed and the original packet is visible, it is processed as a normal clear text packet

and any associated traffic prioritization is applied on egress from the FortiGate towards the core. Essentially this is

thought of as an inner-to-outer copying process, although no copying actually takes place.

However, there may be situations where an MNO needs to rewrite the DSCP marker for the payload packet based on the

DSCP value of the ESP header. A good example may be when the MNO has a RAN share with another MNO. In such a

case, the originating eNB/gNB owned by the partner MNO may mark the packet with a specific DSCP value, but when

the packet arrives at the SecGW, the local MNO requires the DSCP value from the ESP header to be copied to the

payload packet header. Such functionality can be supported by the FortiGate system and it is called inbound-dscp-

copy.

To use the inbound-dscp-copy function, the following CLI parameter is needed:

Deployment Guide 59

DSCP COPY-DOWN AND INBOUND-DSCP-COPY FOR IPSEC

config vdom

edit SecGW

config vpn ipsec phase1-interface

edit "IPsecGW4"

set inbound-dscp-copy enable

end

config vdom

edit SecGW

config vpn ipsec phase2-interface

edit "IPsecGW4"

set inbound-dscp-copy enable

end

Deployment Guide 60

Management, orchestration, and API

MNOs have multiple ways to manage and orchestrate the SecGW solution. Flexibility is important to MNOs. The

FortiGate can be managed by a wealth of tools and protocols, such as SSH and web-based GUI, REST API, or

FortiManager.

For the most part, many MNOs traditionally look to an Element Management System (EMS) as part of any SecGW

solution, which for the Fortinet solution is fulfilled by FortiManager.

However, the solution is not constrained or restricted to using FortiManager. Utilizing other functions, such as REST API,

allows the FortiGate to be managed by other northbound management and orchestration systems, ultimately providing

flexibility.

FortiManager

FortiManager is the recommended management system for SecGW solution stack, providing many benefits for

centralized configuration and policy management. We will not go into any depth around the features and benefits of

FortiManager in this document.

The normal guidance applies: if the deployment is based on a small number of FortiGates (for example, 10 or less), then

the value that FortiManager brings is minimal. In such cases, direct management of the FortiGates using CLI or GUIis

generally more practical. For larger deployments, which is especially true for distributed architectures, FortiManager

brings significant operational value and should be factored into the solution.

FortiManager is arguably more important when managing an FGSP deployment. In this case, allowing the SecGWs to

share a policy package that can be pushed centrally may be extremely important especially when considering anycast. In

this case, it is worth considering FortiManager with a smaller number of FortiGates.

FortiManager JSON API

FortiManager includes a JSONAPI that can be used to manage both firewall policy packages and FortiGate device

configuration.

FortiManager's proxy API proxies APIrequests to FortiGate. This limits the number of management TCP ports that must

be opened towards the SecGW as the FGFM tunnel is used for API communication.

FortiManager 7.2.2 and later also supports API tokens, which streamlines the support for orchestration tools, such as

Terraform and Ansible.

Deployment Guide 61

FORTINET DEVELOPER NETWORK (FNDN)

Fortinet Developer Network (FNDN)

All Fortinet API documentation is available on FNDNat https://fndn.fortinet.net.

An account is required to access FNDN.

Links:

https://fndn.fortinet.net

Ansible

Both FortiOS and FortiManager support Ansible. Ansible galaxy collections can be installed directly like this: ansible-

galaxy collection install fortinet.fortimanager.

Documentation is available at FNDN.

Terraform

Fortinet has prepared both FortiOS and FortiManager Terraform providers.

Documentation is available at FNDN.

FortiOS REST API

The FortiOS REST API is a powerful and flexible way to administer the FortiGate system. API-based management of

systems has become one of the most popular, and preferred, methods for MNOs to manage the network equipment.

The REST API can be used to configure settings as well as gather information and statistics on activity. FortiOS REST API

is split into the following catagories:

Configuration API

Log API

Monitor API

For the SecGW use case, this section gives examples of relevant APIs.

Typically CMPv2 is used for certificate lifecycle management, but prior to the CMP Initialization Request (IR), some

certificates must be present on the SecGW, such as the CMP server certificate. This section also show examples on how

to import certificates on FortiOS by using the REST API.

Full REST API documentation is available at FNDN.

Admin profile, admin user, and token APIs on page 63

VPN configuration APIs on page 64

Log VPN APIs on page 65

Monitor VPN APIs on page 65

Deployment Guide 62

FORTIOS REST API

Admin profile, admin user, and token APIs

The FortiOS REST API uses token-based authentication as the preferred method. It is important to note that you must

define an administrator profile with sufficient privileges to conduct the level of access and administration required. Once

you have a defined administrator profile, you can create the REST API admin.

Following is an example of how to define a REST API admin:

1. Create an API admin profile:

config system accprofile

edit "super-api"

set scope global

set secfabgrp read-write

set ftviewgrp read-write

set authgrp read-write

set sysgrp read-write

set netgrp read-write

set loggrp read-write

set fwgrp read-write

set vpngrp read-write

set utmgrp read-write

set wanoptgrp read-write

set wifi read-write

end

2. Create API user:

config system api-user

edit "api-admin"

set comments "admin for API access only"

set api-key ENC {{key}}

set accprofile "super-api"

set vdom "root" "secgw"

config trusthost

edit 1

set ipv4-trusthost 192.168.10.0 255.255.255.0

end

3. Generate the API token:

execute api-user generate-key {{api-user-name}}

You can also generate the API token in the FortiOSGUIor through the REST API itself, if

required.

Deployment Guide 63

FORTIOS REST API

VPN configuration APIs

The configuration API allows CRUD operations of FortiOS Configuration Management DataBase (CMDB). Here are a few

of the relevant SecGW configuration APIs.

GET|PUT|POST|DELETE /api/v2/cmdb/vpn.ipsec/phase1-interface/

Configure phase-1, example:

curl -k -X POST 'https://{{ip}}:{{port}}/api/v2/cmdb/vpn.ipsec/phase1-interface?vdom=secgw'

--header 'Authorization: Bearer {{token}}' \

--data '{

{

"name": "secgwr1",

"type": "dynamic",

"interface": "Lo1-IPsecGW",

"ip-version": "4",

"ike-version": "2",

"local-gw": "100.64.1.1",

"certificate": [

{

"name": "secgw1",

}

"authmethod": "signature",

"peertype": "peer",

"peer": "mno_SubCA_peers",

"proposal": "aes256gcm-prfsha384 aes128gcm-prfsha256 ",

"dpd": "on-idle",

"dpd-retrycount": 3,

"dpd-retryinterval": "60",

"dhgrp": "21 32 31",

"fgsp-sync": "enable",

GET|PUT|POST|DELETE /api/v2/cmdb/vpn.ipsec/phase2-interface/

Configure phase-2, example:

curl -k -X POST 'https://{{ip}}:{{port}}/api/v2/cmdb/vpn.ipsec/phase2-interface?vdom=secgw'

--header 'Authorization: Bearer {{token}}' \

--data '{

"name": "secgwr1",

"phase1name": "secgwr1",

"proposal": "aes256gcm aes128gcm",

GET /api/v2/cmdb/vpn.certificate/ca

Get CA certificates, example:

curl -k -X GET 'https://{{ip}}:{{port}}/api/v2/cmdb/vpn.certificate/ca' \

--header 'Authorization: Bearer {{token}}'

GET /api/v2/cmdb/vpn.certificate/remote

Get remote certificates, example:

Deployment Guide 64

FORTIOS REST API

curl -k -X GET 'https://{{ip}}:{{port}}/api/v2/cmdb/vpn.certificate/remote' \

--header 'Authorization: Bearer {{token}}'

Log VPN APIs

The log API enables API access to the log framework in FortiOS. It is not intended to replace syslog or log-to-FAZ for

telco grade installations with intensive logging requirements. It can be useful for pulling filtered logs in an automation

loop, for example, in a situation where an event like a VPN tunnel is down.

Monitor VPN APIs

FortiOS' monitor API lets you gather information and statistics on FortiOS. Importing certificates is also part of the

monitor API.

Here are some of the SecGW relevant monitor APIs:

POST /api/v2/monitor/vpn/ike/clear

Clear IKE gateways, for example:

curl -k -X POST 'https://{{ip}}:{{port}}/api/v2/monitor/vpn/ike/clear?vdom=secgw' \

--header 'Authorization: Bearer {{token}}' \

--data '{

"mkey": "secgwr1_2"

POST /api/v2/monitor/vpn/ipsec/tunnel_reset_stats

Reset statistics, for example:

curl -k -X POST 'https://35.228.35.214:14001/api/v2/monitor/vpn/ipsec/tunnel_reset_

stats?vdom=secgw' \

--header 'Authorization: Bearer {{token}}' \

--data '{

"p1name": "secgwr1"

GET /api/v2/monitor/vpn/ipsec

Return an array of active IPsec VPNs.

This API returns statistics for all VPNtunnels, so be careful to request it for all tunnels as the response can be large.

Instead use filtering to only return statistics for a single tunnel.

Example filtering on CN in the peer certificate:

curl -k -X GET 'https://{{ip}}:{{port}}/api/v2/monitor/vpn/ipsec?vdom=SecGW-

R1&[email protected]' \

--header 'Authorization: Bearer {{token}}'

Example output:

...

"name":"IPsecGW6_11",

"parent":"IPsecGW6",

"comments":"",

"wizard-type":"custom",

"connection_count":1,

Deployment Guide 65

LOGGING

"creation_time":263,

"username":"C = CA, O = mno, OU = RAN, CN = enodeb2.mno.com",

"type":"dialup",

"incoming_bytes":26832,

"outgoing_bytes":26312,

"rgwy":"2001:470:6830::102:2000:12",

"tun_id":"10.0.0.4",

"tun_id6":"2001:470:6830::dead:cafe:2",

"dialup_index":17

}

...

POST /api/v2/monitor/vpn-certificate/ca/import

Importing certificates is also part of the monitor APIs. Here is an example of importing a CA certificate.

Notice the certificate must be in base64 format. Take the PEM formatted certificate and encode it like this in Linux:

$ base64 -i mno_CA.crt -o mno_CA.crt.64

Example:

% curl -k -X POST 'https://{{ip}}:{{port}}/api/v2/monitor/vpn-certificate/ca/import' \

--header 'Authorization: Bearer {{token}}' \

--data '{

"import_method": "file",

"scope": "global",

"file_content": "<your base64 encoded ca cert>"

POST /api/v2/monitor/vpn-certificate/remote/import

Importing a remote certificate is useful, for example, for importing the CMP server certificate.

Example:

curl -k -X POST 'https://{{ip}}:{{port}}/api/v2/monitor/vpn-certificate/remote/import' \

--header 'Authorization: Bearer {{token}}' \

--data '{

"import_method": "file",

"scope": "global",

"file_content": "<your base64 encoded remote cert>"

}

Links:

FNDN

Logging

When considering logging, the primary requirement here is event data from the IPsec functions. While you can perform

traffic logging of the flows passing through the system, it is somewhat unnecessary and quite an overhead. Furthermore,

the FortiGate will generally log the traffic as SCTP or UDP, and you will not gain any user plane visibility unless you are

using the IPS-based GTP-U inspection capabilities. Generally, if you are allowing the traffic, then it does not require

logging.

From an operational perspective, logging of event data from the FortiGate to identify possible issues or problems with

the IPsec functions, certificate authentication, routing problems, or general system-wide platform issues is the most

appropriate and valuable function.

Deployment Guide 66

FAULT AND PERFORMANCE MONITORING (FM/PM)

While FortiAnalyzer can be proposed for log collection, reporting, and some analytics, most MNO's already have logging

systems, and the predominant requirement is for the SecGW (FortiGate) to send log data to these existing systems,

based on common formats such as SYSLOG.

A complete FortiOSlog message reference is available.

Links:

FortiOS Log Message Reference

Fault and performance monitoring (FM/PM)

MNO's utilize Fault Monitoring and Performance Monitoring heavily throughout the network. The SecGW is an intrinsic

component of the network, and will also require the same level of FM/PM to be applied. This is generally achieved by

way of SNMP within the FortiGate platforms. The requirement here is to forward SNMP trap and event data to the MNO's

existing FM and PM systems, rather than enforcing the use of yet another tool or platform to perform this task.

The SNMP specific settings, protocol version, and levels of security will be dictated by the MNO. FortiGate can be

configured to send to multiple destinations and utilize secure forms of SNMP, such as SNMPv3. Generally, unless

otherwise specified by the MNO, SNMPv3 should be used.

config system snmp user

edit <index_number>

set security-level [auth-priv | auth-no-priv | no-auth-no-priv}

set queries enable

set query-port <port_number>

set notify-hosts <ip_address>

set events <event_selections>

end

When considering an HA cluster, each device in a cluster sends its own traps, and an SNMP manager can query both

devices. The use of dedicated HA management ports has to be enabled in the HA settings:

config system ha

set ha-mgmt-status enable

set ha-mgmt-interface “interface"

set ha-mgmt-interface-gateway x.x.x.x

end

And HA-direct specified in the SNMP config too:

config system snmp user

edit <index_number>

set ha-direct enable

end

You will also need to ensure the interface to which SNMP communications will be directed has SNMP enabled for

access:

config system interface

edit <interface_name>

set allowaccess snmp

end

Deployment Guide 67

IKE AND IPSEC MONITORING

Many MNOs also use APIs as part of FM/PM activities, so the use of the REST API access can also be considered for

FM/PM requirements too. However, be aware that some monitoring elements may not be exposed in the API, so this

cannot be the sole method of gathering FM/PM information.

IKE and IPsec monitoring

A number of key diagnostics commands can be used in FortiOS to monitor the IKE and IPsec activity, which are

especially useful during operational troubleshooting and interoperability testing.

This document will not cover exhaustive troubleshooting or diagnostics, merely some useful commands:

Interface statistics

get hardware nic <nic name>

Check link status, speed, and counters. Also pay attention to unexpected fragmentation.

IKE Packet sniffer (without NAT-T):

diagnose sniffer packet any 'udp port 500' 4

ESP Packet sniffer (without NAT-T):

diagnose sniffer packet any 'esp' 4

Flow debug, adjust the filters as needed:

diagnose debug enable

diagnose debug flow filter port 500

diagnose debug flow show function-name enable

diagnose debug flow trace start 100

IKE debug for more detailed diagnostics of negotiations:

diagnose debug enable

diagnose debug application ike -1

Using filters will help to isolate the specific information as this diagnose command can produce quite a busy output,

for example:

diagnose vpn ike log-filter dst-addr4 11.101.1.1.

Daemon IKE summary information:

diagnose vpn ike status

IPsec phase1 interface status:

diagnose vpn ike gateway list

IPsec phase2 tunnel status:

diagnose vpn tunnel list

Packets encrypted/decrypted counter:

diagnose vpn ipsec status

As usual remember to disable off-load when doing packet sniffer and debug.

Manually sync IPsec tunnels between FGSP peers. The execute sync-IPSec command can be relevant in

troubleshooting situations:

Links:

Troubleshooting Tip: Network Interface Card NIC commands

Technical Tip: Different methods to capture packets for IPsec VPN tunnels troubleshooting

Debugging the packet flow

Deployment Guide 68

IKE AND IPSEC MONITORING

Disabling NP offloading for individual IPsec VPN phase 1s

Technical Tip: How to sync sessions and IPSEC tunnels between the FGSP peers manually

NP7 IPsec troubleshooting commands

For the majority of issues it is sufficient to use the commands outlined in IKE and IPsec monitoring on page 68, but

sometimes it is not possible to disable hw-offload, or you just need to troubleshoot directly in NP7. It is well documented

in the Hardware Acceleration > NP7 diagnose commands. Here are a few examples relevant for SecGW:

NP7 Packet sniffer. This is handy for sniffing NP7 offloaded packets. Similar to the kernel packet sniffer:

diagnose npu sniffer filter

diagnose npu sniffer filter intf port23

diagnose npu sniffer filter dir 2

diagnose npu sniffer filter protocol 50

diagnose npu sniffer start

diagnose sniffer packet npudbg

NP7 performance monitor brief:

diagnose npu np7 pmon <npu_id> b

Check the overall IPsec utilization.

NP7 performance monitor verbose:

diagnose npu np7 pmon <npu_id> v

Check the IPsec per engine utilization and also L2P_IPS (Layer-2 processing IPSec module).

Drop counters in the Ingress Header Processing(XHP) sub-module in the IPsec module:

diagnose npu np7 dce-xhp-drop <npu_id> v

Drop counters related to the IPsec module:

diagnose npu np7 dce-ipsec-drop <npu_id> <action>

Links:

NP7 diagnose commands

NP7 packet sniffer

Deployment Guide 69

Conclusion

In this deployment guide we have demonstrated how to deploy FortiGate as a SecGW. We started with IPsec on page 7

and a description of how to configure IPsec and CMPv2.

We then described the various High availability on page 21 methods, such as FGCP and FGSP, and combinations of

them.

FortiGate is more than an IPsec concentration, so we showed some of the Security functions on page 50, which add

additional layers of security for a SecGW on the control-plane, such as SCTP firewalling, S1-AP/NGAP security, and DoS

protection.

Virtualization recommendations tells about how to utilize VDOMs and VRFs for a SecGW. See VDOM recommendations

on page 56 and VRF recommendations on page 57.

The QoS section describes how traffic prioritization works with QoS for a SecGW. See Traffic prioritization and QoS on

page 59.

Finally there is a section about how the SecGW solution can be managed and orchestrated with APIs, SNMP, and

FortiManager. See Management, orchestration, and API on page 61.

In the Appendices on page 71, you will find IPv6 configuration references, GTP-U payload inspection, S1-AP/NGAP

message flooding protection, and interoperability with strongSwan as VPN client.

It should be clear by now that FortiGate is an excellent SecGW, and it integrates well in a 3GPP MNO network.

Deployment Guide 70

Appendices

The following appendices provide further detail and references based on the topics and sections discussed throughout

this document:

IPv6 and SecGW configuration examples on page 71

GTP-U payload inspection on page 75

S1-AP/NGAP message flooding configuration and testing on page 78

S1-AP S1 setup request message flooding on page 84

Interoperability with strongSwan on page 85

IPv6 and SecGW configuration examples

Many MNOs utilize IPv6 throughput the network. In many cases, connectivity and configuration will be based on the use

of IPv6 over and above IPv4. The FortiGate supports both IPv4 and IPv6 for the SecGW use case.

This section provides some guidance on reference configuration elements where IPv6 specific elements are needed or

must be considered.

The following configuration stanzas build upon the configuration described in earlier sections. Of course IPv4 addressing

is not necessary on the interfaces if you have IPv6, but this document assumes running a dual stack.

VLAN interface on page 71

Loopback interface on page 72

BGP and referenced configuration on page 72

IPsec VPN on page 74

Firewall policy on page 75

FGSP synchronization on page 75

VLAN interface

For IPv4 we used /31, and equally we could used /127 ranges in IPv6. However, there shouldn't be such short supply of

address space, so there is no real need to go so tight! When the prefix is less than /127, do not use the first address in the

range as that is considered a built-in anycast address. (IPv6 does not have the concept of network ID, but it does use

this allocation for anycast.)

Deployment Guide 71

IPV6 AND SECGW CONFIGURATION EXAMPLES

config global

config system interface

edit "Agg1.1001"

config ipv6

set ip6-address 2001:470:6830::101:3002:e/124

end

Loopback interface

The loopback address can really be anything, but making it recognizable by perhaps encoding the region name or

number in it potentially makes troubleshooting easier. (The following one has something close to fortigateseg in it!)

config global

config system interface

edit "Lo1-IPsecGW"

config ipv6

set ip6-address 2001:470:6830:0:f047:16a7:e:5e6/128

end

BGP and referenced configuration

Note in the configuration that the IPv4 peering is explicitly disabled (similar to that discussed earlier about disabling IPv6

in the IPv4 configuration).

config vdom

edit SecGW

config router prefix-list6

edit "IPsecGW6-loopback-addresses"

config rule

edit 1

set prefix6 2001:470:6830:0:f047:16a7:e:5e6/128

unset ge

unset le

end

edit "Client-Inner-Addresses"

config rule

edit 1

set prefix6 2001:470:6830::dead:cafe:0/112

unset ge

set le 128

end

edit "LTE-IPv6-Addresses"

config rule

edit 1

set prefix 2001:470:6830::108:3003:0/124

unset ge

set le 128

end

Deployment Guide 72

IPV6 AND SECGW CONFIGURATION EXAMPLES

end

config router route-map

edit "RAN_TRUST-eBGP-Check"

config rule

edit 2

set match-ip6-address "LTE-IPv6-Addresses"

end

edit "IPsecGW-loopback-addresses"

config rule

edit 2

set match-ip6-address "IPsecGW6-loopback-addresses"

end

config router bgp

config neighbor

edit "2001:470:6830::101:3002:1"

set activate disable

set bfd enable

set soft-reconfiguration6 enable

set prefix-list-out6 "IPsecGW6-loopback-addresses"

set remote-as 65001

set route-map-in6 "set-lp-high"

set route-map-out6 "set-primary-community"

set update-source "Agg1.1001"

config conditional-advertise

edit "IPsecGW-loopback-addresses"

set condition-routemap "RAN_TRUST-eBGP-Check"

end

edit "2001:470:6830::101:3003:1"

set activate disable

set bfd enable

set soft-reconfiguration6 enable

set prefix-list-out6 "Client-Inner-Addresses"

set remote-as 65001

set route-map-in6 "set-lp-high"

set route-map-out6 "set-primary-community"

set update-source "Agg1.1002"

edit "2001:470:6830::102:3002:1"

set activate disable

set bfd enable

set soft-reconfiguration6 enable

set prefix-list-out6 "IPsecGW6-loopback-addresses"

set remote-as 65001

set route-map-in6 "set-lp-low"

set route-map-out6 "set-secondary-community"

set update-source "Agg2.1001"

config conditional-advertise

edit "IPsecGW-loopback-addresses"

set condition-routemap "RAN_TRUST-eBGP-Check"

end

Deployment Guide 73

IPV6 AND SECGW CONFIGURATION EXAMPLES

edit "2001:470:6830::102:3003:1"

set activate disable

set bfd enable

set soft-reconfiguration6 enable

set prefix-list-out6 "Client-Inner-Addresses"

set remote-as 65001

set route-map-in6 "set-lp-low"

set route-map-out6 "set-secondary-community"

set update-source "Agg2.1002"

end

config redistribute6 "connected"

set status enable

end

config redistribute6 "static"

set status enable

end

IPsec VPN

config vdom

edit SecGW

config firewall address6

edit "LTE-IPv6-Range"

set ip6 2001:470:6830::108:3003:0/124

end

config user peer

edit "SecGW6"

set ca "SEG-LAB_SEGCA"

end

config vpn ipsec phase1-interface

edit "IPsecGW6"

set type dynamic

set interface "Lo1-IPsecGW"

set ip-version 6

set ike-version 2

set local-gw6 2001:470:6830:0:f047:16a7:e:5e6

set authmethod signature

set net-device disable

set mode-cfg enable

set proposal aes256-sha256

set dhgrp 14

set certificate "PKI-SEG_CMP"

set peer "SecGW6"

set ipv6-start-ip 2001:470:6830::dead:cafe:1

set ipv6-end-ip 2001:470:6830::dead:cafe:ffff

set ipv6-prefix 112

set ipv6-split-include "LTE-IPv6-Range"

end

config vpn ipsec phase2-interface

edit "IPsecGW6"

set phase1name "IPsecGW6"

Deployment Guide 74

GTP-U PAYLOAD INSPECTION

set proposal aes256-sha256

set dhgrp 14

set src-addr-type subnet6

set dst-addr-type subnet6

end

Firewall policy

With later FortiOS releases, it is possible to have a policy that contains both IPv4 and IPv6. For this example, IPv4 and

IPv6 are kept separate.

config vdom

edit SecGW

config firewall policy

edit 60001

set srcintf "SZ_RAN-UNTRUST"

set dstintf "Lo1-IPsecGW"

set srcaddr6 "all"

set dstaddr6 "all"

set action accept

set schedule "always"

set service "IKE" "ESP"

set logtraffic disable

edit 60002

set srcintf "IPsecGW6"

set dstintf "SZ_RAN-TRUST"

set srcaddr6 "all"

set dstaddr6 "all"

set action accept

set schedule "always"

set service "ALL"

set logtraffic all

end

FGSP synchronization

FGSPcan only be configured using IPv4.

GTP-U payload inspection

FortiOS can decapsulate GTP-U packets and inspect the payload data, which applies to the S1-U/N3 logical

connections. The IPS Engine in FortiOS performs this operation on the clear-text traffic.

Using the IPS engine in this way allows the SecGW to be used to further inspect the original user plane data. Such

payload inspection can provide significant value to the MNO to be able to deliver two key security functions:

Deployment Guide 75

GTP-U PAYLOAD INSPECTION

Visibility: decapsulating the GTP-U data to reveal the original user plane payload offers significant value from a

visibility perspective by providing a view of the traffic types and applications in use. Such visibility is the foundation

for security, but also provides rich visibility of application use that can be valuable for traffic forecasting and

analytics.

Control:because the user plane payload is decapsulated, and the protocols and applications are seen, a control

decision can be made at this point. This can be valuable as traffic that should be dropped can be dropped, negating

the user plane having to be passed to the UPF/SGW/PGW to be processed to then be dropped on the sGi/N6

interface. A good example here is traffic destined to Botnet C&C servers. This is a more efficient and optimal use of

the core resources and controls attacks and threats closer to the origin at the UE.

There are however some specific considerations that should be taken into account to use this function, which will be

covered in the following sections:

Security policy on page 76

Logging on page 77

Performance considerations on page 77

GTP-U payload inspection is accomplished using the IPS Engine on the FortiGate. This means

that such a function requires IPS usage on the FortiGate and therefore an IPS subscription

license is needed.

Security policy

The GTP-U payload inspection is invoked by way of the security policy, and specifically using an IPS security profile

assigned to the policy, with the IPS Engine performing the GTP-U payload inspection itself. The IPS Engine effectively

strips away the GTP header and reveals the payload contained within the encapsulated packet. The IPS Engine makes

no specific checks on the GTP-U header. It is merely concerned with getting at the payload to perform its analysis. Once

the payload is identified and matched against the IPS sensor configuration, the packet is then actioned. If the packet

action is to block on match, then the packet is dropped with an optional log generated. If the packet action is to monitor,

it is forwarded and a log generated. And if the action is to allow, then the packet is forwarded with an optional log

generated. For the monitor and allow actions, the packet header that was previously stripped off by the IPS Engine

decoders is added back on to the packet again before being forwarded in its original form, unchanged.

The following example aims to inspect GTP-U payload data from IoT specific traffic from the RAN destined to the IoT

specific UPF/s, to identify IoT UEs trying to access known Botnet C&C destinations. The intention here is that such IoT

devices should not be trying to communicate with Botnet C&C destinations under normal operation, and such attempts

are positive indicators of compromise or infected IoT devices and should be investigated. Such information can prove

valuable to the MNOs customer and be offered as part of a value-added service. The IPS sensor will also scan any URL

seen in the payload and block access to known malicious sites based on the URL.

Blocking access to malicious URLs will not apply to encrypted HTTPS connections, as the

FortiGate cannot intercept the encrypted connection and analyze it.

Firstly we require an IPS Sensor which scans the traffic for connections to Botnet C&C destinations and malicious URLs:

config vdom

edit SecGW

config ips sensor

edit "IoT-block-CandC"

set scan-botnet-connections monitor

set block-malicious-url enabled

end

Deployment Guide 76

GTP-U PAYLOAD INSPECTION

end

Next, if a custom service for GTP-U has not already been defined as part of the normal SecGW security inspection

functions, then it needs to be defined with the explicit UDP port range used by GTP-U:

config vdom

edit SecGW

config firewall service custom

edit "GTP-U"

set udp-portrange 2152

end

Then we need to assign the IPS Sensor to a security policy which matches the flows between the RAN and the IoT

specific UPF/s:

config vdom

edit SecGW

config firewall policy

edit 3

set name "UPF-IoT-GTPU-Botnet"

set srcintf "IPsecGW4"

set dstintf "SZ_RAN-TRUST"

set srcaddr "RAN-Nets"

set dstaddr "UPF-IoT-Group"

set action accept

set schedule "always"

set service "GTP-U"

set logtraffic all

set ips-sensor "IoT-block-CandC"

set nat disable

end

We have used as specific a policy as possible here to focus on the IoT related flows only,

which assumes that IoT systems are terminating their user plane connections on specific

UPFs associated with the IoT network slice only.

Logging

The IPS sensor will log information in the UTM log of the FortiGate, while the traffic log is used for the logging associated

with the security policy. Importantly since FortiOS 6.4.2, the traffic log also records the TEID of the GTP packet. The

session ID can be used to correlate between the UTM log and Traffic log to identify the TEID of the particular session

that caused the log entry. Such correlation can be conducted using FortiAnalyzer. FortiSIEM can also be used, or logs

can be sent to a third-party logging and analytics system that may reside in the MNOs network. Logs sent to external

logging servers, including FortiSIEM, are sent as SYSLOG.

Performance considerations

When using GTP-U payload inspection, it is important to understand the performance impact on the FortiGate system.

As mentioned previously, this GTP-U payload inspection is performed by the IPS Engine. As such, every GTP-U packet

Deployment Guide 77

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

that matches (when enabled by policy) must be decapsulated and inspected by the IPS Engine, a security decision made

and log generated (if configured), and then re-encapsulated and forwarded on if the policy permits. This is an intensive

task, and adds significant overhead to the packet processing.

For this reason, GTP-U payload inspection should be used selectively and not in a blanket method. The performance

consideration is effectively the maximum IPS performance of the chosen FortiGate system in use. This IPS performance

will almost certainly be significantly lower than the associated IPsec performance. Which means that the overall

performance of the system is then gated on this IPS processing. Careful consideration should be made, and the

customer should be fully aware of these implications. If in any doubt, please consult the Carrier CSE Team for guidance.

S1-AP/NGAP message flooding configuration and testing

This appendix explains how to configure custom IPS signatures, Intrusion Prevention profile, and Firewall policies. It also

describes how to verify the configuration in logs and packet captures.

Custom IPS signatures on page 78

Intrusion Prevention profile on page 81

Firewall policy on page 82

Logs on page 83

Tuning thresholds on page 84

Custom IPS signatures

First create the custom IPS signatures shown in this document using either in the CLI or GUI. For your convenience, both

ways are shown.

See also the FortiOSAdministration Guide.

To add custom IPS signatures in the CLI:

1. In CLI paste this for the S1-AP basic signature.

config ips custom

edit "SCTP.Client.Chunk.Data.PPID.S1AP.Rate.Custom"

set signature "F-SBID( --name \"SCTP.Client.Chunk.Data.PPID.S1AP.Rate.Custom\"; -

-severity medium; --protocol 132; -- flow from_client; --pattern \"|00|\"; --context

packet; --within 1,packet; --pattern \"|00 00 00 12|\"; --distance 11; --within 4; --

rate 10,1;)"

end

2. Then similar for the signature when the first chunk is a SACK.

config ips custom

edit "SCTP.Client.Chunk.SACK.Chunk.Data.PPID.S1AP.Rate.Custom"

set signature "F-SBID(--name

\"SCTP.Client.Chunk.SACK.Chunk.Data.PPID.S1AP.Rate.Custom\"; --severity medium; --

protocol 132; --flow from_client; --pattern \"|03|\"; --context packet; --within

1,packet; --byte_jump 2,1,relative; --pattern \"|00|\"; --distance -4; --within 1; --

pattern \"|00 00 00 12|\"; --distance 11; --within 4; --rate 10,1;)"

end

Deployment Guide 78

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

To add custom IPS signatures in the GUI:

1. In GUI, go to Security Profiles > IPS Signatures > Create New.

2. Paste the signature from this guide: copy and paste the --name ... value in the Name field.

After the signature has been created, FortiGate assigns an available attack_id.

3. Use the same method for the signature when the first chunk is a SACK.

When both signatures are successfully created, they should display under Custom IPS Signature in the list of IPS

Signatures as shown in the following screen:

Deployment Guide 79

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

The attack_id is visible when hovering your mouse over the signature in GUI.

The attack_id is also visible in the GUIwhen you double-click on a custom signature after is has been created.

Deployment Guide 80

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

In the CLIthe attack_id can be found using this show command:

FortiGate # show ips custom

config ips custom

edit "SCTP.Client.Chunk.Data.PPID.S1AP.Rate.Custom"

set signature "F-SBID( --attack_id 7549; --name

\"SCTP.Client.Chunk.Data.PPID.S1AP.Rate.Custom\"; --severity medium; --protocol 132; --

flow from_client; --pattern \"|00|\"; --context packet; --within 1,packet;--pattern

\"|00 00 00 12|\"; --distance 11; --within 4; --rate 10,1;)"

set comment ''

edit "SCTP.Client.Chunk.SACK.Chunk.Data.PPID.S1AP.Rate.Custom"

set signature "F-SBID( --attack_id 2983; --name

\"SCTP.Client.Chunk.SACK.Chunk.Data.PPID.S1AP.Rate.Custom\"; --severity medium; --

protocol 132; --flow from_client; --pattern \"|03|\"; --context packet; --within

1,packet; --byte_jump 2,1,relative; --pattern \"|00|\"; --distance -4; --within 1; --

pattern \"|00 00 00 12|\"; --distance 11; --within 4; --rate 10,1;)"

set comment ''

end

Because the FortiGate dynamically allocates the attack_id, it is not the same on all FortiGates. Create the signature

without any attack_id and let FortiGate allocate the ID.

Intrusion Prevention profile

Create an Intrusion Prevention profile which can be applied to Firewall Policies.

To create an intrusion prevention profile in the CLI:

config ips sensor

edit "SCTP S1-AP Security"

set scan-botnet-connections monitor

config entries

edit 1

set rule <attack_id>

set status enable

set action pass

edit 2

set rule <attack_id>

set status enable

set action pass

end

Replace <attack_id> with the attack_ids FortiGate dynamically allocated.

To create an intrusion prevention profile in the GUI:

1. Go to Security Profiles > Intrusion Prevention > Create New.

2. Add the custom signatures one by one as shown in the following screen.

Deployment Guide 81

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

3. When adding a custom IPS Signature to an Intrusion Prevention profile, the action has multiple choices. It is

recommended to set it to Monitor at a minimum, so log entries are generated when the signature is triggered.

The custom signature rate-based settings can be overridden in GUI or CLI. This is useful for setting two different

thresholds for monitor and block, but still retain a single custom signature.

Firewall policy

Apply the Intrusion Prevention Profile to a Firewall Policy.

Deployment Guide 82

S1-AP/NGAP MESSAGE FLOODING CONFIGURATION AND TESTING

Logs

When the custom signature is triggered with action set to monitor, a log entry is created. You can inspect the log entry in

the GUI under Log & Report > Intrusion Prevention.

In the following example, FortiGate has detected a number of SCTP packets where the first chunk is S1-AP and the

signature is triggered.

A packet capture of the packet that triggered the signature. The PPID field in the DATA chunk is highlighted.

In the following example, FortiGate has detected a number of SCTP packets where the first chunk is SACK, and the

subsequent chunk is DATA with PPID=18(S1-AP).

Deployment Guide 83

S1-AP S1 SETUP REQUEST MESSAGE FLOODING

A packet capture of the packet that triggered the signature. The first chunk is SACK and the subsequent chunk is DATA.

The PPID field in the DATA chunk is highlighted.

Tuning thresholds

When deploying this in a production network, the threshold values should be adjusted to minimize the amount of false

positives. It is recommended as a first step to set the action to monitor to learn the amount of messages in a normal

scenario, and then adjust the threshold after.

Blocking SCTP messages protects the mobile core elements, but remember discarding SCTP packets means that the

sender will retransmit. This is important to keep in mind when troubleshooting a flooding attack.

S1-AP S1 setup request message flooding

In case of a geographically larger service outage, it is relevant to protect the MME from a signaling storm. One method to

mitigate this is to rate limit the S1 setup procedure, which is the first S1AP procedure initiated by the eNodeB. Applying

Deployment Guide 84

INTEROPERABILITY WITH STRONGSWAN

rate limitation on S1SetupRequests from eNodeB implies FortiGate drops the request, such that the eNodeB will retry.

This will smooth out the peak load that an MME can experience during a outage of many eNodeBs.

A custom IPS signature to match on S1SetupRequests from client side (eNodeB) has been developed and can be

configured similar to other custom IPS signatures in this guide.

F-SBID( --attack_id 9236; --name

"SCTP.Client.Chunk.Data.PPID.S1AP.S1SetupRequest.Rate.Custom"; --severity medium; --

protocol 132; --flow from_client; --pattern "|00|"; --context packet; --within 1,packet; -

-pattern "|00 00 00 12|"; --distance 11; --within 4; --pattern "|00 11|"; --within 2; --

rate <COUNT>,<DURATION>;)

Option-value pairs are explained in the table below.

Option Value

name Name of the signature, also displayed in logs

severity

Medium = S1-AP is a valid PPID. No need to set it to high or critical. Medium matches well with

other SCTP signatures.

protocol 132 = SCTP

flow from_client = Match packets sent from the client to the server

pattern "|00|" = chunk type DATA

context packet = Searches for the pattern in the entire packet

within 1,packet = match one byte from the beginning of the packet

pattern

"|00 00 00 12|" = PPID field is four bytes. 12hex = 18dec = S1-AP. Network byte order for IPS

engine and SCTP is big-endian.

distance 11 = start pattern matching 11 bytes in the packet relatively from the previous match

within 4 = match within four bytes. Same size at the pattern.

pattern "|00 11|" = 00 is S1AP-PDU InitiatingMessage, 0x11 = 17dec is ProcedureCode for id-S1Setup

within

2 = match within two bytes. Same size at the pattern. No distance key word needed as it is

next two bytes.

rate

<COUNT>,<DURATION>. Number of matches(COUNT) that must be seen over a period

(DURATION) in seconds before a log entry is generated. Actual values must be set depending

on the size of the network, similar to DoS thresholds. The DoS polices are seconds based, so

setting duration to 1 makes them comparable.

According to 3GPP TS 36.413, the S1 Setup is the first S1AP procedure to be triggered after the TNL association has

become operational; hence, no signature is required where the first SCTP chunk is a SACK.

Interoperability with strongSwan

It is important for the SecGW to work with multiple base station vendors. Several of the major base station vendors use

the Open Source IPsec implementation strongSwan inside their base station SW. StrongSwan is an open source multi-

platform IPsec implementation. It fully supports IKEv1, IKEv2, and X.509 certificates.

Fortinet has conducted interoperability tests with strongSwan and demonstrate that the Fortinet SecGW solution can

fully interoperate with strongSwan. Our Solution Validation document can be shared on request.

Deployment Guide 85

www.fortinet.com

Copyright© 2024 Fortinet, Inc. All rights reserved. Fortinet®, FortiGate®, FortiCare® and FortiGuard®, and certain other marks are registered trademarks of Fortinet, Inc., and other Fortinet names herein may also be registered and/or common law trademarks of Fortinet.

All other product orcompany names may be trademarks of their respective owners. Performance and other metrics contained herein were attained in internal lab tests under ideal conditions, and actual performance and other results may vary. Network variables,

different network environments and other conditions may affect performance results. Nothing herein represents any binding commitment by Fortinet, and Fortinet disclaims all warranties, whether express or implied, except to the extent Fortinet enters a binding written

contract, signed by Fortinet’s SVP Legal and above, with a purchaser that expressly warrants that the identified product will perform according to certain expressly-identified performance metrics and, in such event, only the specific performance metrics expressly

identified in such binding written contract shall be binding on Fortinet. For absolute clarity, any such warranty will be limited to performance in the same ideal conditions asin Fortinet’s internal lab tests. Fortinet disclaims in full any covenants, representations, and

guarantees pursuant hereto, whether express or implied. Fortinet reserves the right to change, modify, transfer, or otherwise revise this publication without notice, and the most current version of the publication shall be applicable.

01-700-941920-20230925