CLI Commands for Troubleshooting FortiGate Firewalls
This blog post is a list of common troubleshooting commands I find usefull in my daily work. It is not complete nor very detailled, but provides the basic commands for troubleshooting network related issues that are not resolvable via the GUI. I am not focused on too many memory, process, kernel, etc. details. These must only be used if there are really specific problems. I am more focused on the general troubleshooting stuff. I am using it personally as a cheat sheet / quick reference and will update it from time to time.
# fnsysctl
# MV
usage: mv [-f] src dst
# GREP
Usage: grep [-ilHhnqvscABC] PATTERN [FILE...]
Options:
-i Ignore case distinctions
-l List names of files that match
-H Prefix output lines with filename where match was found
-h Suppress the prefixing filename on output
-n Print line number with output lines
-q Quiet
-v Select non-matching lines
-s Suppress file open/read error messages
-c Only print count of matching lines
-A Print NUM lines of trailing context
-B Print NUM lines of leading context
-C Print NUM lines of output context
# KILL
# KILLALL
# LS
# CAT
usage: cat [file ...]
# IFCONFIG
# PWD
# DU
usage: du
Summarize disk space used for each FILE and/or directory
-a Show file sizes too
-L Follow all symlinks
-H Follow symlinks on command line
-d N Limit output to directories (and files with -a) of depth < N
-c Show grand total
-l Count sizes many times if hard linked
-s Display only a total for each argument
-x Skip directories on different filesystems
-i Show number of inodes
# PS
# DATE
# SH
# DF
In Cisco, everything is “show”. With Fortinet, you have the choice between show | get | diagnose | execute.
Likewise the sys | system keyword. It is always “diagnose sys” but “execute system”
| Fortigate Cisco | |
| show full-configuration show run | |
| execute factory-reset write erase | |
| show system interface show run interface brief | |
| diagnose hardware deviceinfo nic show interface | |
| get system status show version | |
| get system arp | diagnose ip arp list show arp | |
| get router info routing-table all show ip route | |
| diagnose system session list show ip nat translation | |
| diagnose system session clear clear ip nat translation | |
| get router info ospf neighbor show ip ospf neighbor | |
| get router info bgp neighbor show ip bgp neighbor | |
| get router info bgp summary show ip bgp summary |
Entering the correct vdom/gobal Config
Remember to enter the correct vdom or global configuration tree before configuring anything:
config global
config vdom
edit <vdom-name>
To show the running configuration (such as “show run”), simply type in:
showTo show the entire running configuration with default values, use:
show full-configuration
To omit the “–More–” stops when displaying many lines, you can set the terminal output to the following, which will display all lines at once. This is similar to “terminal length 0” from Cisco. Be careful with it, because this command is persistent. Set it to default after usage!
config system console
set output standard
end
To find a CLI command within the configuration, you can use the pipe sign “|” with “grep” (similar to “include” on Cisco devices). Note the “-f” flag to show the whole config tree in which the keywords was found, e.g.:
show | grep -f ipv6
show full-configuration | grep -f ipv6
General Information
get system interface physical #overview of hardware interfaces
get hardware nic <nic-name> #details of a single network interface, same as: diagnose hardware deviceinfo nic <nic-name>
fnsysctl ifconfig <nic-name> #kind of hidden command to see more interface stats such as errors
get system status #show version
get system performance status #CPU and network usage
diagnose sys top #top with all forked processed
diagnose sys top-summary #top easier, incl. CPU and mem bars. Forks are displayed by [x13] or whatever
execute dhcp lease-list
diagnose ip arp list
diagnose ipv6 address list
diagnose ipv6 neighbor-cache list
diagnose sys ntp status
diagnose autoupdate versions #lists the attack definition versions, last update, etc.
diagnose log test #generated all possibe log entries
diagnose test application dnsproxy 6 #shows the IP addresses of FQDN objects
diagnose debug crashlog read #shows crashlog, a status of 0 indicates a normal close of a process!
After rebooting a fresh device which is already licensed, it takes some time until it is “green” at the dashboard. The following commands can troubleshoot and start the “get license” process. Use the first three to enable debugging and start the process, while the last one disables the debugging again:
diag debug app update -1
diag debug enable
exec update-now
diag debug disable
General Network Troubleshooting
execute ping-options ?
execute ping-options source <ip-address-of-the-interface>
execute ping <hostname|ip>
execute ping6-options ?
execute ping6 <hostname|ip>
execute traceroute <hostname|ip>
execute tracert6 <hostname|ip>
get router info routing-table all #routing table
get router info6 routing-table #IPv6 without the "all" keyword
get router info kernel #Forwarding Information Base
get router info6 kernel
get router <routing-protocol> #basic information about the enabled routing protocol
diagnose firewall proute list #policy-based routing
diagnose firewall proute6 list
diagnose ip rtcache list #route cache = current sessions w/ routing information
HA
diagnose sys ha status
execute ha manage ? #switch to the CLI of a secondary unit
execute ha manage <device-index>
diagnose sys ha showcsum #verify the checksum of all synchronized peers
get system session list #rough view with NAT, only IPv4
Session Table
diagnose sys session filter clear
diagnose sys session filter ?
diagnose sys session filter dst 8.8.8.8
diagnose sys session filter dport 53
diagnose sys session list #show the session table with the filter just set
1#diagnose debug disable
2#diagnose debug flow trace stop
3#diagnose debug flow filter clear
4#diagnose debug reset
5#diagnose debug flow filter addr x.x.x.x
6#diagnose debug flow show console enable
7#diagnose debug flow show function-name enable
8#diagnose debug console timestamp enable
9#diagnose debug flow trace start 100
10#diagnose debug enable
View specific traffic session with:
#diag sys session list
or
#diag sys session filter sport:dport:dst:src:proto:policy
View per-ethernet frame operations:
#diag sniffer packet
View object dependencies:
# diagnose sys checkused firewall.address:name 'SSLVPN_TUNNEL_ADDR1'
# diagnose sys checkused firewall.address:name 'iPhoneVPN_range'
# diagnose sys checkused firewall.address:name 'Local LAN'
# diagnose sys checkused system.interface:name <interface name>
Debug
Sniffer - URL https://gateIP/ng/page/p/firewall/sniffer/
Sniff packets like tcpdump does. This can be used for investigating connection problems between two hosts. There are no details of the firewall policy decisions. Use the debug flow (next paragraph) for analysis about firewall policies, etc.
1
|
diagnose sniffer packet <interface|any> '<tcpdump-filter>' <verbose> <count> <time-format>
|
verbose:
1: print header of packets
2: print header and data from ip of packets
3: print header and data from ethernet of packets (if available)
4: print header of packets with interface name
5: print header and data from ip of packets with interface name
6: print header and data from ethernet of packets (if available) with intf name
count: number of packets
time-format:
a: UTC time
l: local time
Examples: (Thanks to the comment from Ulrich for the IPv6 example)
1
2
3
4
|
diagnose sniffer packet any 'host 8.8.8.8' 4 4 l
diagnose sniffer packet any 'host 8.8.8.8 and dst port 53' 4 10 a
diagnose sniffer packet wan1 'dst port (80 or 443)' 2 50 l
diagnose sniffer packet any 'net 2001:db8::/32' 6 1000 l
|
LLDP:diagnose sniffer packet port1 'ether proto 0x88cc' 4 1 a
CDP: diagnose sniffer packet port1 'ether[20:2] == 0x2000' 6 1 a
|
https://kb.fortinet.com/kb/documentLink.do?externalID=11186&languageId
Flow
If you want to see the FortiGate details about a connection, use this kind of debug. E.g., it shows the routing decision and the policy, which allowed the connection.
1
2
3
4
5
6
7
8
|
diagnose debug reset
diagnose debug flow filter ?
diagnose debug flow filter saddr 172.16.27.148
diagnose debug flow filter daddr 8.8.8.8
diagnose debug flow show console enable
diagnose debug enable
diagnose debug flow trace start 10 #display the next 10 packets, after that, disable the flow:
diagnose debug disable
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
fd-wv-fw04 # diagnose debug reset
fd-wv-fw04 # diagnose debug flow filter daddr 8.8.8.8
fd-wv-fw04 # diagnose debug flow show console enable
show trace messages on console
fd-wv-fw04 # diagnose debug enable
fd-wv-fw04 # diagnose debug flow trace start 20
id=20085 trace_id=11 func=print_pkt_detail line=4420 msg="vd-root received a packet(proto=17, 192.168.160.10:55859->8.8.8.8:53) from fg-trust. "
id=20085 trace_id=11 func=init_ip_session_common line=4569 msg="allocate a new session-0001f004"
id=20085 trace_id=11 func=vf_ip4_route_input line=1596 msg="find a route: flags=00000000 gw-172.16.1.1 via wan1"
id=20085 trace_id=11 func=fw_forward_handler line=671 msg="Allowed by Policy-16:"
id=20085 trace_id=11 func=__ip_session_run_tuple line=2601 msg="run helper-dns-udp(dir=original)"
id=20085 trace_id=12 func=print_pkt_detail line=4420 msg="vd-root received a packet(proto=17, 192.168.160.10:63624->8.8.8.8:53) from fg-trust. "
id=20085 trace_id=12 func=init_ip_session_common line=4569 msg="allocate a new session-0001f005"
id=20085 trace_id=12 func=vf_ip4_route_input line=1596 msg="find a route: flags=00000000 gw-172.16.1.1 via wan1"
id=20085 trace_id=12 func=fw_forward_handler line=671 msg="Allowed by Policy-16:"
id=20085 trace_id=12 func=__ip_session_run_tuple line=2601 msg="run helper-dns-udp(dir=original)"
fd-wv-fw04 # diagnose debug disable
|
VPN
To show details about IKE/IPsec connections, use these commands:
1
2
3
4
5
6
|
get vpn ike gateway <name>
get vpn ipsec tunnel name <name>
get vpn ipsec tunnel details
diagnose vpn tunnel list
diagnose vpn ipsec status #shows all crypto devices with counters that are used by the VPN
get router info routing-table all
|
1
2
3
4
5
6
7
|
diagnose debug reset
diagnose vpn ike log-filter clear
diagnose vpn ike log-filter ?
diagnose vpn ike log-filter dst-addr4 1.2.3.4
diagnose debug app ike 255 #shows phase 1 and phase 2 output
diagnose debug enable #after enough output, disable the debug:
diagnose debug disable
|
diag vpn tunnel reset <phase1 name>
To bring up particular Phase 2 use: to list diag vpn tunnel list and then run execute vpn ipsec tunnel up vpn-name-P2 vpn-name-P1 4 ( 4 is the serial of the P2 tunnel in my case |
Log
For investigating the log entries (similar to the GUI), use the following filters, etc.:
1
2
3
4
5
6
7
|
execute log filter reset
execute log filter category event
execute log filter field #press enter for options
execute log filter field dstport 8001
execute log filter view-lines 1000
execute log filter start-line 1
execute log display
|
Defaults
Just a reminder for myself:- IP: 192.168.1.99
- Login: admin
- Password: <blank>
1
2
3
4
5
6
|
config system interface
edit mgmt
set ip 192.168.1.1 255.255.255.0
set allowaccess ping https ssh
next
end
|
Options:
application - Application
cli - debug cli
console
crashlog - crash log info
disable
flow - trace packet flow
haproxy
info - show active debug level settings
report - report for tech support
reset - reset all debug level to default
Capture TCP packets with SYN flag set on
# diagnose sniffer packet any 'port 80 and tcp[13]&2==2' 4
// Health and Status
show [enter] //Note that output is only non-default values.
show full-configuration // Show all configurations on the device.
show system interface wan1 | grep -A2 ip // Show WAN and interface information.
get system info admin status // Show logged in users
get system status // Show system hardware/software update versions
get hardware status // Detailed hardware model information
get system performance status
get system performance top
show system interface // Interface Configuration
DEBUG
diagnose hardware deviceinfo nic // Interface Statistics/Settings
diagnose hardware sysinfo memory
diag debug crashlog read
diag hardware sysinfo shm // Device should be in 0, if (>0) then conserve mode
get system global | grep -i timer // Show tcp and udp timers for half open and idle
get system session-ttl // System default tcp-idle session timeout
execute ha manage <devid> // send heartbeat across management link.
get hardware nic
diagnose ip address list
get system interface physical // ARP
diagnose ip arp list // Track and Troubleshoot
get system session status // Connection count for ingress/egress
get system session-info full-stat // Displays session status with breakdown by state
get system session list // Session list, protocol, expire, src nat, dst nat
diag sys session // Basic output with no filters of diag sys session
diag sys session filter <option> <value> // Capture filter based on src, dst, duraction, policy id, vd // Packet capture
diag debug info // Displays active debug
diag debug enable // Enable debug
#diagnose debug flow filter (shows what filters are configured)
#diagnose debug flow filter clear (clear all filter)
#diagnose debug flow filter <options> <value> (configures the filter)
#diagnose debug flow show con enable <show output on console>
#diagnose debug flow show fun enable <show functions>
#diagnose debug flow trace start <number of lines> (to start the trace)
#diagnose debug flow trace stop (to stop the trace)
Example:
diagnose debug reset
diagnose debug enable
diagnose debug flow filter clear
diagnose debug flow filter saddr 192.168.10.1
diagnose debug flow filter dport 80
diagnose debug flow show con enable
diagnose debug flow show fun enable
diagnose debug flow trace start 20
diagnose sniffer packet <interface or ANY> ‘<arguments>’ <level 1-6>
example:
diagnose sniffer packet ANY ‘net 192.168.10.0/24 and not host 192.168.10.1 and port 80 and TCP’ 6
Syn packets only:
diag sniffer packet internal ‘tcp[13] == 2′
Enable packet capture in GUI - System -> Config -> Advanced
Setup packet capture filter, Check box to start, Uncheck to stop.
Download Debug Log
// Show identified devices
diag user device list
Routing
diagnose ip rtcache list // rt cache table
Interface Up -> Multiple: Select lowest distance -> Dynamic: If same distance choose lowest metric -> Dynamic: If multiple have same distance/metric, depends on protocol -> All “Best Routes” places in table. Match goes to most exact subnet -> Policy routing applied before table lookups.
Route lookups are only for the first packet of each session.
All packets will use same path.
After topology change, routes are flushed and sessions relearned.
get system arp // ARP Table
get router info routing-table all // All routing table entries
get router info routing-table details <ip> // Shows if custom static or dynamic routes exist for dest.
get router info kernel // Raw kernel routing table
show router static // Display static routes
show router policy //
Restore image
execute restore image <firmware_file_name> <TFTP server_ipaddress> // Restore an image from TFTP
Provisioning
config system settings // Configure for layer-3
set opmode nat
end
config system settings // Configure transparent
set opmode transparent
end
config system global // Set port for admin if VPN is sharing
set admin-sport 8443
set sslvpn-sport 443
end
config system global // Enable SCP
set admin-scp enable
cofng system ntp // Setup NTP
config ntpserver
edit 1
set server 10.0.0.0
end
edit 2
set server 10.0.0.1
end
set ntpsync enable
end
execute time
config system dns // Setup DNS
set primary 0.0.0.0
set secondary 0.0.00
config log syslogd(2|3) setting // Enable syslog
set status enable
set server <IP address>
set port 514
set facility user
end
diagnose log test // Test logging
config system interface // Setup IP Address
edit wan1
set mode static
set ip 172.16.0.0 255.255.255.0
set vlan id 50
end
config system interface // LACP port aggregation
edit aggr1
set member “port8″ “port9″
end
config system zone // Add interfaces to zone
edit outside
set interface internal1 internal 2
enable intrazone traffic
set intrazone allow
end
config router static // Add default route
edit 1
set gateway 172.16.0.0
end
config router static // Static route
edit2
set device port1
set dst 10.0.0.0 255.0.0.0
set gateway 10.0.1.1
// Vendor Notes
http://docs.fortinet.com/fgt.html
http://docs.fortinet.com/fgt/handbook/40mr3/fortigate-cli-40-mr3.pdf
http://docs.fortinet.com/fgt50.html
