Fortigate – Very high CPU utilization usage after up-gradation of Fortigate OS 6.2.2

Firmware upgrade from version 6.0.6 to 6.2.2
(Latest version is 6.2.2, N-1 is 6.2.1 –Why we upgrade 6.2.2 ? because as per Fortinet advises customers to upgrade to FortiOS 5.4.13, 5.6.11, 6.0.6 or 6.2.2 or above and found the below vulnerabilities.

“ SSL VPN VULNERABILITIES : Security vulnerabilities discussed at the BlackHat 2019 conference

At the recent Black Hat 2019 conference held in Las Vegas August 3-8, security researchers discussed their discovery of security vulnerabilities that impacted several security vendors, including Fortinet. All of the vulnerabilities impacting Fortinet were fixed in April and May of 2019. FortiOS 5.4.13*, 5.6.11, 6.0.6 or 6.2.2 are recommended.”

More Read https://kb.fortinet.com/kb/microsites/microsite.do?cmd=displayKC&docType=kc&externalId=FD46513

We have upgraded successfully from FortiOS 6.0.6 to 6.2.2 and after some time we observed the CUP utilization goes very high unexpected.

It may be the BUG for the high CPU process.

Perform below command output from the Fortinet CLI

CLI# diagnose sys top

CLI# diagnose sys top-summary

OUTPUT

# diagnose sys top-summary

[H[JRun Time: 10 days, 18 hours and 27 minutes
14U, 0N, 45S, 4I, 37WA, 0HI, 0SI, 0ST; 1008T, 493F
wad 17778 S 1.4 0.6
wad 4724 D 0.9 0.2
wad 4725 D 0.9 0.2
newcli 4669 R 0.4 0.6
httpsd 12792 S 0.0 3.1
cmdbsvr 108 S 0.0 2.6
forticron 155 S 0.0 2.5
pyfcgid 12455 S 0.0 2.3
httpsd 5924 S 0.0 2.3
cw_acd 181 S 0.0 1.8
httpsd 148 S 0.0 1.7
miglogd 227 S 0.0 1.7
updated 169 S 0.0 1.6
miglogd 145 S 0.0 1.6
pyfcgid 12470 S 0.0 1.5
pyfcgid 12471 S 0.0 1.5
pyfcgid 12469 S 0.0 1.5
fgfmd 180 S 0.0 1.3
newcli 4483 S 0.0 1.2
initXXXXXXXXXXX 1 S 0.0 1.1

# diagnose sys top-summary

[H[J CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
Processes: 20 (running=2 sleeping=108 disk sleep=1)

PID RSS ^CPU% MEM% FDS TIME+ NAME

  • 22828 6M 0.0 0.6 30 00:05.55 scanunitd [x3]
    144 5M 0.0 0.6 12 00:29.32 uploadd
    145 17M 0.0 1.8 57 37:03.36 miglogd [x2]
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    152 5M 0.0 0.6 11 37:17.80 merged_daemons
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap
    164 6M 0.0 0.6 41 07:49.53 proxyd [x2]
    165 7M 0.0 0.7 21 02:21.30 voipd
    [H[J CPU [||||||||||||||||||||||||||||||||||||||| ] 97.9%
    Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
    Processes: 20 (running=1 sleeping=110) PID RSS ^CPU% MEM% FDS TIME+ NAME
  • 174 16M 14.2 1.7 13 00:00.65 sshd [x4]
    17778 8M 1.7 0.9 71 28:59.29 wad [x6]
    145 17M 0.8 1.8 57 37:03.37 miglogd [x2]
    152 5M 0.8 0.6 11 37:17.90 merged_daemons
    22828 6M 0.0 0.6 30 00:05.55 scanunitd [x3]
    144 5M 0.0 0.6 12 00:29.32 uploadd
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap
    [H[J CPU [|||||||||||||||||||||||||||||||||||||| ] 96.6%
    Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
    Processes: 20 (running=2 sleeping=108 disk sleep=1) PID RSS ^CPU% MEM% FDS TIME+ NAME
  • 174 16M 13.3 1.7 13 00:00.81 sshd [x4]
    17778 8M 0.8 0.9 71 28:59.30 wad [x6]
    22828 6M 0.0 0.6 30 00:05.55 scanunitd [x3]

    144 5M 0.0 0.6 12 00:29.32 uploadd
    145 17M 0.0 1.8 57 37:03.37 miglogd [x2]
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    152 5M 0.0 0.6 11 37:17.90 merged_daemons
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap
    [H[J CPU [|||||||||||||||||||||||||||||||||||||| ] 95.4%
    Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
    Processes: 20 (running=1 sleeping=109 disk sleep=1) PID RSS ^CPU% MEM% FDS TIME+ NAME
  • 174 16M 13.4 1.7 13 00:00.96 sshd [x4]
    17778 8M 1.7 0.9 71 28:59.32 wad [x6]
    22828 6M 0.8 0.6 30 00:05.56 scanunitd [x3]
    144 5M 0.0 0.6 12 00:29.32 uploadd
    145 17M 0.0 1.8 57 37:03.37 miglogd [x2]
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    152 5M 0.0 0.6 11 37:17.90 merged_daemons
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap
    [H[J CPU [||||||||||||||||||||||||||||||||||||| ] 92.5%
    Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
    Processes: 20 (running=3 sleeping=107 disk sleep=1) PID RSS ^CPU% MEM% FDS TIME+ NAME
  • 174 16M 12.6 1.7 13 00:01.12 sshd [x4]
    17778 8M 0.8 0.9 71 28:59.33 wad [x6]
    22828 6M 0.0 0.6 30 00:05.56 scanunitd [x3]
    144 5M 0.0 0.6 12 00:29.32 uploadd
    145 17M 0.0 1.8 57 37:03.37 miglogd [x2]
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    152 5M 0.0 0.6 11 37:17.90 merged_daemons
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap
    [H[J CPU [||||||||||||||||||||||||||||||||||||| ] 94.6%
    Mem [|||||||||||||||||||| ] 51.0% 519M/1008M
    Processes: 20 (running=5 sleeping=107 disk sleep=1) PID RSS ^CPU% MEM% FDS TIME+ NAME
  • 174 16M 12.0 1.7 13 00:01.26 sshd [x4]
    17778 8M 1.7 0.9 71 28:59.35 wad [x6]
    22828 6M 0.0 0.6 30 00:05.56 scanunitd [x3]
    144 5M 0.0 0.6 12 00:29.32 uploadd
    145 17M 0.0 1.8 57 37:03.37 miglogd [x2]
    147 5M 0.0 0.5 8 00:00.00 kmiglogd
    148 37M 0.0 3.7 23 03:36.18 httpsd [x4]
    150 5M 0.0 0.6 8 01:42.71 getty
    151 6M 0.0 0.6 12 00:35.65 ipsmonitor
    152 5M 0.0 0.6 11 37:17.90 merged_daemons
    153 8M 0.0 0.9 15 00:13.70 fnbamd
    154 5M 0.0 0.6 11 00:42.72 fclicense
    155 25M 0.0 2.6 24 13:12.92 forticron
    156 10M 0.0 1.0 17 01:18.52 forticldd
    157 8M 0.0 0.8 44 00:07.82 authd [x3]
    158 8M 0.0 0.8 23 00:03.44 foauthd
    159 5M 0.0 0.6 14 00:45.79 clearpass
    160 6M 0.0 0.6 10 00:00.44 httpclid
    161 6M 0.0 0.6 11 00:00.12 fas
    163 5M 0.0 0.6 10 00:02.38 fsso_ldap

noticed that couple of WAD process was in D state.

So I would suggest you to reboot the Fortigate device to recover from D state.

If still problem persist please share below out put.

get sys status
get sys per status (run this command 5 times in interval of 1 minutes)
diag sys session stat
diag hardware sysinfo memory
diag sys top 4 40 (run this command for 40 seconds)
diag sys top-summary
diagnose sys session full-stat
diag sys session stat
fnsysctl cat /proc/stat
fnsysctl cat /proc/interrupts
diag hard sys slab
fnsysctl df -k
fnsysctl ls -l /tmp
diag ips session status
diag ips memory pool
diag ips share pool
diag ips signature status
diag ips dissector status
diag ips packet status
diag test application ipsmonitor 13
diag debug report

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at WordPress.com.

Up ↑

%d bloggers like this: