I once encountered a bug on an MX204 running firmware 19.4R3-S6.1 that caused BGP routes to take a very long time to clear when a neighbor was lost. A firmware update was necessary to resolve this issue, as it was fixed in version 21+.
At the time of the bug, the RE load rose from 3% to 23%, the rpd process was loaded at 100%, and when trying to view received routes from offline neighbors, I received a timeout response:
show system processes extensive | match rpd
show route receive-protocol bgp IP
error: timeout communicating with routing daemon
show bgp neighbor IP
Flags: <PurgeInProgress PurgeImpatient PurgeActive>
Several million routes were deleted over the course of a few hours. At that point, some sites might not work, and then gradually resume working as the routes were deleted. A quick solution would be to restart rpd and all related processes, but this would also break all other BGP sessions:
restart routing
So I decided to update the firmware as quickly as possible and looked at the current version:
show version
Looked at the system backup:
show vmhost snapshot
It was old, so I made a new one. The process takes about 5 minutes. The command will overwrite the existing snapshot:
request vmhost snapshot
If necessary, you can restore from a backup snapshot:
request vmhost snapshot recovery
I looked at the available space:
show system storage
file list /var/tmp/
Let’s save the configuration to the server using any method (normally, it should be copied to the server with backups daily):
file copy /config/juniper.conf.gz /var/tmp/juniper.conf.gz
file copy /config/juniper.conf.gz [email protected]:/dir/
scp -P 2222 [email protected]:/config/juniper.conf.gz /srv/tftp/`date +%Y-%m-%d`_0201_nas_58_.conf.gz
Uploaded the firmware image to the device in the /var/tmp/ directory:
scp -P 2222 junos-vmhost-install-mx-x86-64-21.4R3-S10.13.tgz [email protected]:/var/tmp/
If desired, you can check the integrity and compatibility of the firmware file, but this will already be done when you run the update:
request vmhost software validate /var/tmp/...
An example of a firmware installation command for platforms with a virtualized host, such as the MX204 (the process took me 16 minutes):
request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-21.4R3-S10.13.tgz
After installation, to start the device with the new firmware, we reboot (mine took 10 minutes):
request vmhost reboot
Let’s connect and make sure the device is running with the new firmware version:
show version
I upgraded the firmware version again in a similar way, trying not to upgrade it to a version higher than +2 from the current one at once, but gradually from 19 to 21 to 23:
scp -P 2222 junos-vmhost-install-mx-x86-64-23.4R2-S5.6.tgz [email protected]:/var/tmp/
request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-23.4R2-S5.6.tgz
request vmhost reboot
show version
You can leave the firmware archives in /var/tmp/ in case you need to install a different version from them. If everything works fine after a while, you can delete the old ones:
file list /var/tmp/
file delete /var/tmp/ixnfo.com.tgz
If the device has DHCP and RADIUS sessions, they will be completely lost, and all connected users will be required to re-authenticate. Therefore, before rebooting, I recommend clearing the sessions in the billing database to prevent desynchronization and subsequent unnecessary requests, and to immediately restore working sessions.
See also my articles:
Juniper MX204 setup
Juniper MX configuration backup