I was installing Fedora 26 Workstation in VirtualBox 5.1.22r115126 with extension pack. I could not install the guest additions as I was missing some packages. First, make sure you have the latest available kernel installed:
$ sudo dnf update kernel*
If you get a new kernel installed, reboot the system to use the new kernel.
Install all packages needed for building the kernel modules for VirtualBox guest additions:
$ sudo dnf install gcc kernel-devel kernel-headers dkms make bzip2 perl
You will find the installer in: /run/media/USERNAME/VBOXADDITIONS_5.1.22_115126/
USERNAME is your username, and the version might be different.
Start the installer with ./VBoxLinuxAdditions.run
and let the installer run.