Before installing mods, a few things needs to be setup

As Grand Theft Auto 5 was not designed to use mods, and because RockStar Games wants to keep the online version of the game free from hackers and cheaters, it's HIGHLY recommended to have two installations of GTA 5. One installation with mods for single-player, and one separate installation without mods for online gaming. RockStar Games WILL ban anyone using modified game-files online, even pure graphic enhancements. *But it's perfectly safe to play single-player with mods, so don't let this discourage you.

1 - Make sure that GTA 5 is updated to the latest version.

2 - Make a copy of the game folder.
You don't need to install the game twice. Just make a new folder for the second installation, then copy all game files into it. That way it's very easy to switch between the modded version, and the mod-free version. See image.

In this example, when you want to play online, just rename the modded installation to "Grand Theft Auto V - Modded" and rename "Grand Theft Auto V - Online" back to "Grand Theft Auto V". That way you don't forget which is which.
*If you don't want to have two installations of the game due to the size (2x70GB), then you can use a mod manager instead. (LINK)
This mod manager operates via creation/deletion of shortcuts inside the game-folder to an external mod folder. Unfortunately it has some limitations with regards to folder locations, and not all mods might work via shortcuts. Try at own risk.

Important note: If you have the Steam version of the game, and the Steam client forces you to update the game, AFTER you finis modding it. Make sure to update the online copy of the game ONLY, and leave the single-player version alone. Don't update the modded version. Disable auto update, then, if you see that an update is pending, make sure that the online game folder is active. After Steam is done updating the online version, you can activate the modded version again, and Steam will leave it alone.

3 - Open the GTA 5 game folder. (The one you want to use for the modded version)
Create a new folder called "mods", and copy the following files & folders into it.
Folders: update, x64
Files: common.rpf, x64a.rpf, x64b.rpf, x64c.rpf, x64e.rpf, x64f.rpf, x64g.rpf, x64h.rpf, x64i.rpf, x64j.rpf, x64t.rpf, x64u.rpf

4 - Windows 10 "Creators update" specific:
The Windows 10 Creators update introduced a wide variety of problems for games, especially when using custom renders like ENB or ReShade. Fortunately you can fix this in two simple steps (LINK)
*See number 4: Fixing the Creators "update"

5 - Make some coffee. Downloading and installing will take some time, but at the end you will get the result of almost two weeks worth of tweaking and testing.

When ready
Proceed to the Mod-List