I'm sure that having inactive mods does nothing to performance. They are not active, so your computer and by extension the engine does not have to load them, or read the instructions on those files. They're just wasting space on your hard drive. Removing them will make no difference on how the game is currently running.
If you have FOMM, you can multi-select all the esps you don't want and you can delete them. Multi-select works with ctrl+left click.