First, I'm not looking for a guide on how to stay safe when downloading mods! I've been using mods for years with no trouble, using all the most trustworthy sources.
Instead, I was wondering if it was even possible for a Minecraft mod that I download to cause any damage to my computer outside of the Minecraft program? So this would be more like stealing data, compromising my system or the like, rather than just something like destroying all my Minecraft worlds.
Basically, is Minecraft (with Forge) perfectly sandboxed, or can mods leak out?
In nearly ten years playing Minecraft I have never heard of such a case. I think java simply doesn’t support to program things that can cause damage outside of it. When it comes to viruses and Minecraft, an .exe-installer always plays a role.
Some "Mod-Download"-sites often comes with mod-installer.exe or something like "modname".exe
Java is not safe; there is a good reason why there was so much concern over security back when it was supported as a browser plugin (basically the same thing as a standalone JVM), and subsequently dropped for these reasons - Minecraft can access things outside of the JVM, particularly through native libraries (e.g. disk, network, graphics, keyboard, etc). In fact, Java has even been described as the single biggest security vulnerability:
That said, you are more likely to get serious malware from a C++ based program since the JVM does prevent a lot of the things that C++ can do (e.g. there are always array bounds checks, preventing buffer overflow attacks; likewise, you can't access random/uninitialized pointers, in both cases at least not without accessing native code), and many malicious mod download sites will bundle other software with the download or serve malicious ads, rather than as part of the mod itself.
I will agree that Java isn't safe (like MasterCaver stated which is the answer to believe the most) and C++ has it's areas. A controlled environment for content is something but that won't really stop it if you find content on sites for Bedrock like behaviour packs compared to anything in the marketplace. The marketplace is well controlled but third party sites, that's where things will be the case for getting malware, viruses, etc.
To me as long as you go to Curseforge or the Modder's own sites (which aren't that bad but they used to with ad site redirects, these days it's usually Github though compared to say MrCrayfish's website which is pretty good, but some were pretty bad) but you should be fine as long as you avoid repost sites (you can find a list here https://stopmodreposts.org/sites.html). I used to use Repost sites when I first got into mods as many people do when they don't know what's safe or not but dropped off and went to Curseforge once I knew, and rarely do I go to Repost sites for older mods (which would be my only reason to use them now) if even if the sites are safe enough as I find no need to anymore (for fun using old versions or reviewing sake).
People are idiots, you never know. They might give you a hacked file. Java can easily hack your computer. It would take a bit of programming, but it is possible. I'd say the main problem is with things like Optifine, where you run a file that will install a version in your .minecraft folder. From there, they can easily install malware, and you'd never know. I agree with all of the previous guys (SuntannedDuck and MasterCaver) in that C++ is a lot easier to hack in. I can code in C++, and it is very much possible. I'd say as long as you only download files that you put in the mods folder, you'll be perfectly fine.