This is a very early and highly conceptual but what about a mod that worked in a similar vein to psi, thaumcraft, and bewitchment/witchery in that you were given chalk and could write your own sigils/rituals that you could then write on the fly. Further if you added some level of integration between mods, you could in theory create multi-block rituals where you put down several sigils and can actually expel or trap things such as lesser demons. I am unclear on if this would even be possible to code but I definitely think it would be a blast to throw down custom sigils in combat.
I don't really know much about mod making and I don't know if this would be any good, but I personally think it has strong potential and could definitely open a new field in the realm of magic mods. I just wanted to get this idea out there, if you have any suggestions, criticisms, etc feel free to comment them. (and while I doubt it ever will, if you think it has potential to become a mod and either are, or know someone who could make that a reality I would love to work to make this at least into a prototype stage to see its potential)
(extra theories and context that aren't required to get the general idea, I don't really know how to describe any of this so expect it to be a bit confusing:
The idea here is that you have lots of potential with VERY little actual model making and so on, if you gave people enough basic shapes to work with (ie : the shapes that "real" rituals are made with and you see being made by characters such as dr strange and john Constantine) and had them perform general enough functions you could really capture the feel of real magic usage. One way of doing this is not to have one shape (the smallest, indivisible part of a player made sigil) feed into another shape, but rather have them all have a heat map like feeding area so instead of just saying "this shape feeds into this shape to do this" it would be a lot more free form and this would even lend way to giving the player less information so they have to get a feel for each shape on their own. (so basically you could have one shape that feeds in a cone with directly forward being most efficient, and its even more efficient if the shape its feeding into is about 75 pixels away but not much more than 100)
Another idea is to have a "hand drawing" item which would be less like john Constantine drawing out rituals on the ground to expel a demon, and more like dr strange writing one in mid air to blast one a few feet back; These would be quicker and easier to fire with more directionality, but also likely weaker and would be relatively hard to animate well.
The main goal with this idea is to stop the chronic issue I feel is facing the "magic" mods we currently have, predictability; most if not all magic mods we have now are VERY predictable and don't really let you feel like a bleeding edge witch/warlock. You end up mostly just using simple rituals to cook food, teleport, or just set the occasional zombie ablaze, and it ends up getting boring fast. With that said, having to improvise a warding ritual as you draw it with the only indicator of where each shape goes being the database you have in your head, THAT would feel like magic. I think a system like this would be playable for years with a remarkably low amount of upkeep since you really only need to add one or two items to get the full functionality of the mod. I would love to see magic taken in a more warlock-y direction with looser rules and stronger potential because I think that magic is in a very unique place in which you have the freedom to do whatever you want, but if you mess it up it can go bad fast. (example : if you don't have efficient enough power transfer from one of your shapes to another, it explodes because that power has nowhere to go) the issue is that I don't feel like many mods have capitalized on this potential and instead end up feeling like a tech mod that doesn't want to accept its a tech mod. I love thaumcraft because I think it takes a step in the right direction with vis and flux being both very useful and very dangerous along with being theoretically controllable, but ultimately wild. With that said however, it does feel like its missing something. Psi is better in the customization department but it is literally just an abstracted programing language so it also misses the feel of "warlock magic".
edit: I should clarify, none of the mods I listed here are bad mods, I just personally think they are relevant examples. My idea of a magic mod that deliberately is quick to use but slow to develop and unclear to the player (in its nature, not in its execution, being able to make a sigil should be easy, being able to understand one should be hard) is just one that I dont think any mods have truly captured and I think it has a lot of potential.
example edit : I thought of a good example that might help convey how you could solve problems in interesting ways with a mod like this: imagine you have icbm or another massive crater making mod, now imagine you have a base. (pretty simple so far right?) but someone knows where your base is, and they dont like you. You could fashion a warding sigil that works in groups so you can place the sigil all around you base, and they all work together across your base to ward it and increase its durability. This allows you to solve the problem of a vulnerable base, not by making a shield or other such clear and defined item/machine, but instead through smart use and development of wards that interlock to create a magical shield. (for balancing you could have it be so that if the ward does something, it starts to break by a proportional amount, this would allow things to break through warding and sigils to deplete. Equally there should be a "self repair" shape so that a ward system like this could SLOWLY reduce the damage its sigils had taken to protect against the ICBM missile. This allows for creative problem solving with both long and short term options (so for instance you could create a "multi-block sigil" (lots of small sigils you created that work together and share power, damage, etc) that allows something to enter it but not leave to trap a wither for instance, this sigil however, does not have a self repair shape because it doesnt have the power to actually "run" one of those shapes and wouldn't be able to sustain its warding if it had one added to it.)
I think that this idea has a ton of potential, and can fit really well into the already implemented systems of Minecraft. It reminds me a lot of redstone, where you give the player a simple set of tools that all have a designated purpose, and allow the player themselves to develop their own systems and abstracted tools with it, to create more complicated mechanisms.
I was actually trying to develop a system similar to it, but I couldn't work out the details for it, so I moved onto other projects. This however, I feel can be an excellent mod given the right circumstances.
I'm really excited to start working on this; I already have some ideas in mind for it myself.