For starters, identify the problem that you wish to solve. Is it, e.g., galaxy rotation curves? The failure to detect dark matter/dark energy? The cosmological constant problem? Something else?
Second, identify your playground. Are you looking for a classical theory of gravitation, presumably an extension of Einstein’s theory of relativity? If so, would your theory respect, e.g., conservation laws or the weak equivalence principle? Or is your playground quantum physics and you’re looking for a quantum (field) theory of gravitation?
Third, identify the constraints. Existing theory works very well in most scenarios, and has been tested exquisitely well in some. Whatever modified theory of gravitation you come up with, it has to reproduce those successes in addition to its own claims.
Now notice that I have not said anything yet about the mathematical machinery. That would be putting the cart before a horse. Would you ever call a plumber to please, “do something with a wrench”? Or a car mechanic to please, do some screwdrivering? Of course not. You tell them what the job is. The job defines the tools, not the other way around.
Having said that, a pretty universal tool in the theorist’s toolchest is variational calculus and the principle of least action. Learn how to use these tools in the case of four dimensions, and in the case of a field theory. Understand what an expression like S=(16πG)−1∫d4x(R+2Λ)−g−−−√ means, why the terms are there, what they do, and how varying this equation with respect to gμν leads to Einstein’s vacuum field equations. Understand what current modified theories do: e.g., scalar-tensor theories, f(R) theories, theories involving unit or arbitrary vector fields, etc. Understand why people tried these approaches and why they were not altogether successful.
Though it’s more than half a century old, recommended reading would be Feynman’s Lectures on Gravitation. That book provides an approach that is different from the usual perspective: instead of presenting gravitation as a theory of 4-dimensional geometry, Feynman approaches it from a particle physicist’s perspective. From it you learn why it is not viable to use anything less than a tensor (or spin-2) theory to describe gravitational phenomena. With that and a bit of grounding in quantum field theory, including non-Abelian gauge theories, you may be ready to take the plunge and study quantum field theory on a curved background, and think about why gravitation is special. Also, why it resists attempts to be “tamed” by way of renormalization.
At this point, mind you, you’d no longer be asking random strangers on Quora how you could create a theory of gravitation. You’d probably have informed ideas of your own.
Answered by Viktor. T.Toth