There are more ways than one for caves to form, depending on the kind of cave.
A lava tube forms as lava is flowing quickly from its vent, and as the sides cool down and solidify, the molten lava within flows out, leaving a tube.
In the case of a water conduit, usually a soft mineral such as gypsum will be dissolved and washed away as water moves from drains on the surface, on its way to emerge somewhere lower down. If it happens to have a hard layer of limestone above it so it doesn’t collapse, you get a cave. These will almost always have two or more entrances, for obvious reasons, and you will want to stay out of them if rain is predicted near the cave.
But the caves with the well-known formations are almost completely in limestone, such as a reef that used to be under the sea but is now raised above sea level. Rain water soaks down through the dirt, and then percolates through the limestone. If it has absorbed enough acid while going through the soil, it will slowly dissolve chambers or passages within the layer of limestone. It will help a lot if the limestone on the top of the passage is more durable, such as dolomite, so it doesn’t collapse.
You get formations when the water percolating through picks up some nice calcite from the limestone, and then deposits it on a surface when it reaches a cavity and evaporates. The exact formation will depend on a variety of factors such as rate of flow, rate of evaporation, the saturation of calcite to water droplet, the shape of its surface and air movement through the cave.
Next time you tour a cave, pay attention to the ceiling and you should see the fault line where the water created the chamber or passage you’re standing in. It will look like a crack following the passage. In a room, you may see two faults intersecting.
An exception to the normal formation of caves as described above: in Carlsbad Cavern, the chambers were dissolved by unusually strong acids caused by petroleum reactions from below. But even then, they were enlarged by water from above, as evidenced by the fault lines that can be seen in the ceiling - and the formations were all caused by water from above, dripping down and leaving calcite deposits as the water evaporated.