We encountered a Vampire Spawn in the 10-foot-high basement of an old church. The first strike was mine, after which the creature attacked my ally and retreated onto the ceiling as the thundering roar of my Booming Blade punished its retreat. Still inexperienced and unsure about my reach, after a short consultation with my mind, I decided to spend my turn switching to my trusty crossbow and make some distance.

However this turned out to be a mistake, since right after, the creature was in in my face trying to gouge out my eyes. Due to my nimbleness I managed to avoid its attacks. Regretting my earlier choice to change weapons I was left with the options to switch weapons and miss another turn, attack with disadvantage, or use my crossbow as an improvised melee weapon and trust in Booming Blade to hold the enemy in place after I disengaged.

Perhaps I should have switched weapons, but I decided that 1d4 (improvised weapon) + 1 (magic ring) + 1d8 (booming blade) was good enough. I missed.

On the plus side, it was my crossbow's sneak attack assisted bolt that finally slayed the Vampire by piercing its heart.

In this situation, would it have been possible to just attack upwards instead of changing to the crossbow?

In session, this was something to be looked up later and decided against, but I haven't found any rules on attacking in 3D.

With me being more than 5 feet in height and my melee attack range considered to be 5 feet, I should have been able to reach the ceiling itself, not to mention the Vampire Spawn hanging from it.

As for exact measurements (read this part only if you enjoy pointlessly over-analyzing things):

My character is 5 feet 6 inches/5.5 feet, which gives an estimated standing reach of 7 feet 4 inches/7.3 feet (for a human, my character is a Kenku), the blade of a short sword is 12 to 20 inches / 1 to 1.6 feet. This should give me a height of 8.3 to 8.9 feet from standing statically.

Testing it in real life, my estimated and standing reach match to less than 0.2 inches of the estimation on the site and I'm just shy of reaching my ceiling standing statically. As a test I tried moving as if I was attacking my ceiling and based on where I could feel my hand touch the ceiling, the length of my fingers and how much distance I lacked to reach my ceiling, I estimate about 0.4 feet in additional height. Considering that I'm taller than my character, let's say that gives 0.3 feet, increasing the height to 8.6 - 9.2 feet and that's still without changing how you hold the handle of the sword. In addition, just lying on the ground, my torso and legs measure about 0.6 feet.

Because my Kenku isn't a human, only a humanoid, it's impossible to estimate arm length which influences standing reach, however, considering that birds usually have a high wingspan and long legs, I think it's reasonable to assume that the arms should be as long, if not longer than, a human's.

So that would require the Vampire Spawn to be flat against the ceiling without any parts hanging down more than 0.2 to 0.8 feet for it to be outside of my reach. I consider that unlikely.

What are the actual rules for this type of scenario, or how do you handle it in your group? Should I be allowed to melee attack an enemy that's on a ceiling or low in the air?

They were in melee range (on a grid)

Think of this as a 10' hallway. If you were standing in a 10' grid of a hallway and you were on one side the spawn was on the other, then your squares are adjacent to each other.

A medium creature controls a 5x5 space. Both you and the spawn are medium. If playing on a grid, your spaces are adjacent and cover the full 10' width.

This works the same way, just vertically.

In order for you to have been out of standard 5' melee range, the hallway would have had to be 15' high in order for there to be a space between you moving the target out of range.

Likely in range for theatre of the mind

In this scenario, there is some more flexibility. But as you've provided your height and we know the general size of the Spawn, it should still have been in range. And if the Spawn was prone on the ceiling to try and squirm away (which with your arms reach above your head is questionnable at best), you may have earned advantage on the attack.

From PHB 182, under "Jumping":

High Jump. When you make a high jump, you leap into the air a number of feet equal to 3 + your Strength modifier if you move at least 10 feet on foot immediately before the jump. When you make a standing high jump, you can jump only half that distance. Either way, each foot you clear on the jump costs a foot of movement. In some circumstances, your DM might allow you to make a Strength (Athletics) check to jump higher than you normally can.

You can extend your arms half your height above yourself during the jump. Thus, you can reach above you a distance equal to the height of the jump plus 1 1/2 times your height.

If you want to argue about attacking upwards, the rules on melee attacks also state:

Most creatures have a 5-foot reach and can thus attack targets within 5 feet of them when making a melee attack.

Which applies in this case and should give you a normal melee attack, or disadvantage if your DM wants to lawyer it that way and use rules like discussed in this thread: Weapon utility when used in confined spaces?

The linked rules do not apply, but your DM could argue that you are in some sort of compromised situation that is not providing you with freedom of movement that a fight requires, and therefore you are at disadvantage similar to fighting in a confined space or underwater, etc.

The jumping rules are in that sense relevant that you can at least stretch your arms upwards half your height if your DM wants to lawyer the undefined vertical/horizontal implication of reach.

I'm not arguing that you should be at a disadvantage - I think it is pretty clear that the reach applies - but there will be situations in which the ceiling might be 15 feet away in which those rules might make a difference.