Ttdfjt

Basically, I would say there is two main things that will make your sentence "Formal" or "Casual":

1) Use of "vous" = formal, versus "tu" = casual.

2) Structure of the sentence itself:
Proper subject-verb inversion is formal (i.e: Comment vas-tu?)
leaving subject-verb with no inversion is casual (i.e: Comment tu vas?)

As a native (metropolitan) French speaker, I would absolutely never use the form "De qui parles-tu?". Instead:
- if speaking to anyone I can use "tu" with, I would say "Tu parles de qui?" or "De qui tu parles?"
- if speaking to a person I must use "vous" with, I would say "De qui parlez-vous?".

The form "De qui parles-tu" is not really used when talking because there is a contradiction between the high level of formality induced by the structure (subject-verb inversion) and the use of "Tu" which is informal.

As a side note, the famous sentence "Voulez-vous coucher avec moi?" suffers the same contradiction :
You wouldn't at the same time use "Vous" and a formal sentense structure if you're offering someone to have an intercourse with you...

Last but important, the ton of your voice is sometimes the only thing that indicates the difference between an affirmation and an informal question.
For instance:
- C'est ma soeur. --> She is my sister.
- C'est ma soeur? --> Is she my sister?
When spoken, the only difference between these 2 sentences lies in the fact that your voice goes high at the end if you want to make it a question.

Hope that helps.
Dan.

"Of whom are you speaking?"

sounds indeed formal, if only because "whom" is used.

It attempts to stick to the French sentence "De qui parles-tu ?" which might also be translated to "Of whom do you speak?" but the word for word translation would be the broken "Of whom speak you?" There is then no point to try too hard to match English grammar and French one.

De qui parles-tu ?

is a correct literary form to ask this question. It is unlikely to be heard in a conversation because the subject-verb inversion has almost disappeared in spoken French. However, it might still be used by native French speakers in formal contexts, in the few cases where the tutoiement¹ would also be used.

If the person who the question is directed to is not someone you would tutoyer, the still literary/formal form would be:

De qui parlez-vous ?

While quite rare in real life French, both of the previous questions are the most commonly found in written material.

In usual conversation, the subject-verb inversion is dropped so the question would be:

De qui tu parles ? or De qui vous parlez ?

De qui tu parles is also common in literature, just slightly less than the previous one.

A last simplification is to build the sentence as a regular subject-verb-complement one leading to:

Tu parles de qui ? or Vous parlez de qui ?

This last form is certainly the most common in relaxed colloquial conversation as it is the more natural to native French. It is slowly making it to books but still lagging behind the more formal variants. Here is a Google NGram graph of the various forms, with quoi used instead of qui as the former is more common.

So to answer to your question, French does have various grammatical constructions that make a sentence formal or colloquial, but they do not particularly match English ones. The examples you found are formal in both languages but that's more by accident than anything else.

You are right considering "Qui parles-tu de ?" as incorrect. The reason it is wrong is that the preposition "de" needs to be followed by something in French, unlike "of".

¹_{While the second person commonly used in formal French is the plural, even when speaking to a single person (vouvoiement), the tutoiement is not by itself turning a sentence to a standard or colloquial register. There are a few situations where the speech is formal but the tutoiement is nevertheless used because that's the way you address to a person. That person might be a child or someone belonging to your family, company, or club. That might even be God, a rare case where tu has survived in English ("Plus près de toi, mon Dieu" for "Nearer, My God, to Thee" or as Minty evoked in a comment "Tu ne tueras point" / "Thou shalt not kill").}

Your assumption at the end of your question is correct. However, the form considered is somewhat formal and numerous are the speakers that will baulk at using it, prefering the colloquial "De qui tu parles ?" (ngram) ; in fact almost nobody uses that form in the second person singular. Nevertheless, it can be used and does not sound yet pedantic; in the second person plural you can use "De qui vous parlez ?", but "De qui parlez-vous ?" is preferable.

COMPLEMENT (to try to explain an interesting constatation made by user Shautieh, see comments)

If we consider the frequency of the same three forms from which the interrogation point has been removed (De qui tu parles,De qui parles-tu,Tu parles de qui) we see that the trend is reversed and that the formal one is by far the most used: ngram. This shows that when there are words added to the form, formality becomes preferable. Here are the most common words that can be found (ngram);

• de, avec, ainsi, en, comme, donc, toujours,

We can then have more complicated questions beginning such as the following;

De qui parles-tu

• de cette façon … ?


• avec ton amie … ?


• ainsi …?


• en ces termes absurdes … ?


• comme si tu le connaissais depuis toujours … ?
  
  etc.

It feels formal to you because you are doing a literal translation including maintaining the word order. To do that you are forcing the sentence not to end with a preposition (which despite bad English teachers admonitions is a natural way for ending a sentence) and using "whom" instead of "who". A more idiomatic English translation would be, "Who are you speaking of?" or, "Who are you talking about?"

While the distinction between "who" and "whom" is formally one of case and "whom" should be used for all direct objects, in many (most?) dialects of English including mine it is essentially extinct and "who" is used for all cases. The exception is fixed expressions, which has led to the tendency to see "whom" as a formal version of "who". It is similar to people viewing "thou" as more formal (it's used all over the bible!) than "you" when in fact when "thou" was last in common use it was informal (and singular) paralleling the T-V split in French and other languages.

In other words the feeling of formality is a result of your choice of English translation. As other answers have stated, in French the level of formality is controlled, among other things, by "tu" versus "vous", the former being only used to address a single person you are on informal terms with.