A Stats Major's Confession: Yes I will bring up I almost majored in Math

We statisticians and mathematicians have natural beef. There is a reason for stats and math being in two separate departments, besides the difference in fundamental logic, there's also beef. For example, in my undergrad we are in two separate buildings, far far away from each other. And yes, it is statistics that got the nicer building.

Like many stats major, I too will say I could have done math. The one class that did it for me was Elementary Number Theory. The entire time i was thinking: This is not elegant at all how is this math? Why is so discrete? Not only by definition but also by method. It would take a literal lightning strike for me to come up with come of the proofs.

So I thought statistics would be easier, without losing the elegance. Boy was I wrong.

What Statistics/Data Science is all about:

It is holding two contradicting truth at the same time and living through that tough headspace of torture.

I "transfered" from math to statistics looking for continuity, elegance, the truth and the absolute truth of the universe. Statistics is pretty much the exact oppisite of that. So I had a couple hard years of learning the subject. It's so messy, chaotic, and when I finally training my own models and doing testing on real, big data, I got into the situation where I thought to myself: I cannot believe this is how it works?!

Later on, I've met MANY, MANY people who understands the importance of the subject, but do not understand the subject at all. They lean towards one of the two directions:

1) I know data is the most important gold right now. But I don't believe your number. I don't think it's that accurate/valuable. I want more rigorous testing!!
2) You are totally getting replaced by AI.

I think type 1 have some level of understanding for data, but they do not understand it is a discipline of restraints. After my 4 years in undergrad I cannot say a full sentence using 10 attributives. That restraint, especially in the "industry" (as opposed to academia/big pharma that has much longer cycles), not only means the restraint you have on the problem/hypothesis, but also, a) burning deadline and anxious clients; b) the lack of data (some of yall don't have big enough data or resources to get data). In both cases, there's simply no way and no point to do any of the fancy stats testing or machine learning or chasing the kind of "accuracy" that they want. Because this is THE subject of ambuiguity.

What I always say, is that: given all the time&resources that I was given, I've tested N different versions, and I say with limited confidence in myself and in my number, that is is the final (decision) And this IS the best and the right call. However, we could be wrong. Even very likely to be wrong.

So this is why it's a valuable, but a grueling subject to learn. It forces you, to kind of believe you are (almost) 100% right and (almost) 100% wrong at the same time. And take the responsibilty for it.

For type 2), they probably never worked with any data at all. The water of data is very deep and dirty, and there will always be a plumber. Sometimes you just have to look into the drain to figure out what's going on. AI inherently don't know how to ask questions.

I hate statistics. But it's the fundamental way to research, to operate in any field, and it will always be. We are just so applied. As I just said, there will always be plumbing.