We know that a lack of understanding feels like chaos, muddled thinking and an uncomfortable lack of confidence about some concept. In short, it is the inability to debug our brain and take it from a question to an answer. Let us call this not having understanding driven by “first principles”. It often leads to a subtle but critical problem: being wrong and not being able to spot it. The anecdote to this state of chaos is learning from first principles.
I’ve often had this experience: certain technical topics look extremely difficult to break into. No matter how many standard resources and popular content that you read through, every one of them seems to speak a foreign language. I’ve felt this while studying topics like complex numbers, Fourier’s transform, relativity, to name a few. I am of the opinion that either I’m too dumb or writers of many popular expositions on such topics have not understood it clearly.
Machine learning field at the moment seems to be in pinnacle of hype cycle. A researcher or a lab or organization creates some marginal improvement over existing status quo in some narrow area, they self-publish it as paper with some preposturous headline and then the circle jerk starts. I have a personal heuristics to deal with such hype cycle based on following observations. Observations Not All Papers are Created Equal In most of the universities it is mandatory for a master’s and PhD student to publish some work.
Life is short, so one should read only the books that give you a new perspective on the world life and all that. This is a central tenet of my reading these days. Considering that I have probably lost more than half of my active life already, books with repetition of trivialities bother me. I stumbled on Paul Lockhart while going through the black hole of learning maths. His essay mathematician’s lament is very famous.
The topic of geometric algebra is a new fascination for me. I first read about David Hestenes while reading Bret Victor’s kill maths project. Then got a copy of David Hestenes’ new foundations of classical mechanics book. One thing led to another and then I ended up reading an old out of print book by Clifford called common sense in exact sciences. The basic premise of geometric algebra seems fascinating to me: a universal and simple mathematical theory for a wide variety of applications in Physics.
Studying quantum mechanics often gives you an eerie feeling. I tend to feel that something very adhoc is happening. Reading about special relativity is different for example: you start with some reasonable axioms and then the maths kind of follows. But not so a case with quantum mechanics. I’ve tried to read many books to get sort of a big picture view of what the hell is happening to no avail.
I sometimes long for those days of internet when you’d land on someone’s personal page where they’d document hard to find links about some (technical) subject matter and some commentary on them. Discovery of such links often meant that that your understanding or interest in certain subject would completely change. I also loved the painstaking effort someone would make to find and then document information for an unknown (web) traveller. There is a kind of romantic hope in doing this, similar to human beings putting out Voyagers spacecrafts with those golden record.
I’ve documented my disdain for using ‘deep learning as a hammer looking for nails in every corner’ in few other entries on this blog. Taking a less cynical and acerbic view, I want to focus on when deep learning can be a good starting point and when it is not useful as starting point. First to clarify bit of terminology. Deep learning is a catch-all term used in popular data science these days.
Andrew Ng wrote insightful article in HBR a while ago about what an AI can and cannot do. He highlights that if there’s a mental task that a human being can do in few seconds, you can probably automate it. In my opinion, these types of problems have a following structure, Single person decisions: these decisions are often replacing decision making of single human being. Scale factor: You need AI to help you in tons of such decisions in order to make meanigful impact.
Data scientists and machine learning engineers in small and medium businesses often end up over-engineering their machine learning workflow and stack. In a series of posts below, I will share a few tricks learnt over the years related to choosing right components of the ML pipeline. In this first post, let us go through mistakes small teams can make. Later posts will explain possible solutions. Preferring Generalized Solutions Many big companies will perfect a very general solution to machine learning problems by investing obscene amount of resources.