Think like a software engineer
Richard’s friends manipulating d… data
Silicon Valley is an American comedy television series which ran successfully from April, 2014 till the end of last year (2019). The series focuses on five young men who found a startup company in Silicon Valley, and parodies Silicon Valley culture.
(Note: This is the second article in a series. The first article is, “How can we solve problems in four ways?” The good news is that, like some television series, you can enjoy this independently without referring to other articles in the series.)
The lead inventor is Richard Hendricks who creates an app known as Pied Piper which contains a revolutionary data compression algorithm. In the Season 1 finale titled, “Optimal Tip-to-Tip Efficiency”, Richard is on the stage in the TechCrunch Disrupt Startup Battlefield. He introduces Pied Piper as simply a compression company. He states that their much bigger rival Hooli built a platform named Nucleus on exactly the same engine and with a richer set of features and, “and their Weissman score was 2.89, same as ours.” He continues, “..we all know that the most obvious way to compress files is with
Shannon coding, right? Top down.” Further, “…Uh, so Shannon coding
and then David Huffman came along, and he pioneered bottom-up, and
Lempel-Ziv with left-to-right, obviously.” He lays down this context before saying, “last night, um, I was watching my friends here have this argument about, you know, uh, manipulating d… Uh, data. And, you know, how many datas could one guy manipulate at once and…(CHUCKLES). And I was just… I was thinking. Uh, maybe it could be another way, you know? Uh, something that I would call, “middle-out.” It’s up-and-down, back-and-forth all at the same time going on at once.”
Without getting into too many details, do note that the Weissman score is a fictional efficiency metric for lossless compression applications. On the other hand, Shannon-Fado coding, Huffman coding and Lempel–Ziv–Welch (LZW) are real algorithms for data compression. How Richard presented his solution is an excellent example of how a software engineer (a good one!) thinks in terms of algorithms. I thought that this example might better illustrate an algorithm than just a definition. For a definition, we can turn to Wikipedia.
In mathematics and computer science, an algorithm is a finite sequence of well-defined, computer-implementable instructions, typically to solve a class of problems or to perform a computation.
The big O of a mathematical kind
An algorithmic thinking is what I dubbed as, “Think like a Software Engineer” in my previous article, “How can we solve problems in four ways?”
One popular example is — how to sort a list, say a set of books in a library in the alphabetical order? Check out this short TED Ed video on, “What’s the fastest way to alphabetize your bookshelf?” Do you know that there are more than three dozen different sorting algorithms? Do you wonder, why people have to come up with so many different ways to sort a list when just one should apparently do fine?
I don’t mean to get into a detailed discussion on algorithms, but briefly it has to do with which algorithms can run faster (time complexity) or need lesser memory (space complexity) as the input size grows. In computer science, the big O notation is used to indicate the computational complexity of an algorithm.
The graph alongside shows some of the common big O values for various algorithms. In the sorting example, “n” can denote the number of objects to be sorted, e.g. the number of books in our bookshelf example. What this means is that while for one algorithm where O is n, when n increases from 10 to 20, complexity also increases from 10 to 20 (i.e. 2 times), for another algorithm where O is n², the complexity increases from 100 to 400 (i.e. 4 times). So, the first algorithm seems to be better or more efficient. The pursuit of efficiency has led people to develop a number of algorithms to solve the same problem.
A mobile app for Pay As You Drive
So much for explaining what I mean by, “Think like a Software Engineer.” Let’s turn to one of the problems which we encountered in the first article, “How can we solve problems in four ways?”
Problem — You are the product manager of a Pay As You Drive (PAYD) solution. You need to develop a mobile app which will be able to tell you how much does someone drive as well as how well do they drive. How will you solve this problem?
In solving this problem, one could apply algorithmic thinking at multiple steps. One is to use the Global Positioning System (GPS) feature in the phone. However, since researchers have already provided good answers for us for trilateration algorithms, which help us to pinpoint the location of our phone with the help of at least four satellites, unless we are doing some research in this space, we don’t need to look for new solutions. Similarly, navigation problem from location A to location B has been solved with Dijkstra’s algorithm and Bellman-Ford algorithm among others. So again, we don’t have to worry about it.
From the above mentioned algorithms, our phone can already track the distance that we cover. So, we are kind of covered on “how much does someone drive.” I am saying “kind of” since there are complexities such as the fact that the phone GPS cannot measure the distance if the GPS is not turned on. However, we will keep that aside for now. Let’s focus on “how well do they drive,” and see how algorithmic thinking can help us.
I have actually worked on such a problem as a product manager. We came up with what we dubbed as ABCD algorithms to find a solution. Without getting into too many details, this is what ABCD stood for:
A — Acceleration and velocity: GPS can provide us with latitude and longitude at any time. With this, we can get the Lat and Long at the initial point, say X₀ and the next set of points, say X₁, X₂, etc. Next we note the timestamps, say T₀, T₁, etc. at each point and can use it to calculate velocity V₁ = (X₁-X₀)/(T₁-T₀). We can similarly calculate acceleration by noting the difference in velocity at different times. Well, actual calculations involve some trigonometry since the earth is a sphere (actually not quite a perfect sphere either) and not a plane. I just want to share the thought process and am not getting into the actual equations which we used. Of course since we are measuring the average velocities and accelerations here, we need to have minute readings at short intervals. This in turn poses the challenge of ingesting, storing and manipulating streaming data. This is an architecture problem and I am not getting into it.
B — Braking is nothing else but deceleration and can be measured in a similar manner as above. One more thing to add. We usually compare acceleration and braking to the g-force (gravitational force) which is roughly 9.81 m/s² and express them as fractions of g to look for hard acceleration and cornering compared to the standard ranges.
C — Cornering is trickier to calculate since here acceleration is at a tangent to the motion of the car and we need both velocities at different times as well as the radius of cornering. However, the principle is still the same and we compare with recommended range of g for different classes of vehicles.
D — Day and time as you can imagine are the easiest to measure, but still very important. Past data shows us that certain days and times of the day are riskier to drive compared to others.
I wanted to share an example of how to “Think like a Software Engineer”, with just glimpses of the thought process. Explaining the entire set of what we dubbed as ABCD algorithms will require a complete article devoted to it and that too a long one.
Train with Aditya, Shakuntala and George
A final few words before I conclude this article. If you ask me, “This is fine, you know, but how should I train to think like a software engineer?” I have three answers for you.
- Algorithmic thinking — If you want to learn algorithms, I would recommend, “Grokking Algorithms: An Illustrated Guide for Programmers and Other Curious People” by Aditya Bhargava. One word of caution is that I don’t program for a living. I do it only to learn new concepts. So, if you find better advice from a professional programmer, go for it.
- Analytical thinking—If you want to practice analytical thinking, I would recommend books such as, “Puzzles to Puzzle You” and “More Puzzles to Puzzle You” by Shakuntala Devi, “The Great Book of Puzzles & Teasers” by George J. Summers. I and my friends used it more than two decades ago while preparing for the selection process of software firms, but I believe that they remain relevant even today. Of course, practicing through Chess, Sudoko or Lumosity is also great.
- Physics and Mathematics — In 2017, WIRED magazine published an article, “Move Over, Coders — Physicists Will Soon Rule Silicon Valley.” It generated a lot of debate. I love Physics and from my personal experience, I highly recommend brushing up your Physics, especially, Mechanics, Thermodynamics and Electromagnetism. Also Mathematics, especially Calculus, Trigonometry and Probability.
That’s it for now. We will discuss another approach to problem-solving the next time round.
