A common theme when converting requirements to code is to talk about new requirements as if each requirement adds complexity to the final product. This is oftentimes true, but in my experience it is more often a myth.

There is a subtle difference between complex and complicated.

Complexity refers to a problem solution that cannot be reduced to something more simple because it is difficult to understand by nature.

Complication refers to a problem solution that can be reduced to something more simple.

Let me give some examples:

Complicated, but not Complex

var x = y + y + y + (2 * y);

The example above is not complex, but it is complicated. It is not complex because the task being performed is not hard to understand, but it is complicated because it can be simplified, like so:

var x = 5 * y;

In this case, we can actually reduce the amount of code and make it easier to understand. We remove the complication and our end result is neither complicated nor complex.

Complex, but not Complicated

There is some code that is inherently complex, but not complicated. A great example of this is the DFT (discreet fourier transform). The amount of mental effort to understand and write your own implementation properly is high. However, in most cases the result cannot be made less complicated. In other words, the final result cannot be reduced to something more simple.

The following is taken from the fft-js library:

var dft = function(vector) {
  var X = [],
      N = vector.length;

  for (var k = 0; k < N; k++) {
    X[k] = [0, 0]; //Initialize to a 0-valued complex number.

    for (var i = 0; i < N; i++) {
      var exp = fftUtil.exponent(k * i, N);
      var term;
      if (Array.isArray(vector[i]))
        term = complex.multiply(vector[i], exp)//If input vector contains complex numbers
      else
        term = complex.multiply([vector[i], 0], exp);//Complex mult of the signal with the exponential term.  
      X[k] = complex.add(X[k], term); //Complex summation of X[k] and exponential
    }
  }

  return X;
};

The DFT is, by nature, a complex algorithm. But the code above is not complicated because the above code is rather simple for a DFT implementation.

Complex and Complicated

Some code is both complex and complicated. If you have ever taken a computer science course and watched yourself, or your classmates, struggle to implement an algorithm you have probably seen your fair share of complicated and complex code. The results are both hard to read and the underlying algorithm is also hard to understand.

Imagine you ask someone to only make a factorial function in Javascript, and that is your only requirement. Then imagine someone hands you the following (taken from the crowd-sourced examples here):

function memoize(func, max) {
    max = max || 5000;
    return (function() {
        var cache = {};
        var remaining = max;
        function fn(n) {
            return (cache[n] || (remaining-- >0 ? (cache[n]=func(n)) : func(n)));
        }
        return fn;
    }());
}

function fact(n) {
    return n<2 ? 1: n*fact(n-1);
}

// construct memoized version
var memfact = memoize(fact,170);

// xPheRe's solution
var factorial = (function() {
    var cache = {},
        fn = function(n) {
            if (n === 0) {
                return 1;
            } else if (cache[n]) {
                return cache[n];
            }
            return cache[n] = n * fn(n -1);
        };
    return fn;
}());

The above code is complex, but not complicated. Wait, you say! That looks super complicated! All I wanted was a factorial function, why is it so difficult to read?

The above solution fulfills another requirement we did not ask for: it optimizes the factorial function. As a result, it is not complicated when we consider both the requirements to write a factorial function and optimize it.

However, it is far too complicated to only fufill the requirement to write a factorial function. The result is complex by nature of the optimization. As a result, it is too complicated for our simple requirement.

Complication depends heavily on the context of the requirement.

In most cases, this is what managers mean when they say not to "overengineer" something. They are saying not to add requirements that add complexity when the actual need is for something with fewer requirements.

As a result, if the only requirement is a factorial function, we could have simply written this:

function fact(n) {
    return n<2 ? 1: n*fact(n-1);
}

Complication has far more to do with how you tell the computer, or other people, how your code solves the problem.

Complexity has to do with the nature of reality. Complication has to do with how well we communicate our solution.

Optimization adds Complexity

Optimization, by its very nature, can (but not always) add complexity to your code. As a result, it is often advised that you do not pre-optimize. Obviously, if you already know the proper optimization and can write clean code that is properly documented it saves time in the long run to optimize from the start. But if you do not already know how to optimize the code, it is best to just get it working first and optimize later.

That said, it takes a senior engineer to know when to optimize from the beginning. For example, if you ask a junior engineer to write a database, they might start with just one big file and everything stored in a JSON blob. However, a senior engineer will understand hard disk caching optimizations and file paging and will begin from the start by writing a much more complex solution involving B-trees and RAM caching. Which would you prefer? Managers and junior engineers will enjoy the first solution more, but it will have to be completely rewritten rather quickly. And that rewrite will end up costing the company tons of money down the line. The senior engineer's solution might take a little more time up front to write but it will last for years.

As a result, in critical systems that require scaling, this is why senior engineers are usually paid much, much more for their time. They are paid for the patterns they have memorized that allow them to write much more scalable code from the beginning.

Modular Programming Reduces Complication

As an engineer gains seniority, then, the top skill they must learn is memorizing software patterns so that they can write optimized code from the beginning, saving the headache of refactoring later down the line.

Consider the following code:

function renderHello() {
  return `<div>Hello</div>`;
}

renderHello();

This code is not modular. It is not complex. And it is not complicated. But it is not scalable. If this pattern is repeated, the code becomes unnecessarily complex and complicated, like so:

function renderHello() {
  return `<div>Hello</div>`;
}

function renderWorld() {
  return `<div>world!</div>`;
}

renderHello();
renderWorld();

This is obviously a crazy example, but it is tempting often as an engineer to just start copy-pasting previous solutions and tack them onto our former code as new requirements come in.

After copy-pasting code for a while, you eventually end up with needlessly complicated code. It would be better to think through a modular approach from the beginning to avoid the complication caused by copy-pasting former solutions.

function render(text) {
  return `<div>${text}</div>`;
}

render('Hello World!');

Hopefully as we grow as engineers, we get better at making our code more modular. Our skills at modularizing begin with building reusable functions and then extend to complex recursive patterns in functional programming or Class and Object patterns in OOP. Eventually you end up with monads and interfaces and abstract classes. These are all added complexity that help made code more reusable and solve problems in a common way.

The end goal of learning and developing software patterns is to build a mental toolkit that help us write code that might at first seem more complex, but in the end allows us to add new features to fulfill requirements with minimal effort.

Nothing New

I would be remiss not to mention the importance of researching existing modular approaches and software patterns or algorithms before trying to solve your own.

I remember this one talk at a Javascript conference that impressed me with this more than anything. A brilliant engineer had discovered that you can compose functions like add(), multiply() and more in a recursive pattern in order to solve problems in a novel way. Furthermore, he wrote an entire library that did this while also allowing you to convert your recursive function calls into text so that they could be saved and run later.

What he did not realize was that he had basically rediscovered the concept of a lexer used in compilers. If he had read first and attempted to solve later he could have saved this time. I have made this mistake myself before and it is not fun at all to discover someone else already solved the problem.

Before you optimize and add unnecessary complexity and time, always research the problem you are solving to see what optimizations, software patterns, or algorithms already exist. With all the Masters and PhD students out there, 9 times out of 10 a generic solution already exists for the problem you are solving.

Conclusion

In my opinion, the skill of making code modular so that you can reduce complication is one of the most fulfilling in software development. Whenever you are given a new requirement for your software, take a little extra time up front to imagine how your former code could be written in a modular manner so that new requirements require minimal effort to implement.

Modularization is a skill that sets apart the big players who can write frameworks and libraries like React, Vue, and even Docker or Kubernetes. These engineers through practice have dug deep into the problems to find repeating patterns. In finding these patterns, they now understand the complexity to such an extent that they know how to reduce the complication by making their code reusable. And once your code is modular, you can move much more quickly as a company.