The Matrix can MULTIPLY?! The SHOCKING truth Big Data doesn’t want you to know (pt 1) (HEARTBREAKING)

Posted on by pharpend

This post is lecture notes for this lecture I gave on zxq9’s stream on 2026-07-13

Our plan is to do a stream every week on American Monday at 8:00 PM Eastern Time (9:00 PM Realistic Time). We’ll see how long that plan lasts.

We’re going to rotate who’s speaking every week. Next week’s speaker is Jeff Thomson. I’m not sure what his topic is, but his background is investment and finance, so probably it will be something about how money or the economy works. I would ask him, but I changed my mind and switched topics 48 hours before the show, and I want to afford him the freedom to do the same. Jeff has written extensively on gajumaru.io and appeared on my YouTube channel numerous times.

Bonus Geogebra applets that I pulled up on stream but didn’t make it into the post:

Embedding them is a huge pain in the ass so I will settle for linking them for now.

Disclaimers

  • I’m going to default to under-explaining things rather than over-explaining.
  • Your end of this bargain is that you have to be very aggressive about asking questions when you don’t understand something.

The ideas being:

  1. mitigate scope creep
  2. I am bad at predicting what other people will not understand
  3. out-of-scope or answer-is-too-long can be topics of future lectures

Setup

Let’s suppose we have a table of height and weight measurements for individual persons1

SEX     HEIGHT (IN)     WEIGHT (LBS)
------------------------------------
0       72              155
0       67              145
0       65              125
1       67              120
1       63              105
...     ...             ...

First observe that there are two different types (categories? taxa? parts?…) of numerical variables:

  1. sex is a categorical variable (what programmers would call an “enum type” or a “union type”) masquerading as a number

    -type sex() :: m | f.

  2. height and weight are truly numerical values, which in the Large Data Analytics world are called continuous variables.

Using our brains we can surmise the following:

  1. The reason the data includes the sex variable is that men (0) and women (1) probably have different respective relationships with their height and weight

  2. Taller people are probably heavier in general than shorter people

  3. We should verify these things that are probably true

The first rule of Large Data Analytics is Always Visualize Your Data

The second rule of Large Data Analytics is Always Draw The Best-Fit Line

Problem

Some questions present themselves:

  • Where does that line come from?
  • How is it computed?
  • How do we know the fit is best?
  • What does it mean for the fit to be best?
  • Is the line good or bad?

Nobody knows the answer to these questions, so it’s best not to ask them. But, according to Big Data, the best-fit line comes from np.polyfit(...) in the special case when deg=1. Unfortunately, nobody except Big Data knows what np is, beyond the fact that np is what happens when you import numpy as np. So that lead is a dead end.

For old times sake, I propose an adventure. We’re going to larp as people from 2009 and think by hand instead of having Big Data do our thinking for us.

Setup for Solution

The first thing we’re going to do is change our problem a bit to collapse out non-essential detail. Specifically:

  1. Re-center our data so the center-of-mass (mean {X, Y} coordinate) is at {0, 0}.

    It seems like a fair assumption that the best-fit line should pass through the center-of-mass of the data.

    Although this is an assumption, so again we will put a pin in this assumption and check it later.

  2. We’re also going to drop the male/female separation. All that matters for the purposes of our adventure is that we have some 2-dimensional data that kind of has a linear-ish relationship.

And I want to be clear that I specifically mean non-essential in a very literal sense. We are going to trim our problem so that we can focus on the essence of the problem at hand: what is the best fit line and how is it computed?

Whatever detail there is that distracts our attention from the essence of the problem is fat that needs to be trimmed out.

It’s pretty easy to deal with accounting overhead once you have a strong command of the essence of some idea. It’s hard to understand the essence of an idea if your brain is overburdened with keeping track of pointless detail.

On that note, the biggest takeaway I have so far writing all this out is that the essence of statistics is in the art of making tasteful assumptions. In this light, I would say that the essence of math is in the art of making tasteful analogies.

Alright so here’s our new data, visualized:

Side quest: refresher on some geometry

Let’s introduce some geometry terminology

  • Quadrance (aka distance squared). The quadrance of a vector A = {X, Y} is the numerical quantity

    q(A) = X*X + Y*Y.

    This is the area of the square with one corner at {0, 0} and the opposite corner at {X, Y}.2

    The square root of the quadrance is the Pythagorean (crow-flies) distance from {0, 0} to {X, Y}.

  • Inner product (aka dot product, scalar product): the inner product of two vectors A = {X1, Y1} and B = {X2, Y2} is the numerical quantity

    ip(A, B) = X1*X2 + Y1*Y2

  • In physics/engineering contexts the inner product is usually thought of as

    ip(A, B) = |A| * |B| * cos(AngleBetweenThem)

    where |A| means the magnitude of A (square root of its quadrance). I think that equational identity is called the law of cosines but the law of cosines might be something else.

  • If you forgot what sine and cosine are:3

  • The physics view is particularly helpful when you focus on the special case where A and B are points on the unit circle (i.e. their magnitudes are both 1).

  • Notice some very important special cases:

    1. The inner product of a vector with itself is its quadrance (cos = 1).
    2. The inner product of a vector with minus-itself is minus-its quadrance (cos = -1).
    3. The inner product between two vectors is 0 precisely when they are orthogonal (cos = 0).

  • Wildberger focuses on the quantity

    cross(A, B) ->
    div(square(ip(A, B)),
        mul(q(A), q(B)))

  • The cross is the cosine squared.

  • Cosine always ranges between -1 and 1, so the cross always ranges between 0 and 1.

  • It’s useful to think of the cosine as measuring the directional similiarity of two vectors:

    • 1 when the directions are the same
    • 0 when orthogonal
    • -1 when the directions are the opposite

  • In this light, it’s useful to think of the cross as measuring similarity of lines; this is almost the same notion as direction, but lines collapse the distinction between forward and backward.

  • If we want to be pedantic, the cross measures the similarity of pencils, where a pencil is a family of parallel lines.

  • The cross measures directional similarity between lines insofar as it is:

    • 1 precisely when the two lines are parallel;
    • 0 precisely when the two lines are orthogonal.

  • Again, ground yourself: it’s just the dot product squared normalized by the lengths of the vectors.

  • It’s capturing the same basic notion as the dot product (rough parallel/perpendicular notion), but it’s collapsing out the accounting details having to do with absolute direction and the sizes of the vectors.

  • In physics terms, Wildberger

    1. starts with the physics view:

      ip(A, B) = |A| * |B| * cos(AngleBetweenThem)

    2. squares both sides

      sq(ip(A, B)) = q(A) * q(B) * cross(A, B)

    3. divides out the quadrances to get the cross

  • The cross is useful for our purposes because it’s a purely algebraic quantity.

  • Concretely, that means it can be computed without resorting to approximation and floating-point arithmetic.

  • Notionally, it means that the cross is more amenable to the use of high school algebra to connect concrete logic with visualizations.

  • Algebra is generally more “portable” than non-algebra. It’s a more powerful tool to be able to precisely relate things to other things, which is kind of what math is about.

  • The only trig identity most of you remember is the Pythagorean theorem pretending it’s trig:

    sin^2 + cos^2 = 1

  • The cross is cosine squared; the complementary sine-squared quantity is called spread:

    spread(A = {X1, Y1}, B = {X2, Y2}) ->
    ((X1*Y2 - X2*Y1) ^ 2) /
    (q(A) * q(B))

  • That inner quantity X1*Y2 - X2*Y1 is called the twist or the determinant. It is the (signed) area of the parallelogram spanned by the two vectors:4

  • Notionally, the spread is separation between lines, just as quadrance is separation between points.

  • In rational trigonometry, spread plays roughly the role that angles do in irrational trigonometry.

  • So to keep track of the rough analogies:

    cos ~> SquaresTo ~> cross  <~ NormalizesTo <~ dot product
sin ~> SquaresTo ~> spread <~ NormalizesTo <~ determinant

  • Much in the same way that the dot product captures the “raw similarity” between two vectors, the determinant captures the “raw independence” between them.

  • The notion of linear independence is crucially important, but in its full generality it really only makes sense when it’s time to generalize to higher-dimensions. It’s kind of trivial in 2d

  • The determinant also generalizes to volume measurements in higher dimensions (so it requires 3 vectors in 3d, 4 in 4d, etc).

  • Bottom lines are:

    • a finite set of NON-ZERO vectors V1, V2, ... VN is linearly dependent if V1 is in the span of V2, ..., VN, meaning there’s some linear combination (scalar-weighted sum) of

      S2*V2 + S3*V3 + ... + SN*VN = V1

      the vectors are linearly independent if this is false; i.e. V1 isn’t in the span of V2, ... VN

      in 2 dimensions this just means two vectors are dependent if and only if they point in the same direction (one is a constant multiple of the other), and they’re independent if they point in different directions

      in 3d this means that the third vector is not in the plane spanned by the first two

    • an equivalent statement is that a set of vectors is linearly independent if the only linear combination of them that equals the 0 vector is the one where every coefficient is 0

      algebraically, this is a trivial equivalence:

              V1 =           S2*V2 + S3*V3 + ... + SN*VN
ZeroVector = (-1)*V1 + S2*V2 + S3*V3 + ... + SN*VN

      in 2d, visualize that there is no linear combination of {1, 0} and {0, 1} that equals {0, 0}, except the trivial one where both are 0

      in 3d, visualize that if V3 is in the plane spanned by V1 and V2 (i.e. V3 is a linear combination of V1 and V2), then it’s possible to get back to 0 simply by using V1 and V2 to get to V3

    • an equivalent statement is that the vectors are linearly independent precisely when the dimension of their span is equal to the number of vectors (span = set of possible linear combinations)

    • an equivalent statement is that the set of N vectors is linearly independent precisely when the determinant of the N vectors is nonzero

      the determinant measures the “volume” of the shape spanned by the vectors. so if the span of the vectors kills one (or more) dimensions, the volume will be 0.

      This is probably easiest to visualize going 3d->2d. if you have 3 vectors that are coplanar, the 3d volume of the weird mutated 3d parallelogram thing they generate will be 0. And likewise if that volume is not 0, then the vectors must not be coplanar

    • regarding the sign: in general, the sign bit of the determinant encodes whether or not the handedness of the V1, ..., VN vectors (depends on order!) is the same or different from the basis vectors

      E1 = {1, 0, ..., 0}
E2 = {0, 1, ..., 0}
...
EN = {0, 0, ..., 1}

    • the determinant depends on order of arguments, and transposing two of them flips the sign. so notice with

      FooX*BarY - BarX*FooY

      if you flip the roles of Foo and Bar you get

      BarX*FooY - FooX*BarY

      which is “minus” the previous expression

    • the sign bit is 0 if Big Data crushed your hand in His hydraulic press and therefore your handedness is irrelevant

  • Algebraically with our definitions

    spread =         (X1*Y2 - X2*Y1)^2
         ---------------------------------
           (X1^2 + Y1^2) * (X2^2 + Y2^2)

       = sine^2

cross  =        (X1*Y1 + Y1*Y2)^2
         --------------------------------
           (X1^2 + Y1^2) * (X2^2 + Y2^2)
       = cosine^2

    The pythagorean identity

    cos^2 + sin^2 = 1

    is simply an algebraic identity (notice common denominators!):

    quad(V1)*quad(V2)
    =   [ dot(V1, V2)   ^2 ]
      + [ twist(V1, V2) ^2 ]

( X1^2 + Y1^2 )*( X2^2 + Y2^2 )
    =   [ (X1*Y1 + X2*Y2) ^2 ]
      + [ (X1*Y2 - X2*Y1) ^2 ]

  • EXERCISE: Do that algebra yourself

    It’s a tedious but extremely valuable exercise to expand out all that algebra by hand and see that yes indeed everything does cancel out.

    I’m not going to write all that out in the notes because I don’t want to. And, let’s be honest, you wouldn’t actually read it. Furthermore, the tiny subset of you who would comb through all that algebra are the same people who are doing the exercise right now instead of reading this paragraph.

  • Also let’s throw in complex numbers for fun:

    Z1 = A + B*i
Z2 = C + D*i
// remember FOIL?
(A + B*i) * (C + D*i)
    =   A*C   // first
      + A*D*i // outside
      + B*C*i // inside
      - B*D   // last (i*i = -1)
    = (A*C - B*D) + i*(A*D + B*C)

    The identity above says that

    q(Z1) * q(Z2) = q(Z1 * Z2)

    Square-rooted: the length of the product of two complex numbers is the product of their lengths. I will not elaborate except to say there’s an analogous algebraic identity in 4 dimensions.

Concrete example

Let’s consider this basic problem from high school: you have two lines and need to find the point where they meet:

Usually the problem is given to you in some form like:

A*x + B*y = P
C*x + D*y = Q

and your task is to find the pair of numbers {x, y} such that the equation is true. Doing the problem mechanically is an invaluable exercise (there’s a reason you spent years in school doing that problem over and over again!).

The equation has a solution if and only if the lines are not parallel. I’m going to try to connect everything I just said above with your childhood trauma.

Let’s focus just on one equation

A*x + B*y = P

And let’s narrow down even to where P = 0. In light of the above discussion, we recognize this as a dot product:

A*x + B*y = 0

This is asking the question “what are the set of vectors {x, y} which are perpendicular to the vector {A, B}”?

Well we know {0, 0} is a solution, so that’s an answer. But that’s a cheap answer, because definitionally the origin is perpendicular to every point.

The answer is that this is precisely the line through the origin which is perpendicular to the vector {A, B}.

Ok well what if P is not zero? You can solve to get y = Mx + B so it must be a line, right? But which line?

I’m going to introduce the term covector for the pair (A, B). The way to think about covectors is that each one corresponds to a family of parallel lines… it’s the family of parallel lines to which the vector {A, B} is perpendicular.

The easiest way to visualize this for now is to draw some pictures keeping the law of cosines in mind:

<V, W> = |V| * |W| * cos(AngleBetween)

think of the inner product <{A, B}, {x, y}> as measuring the “shadow” that {x, y} casts on {A, B}, assuming {A, B} is on the surface of earth (which is flat) and it’s noon.

If {A, B} = {1, 0}, then this “shadow” just measures the x-coordinate.

If {A,B} and {x,y} agree in general direction, then the shadow will be positive. If they’re exactly at right angles, the shadow will be 0. If they disagree, then the “shadow” will be negative.

So far we understand that the equation

A*x + B*y = P

corresponds to the family of parallel lines which are perpendicular to the vector {A, B}.

We can now add another moving part:

A*x + B*y = P
C*x + D*y = Q

We know from being apes that the lines intersect if only if the lines are not parallel:

Well with the covectors in mind, we can now see that the lines intersect (i.e. the equation has a solution) if and only if the covectors are independent.

Generally it’s useful write systems of equations like above as a matrix product:

[A B  * [x   = [P
 C D]    y]     Q]

The rule for matrix multiplication is that (row-column indexing) the RC-th entry in the result is the dot product of row R with column C:5

Matrix multiplication is not symmetrical.

So far we’ve been thinking about the “rows” perspective. And that’s nice because it’s concrete and tangible. An equally powerful—if not much more poweful—perspective is to think instead about “columns”.

That is we think of the problem above as the problem of discovering a linear combination:

[A B  * [x   = x*[A  + y*[B  = [P
 C D]    y]       C]      D]    Q]

Meaning we now think of x and y as “what weights do we apply to {A,C} and {B, D} to get {P, Q}?” The connection to determinants and spans and whatnot outlined above should be a little more straightforward:

  • this equation has a unique solution
  • if and only if the vectors {A, C} and {B, D} are independent
  • if and only if {P, Q} is in their span
  • if and only if the parallelogram traced out by {A, C} and {B, D} has nonzero area
  • if and only if the determinant A*D - B*C is nonzero
  • if and only if no nontrivial linear combination of {A,C} and {B,D} can construct the zero vector.

The other useful perspective is to think of the matrix as a function that maps the source space into its image (this is called the linear transformation perspective).

It turns out that linear transformations map a shape of size S in the source space to a shape of size Determinant*S in the destination space. The sign captures whether or not orientation is inverted. Hence “determinant is the volume of the transformation”.

Back to the campaign: exposing the shocking truth

Let’s back up a minute and look at our original data

SEX     HEIGHT (IN)     WEIGHT (LBS)
------------------------------------
0       72              155
0       67              145
0       65              125
1       67              120
1       63              105
...     ...             ...

So far we have been thinking about this data from the “rows” perspective. That is, we’re thinking about this table as

-record(human, {sex  :: m | f,
                h_in :: integer(),
                w_lb :: integer()}).
-type table() :: [#human{}].

With what we just learned above in mind, we’re going to focus on the columns

Heights = [Humin#human.h_in || Humin <- Table],
Weights = [Humin#human.w_lb || Humin <- Table],

We’re going to take this matrix and multiply it by its transpose. The order matters here. Generally the rule is

AxC * CxB -> AxB

Meaning the middle cancels out, and the dimension of the result is the outer dimensions. The middle dimensions have to agree otherwise we can’t compute the dot product:

So viewing our data as a matrix Data, there’s two possibilities:

72              155
67              145
65              125
67              120
63              105
...             ...

Let’s call the number of entries N, so Data is an Nx2 matrix. If we compute Data * transpose(Data) that’s going to be N x N which might have some information, but it’s too large for us to ascertain anything useful from it because we are not Big Data.

If we instead do transpose(Data) * Data, this is a 2x2 matrix, which when you normalize by (1/N) is called the “variance-covariance matrix”:

(1/N) * transpose(Data) * Data =
    [var(Heights)           cov(Heights, Weights)
     cov(Heights, Weights)  var(Weights)]

If you’ve heard the term “standard deviation”, the variance is its square. In our terminology, the variance is the mean pairwise quadrance between data points (exercise: show algebra).

If you’ve heard people throw around the term “correlation”, its square is the covariance. In our terminology, the covariance is the mean pairwise cross.

We’re almost ready to expose Big Data and talk about those best-fit lines. But we’ll need to talk about matrix factorization first, and learn about fake german words like “eigenvalues”.

So I guess that will have to wait until next time. Heartbreaking.

  1. Data source. No idea if data is real… doesn’t matter for my purposes but it might matter for yours.↩︎

  2. Pythagorean theorem diagram source↩︎

  3. Unit circle diagram source↩︎

  4. Determinant diagram source)↩︎

  5. Matrix multiplication source↩︎

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.