Tensors and Shapes

This concept is your first step into the world of modern AI. Every AI model you've heard of — ChatGPT, image recognition, voice assistants — works with data as tensors: multi-dimensional arrays of numbers. Understanding what a tensor is and how its shape (its dimensions) flows through a computation is the single most important debugging skill you'll develop. It unlocks everything: when you later learn about neural networks, attention, embeddings, and transformers, you'll constantly check and reason about tensor shapes. This page gives you the foundation to understand why a 'shape mismatch' error happens and how to fix it, and it's the key that opens the door to the rest of your AI learning journey.

You already use tensors without knowing it. Think of a spreadsheet in Excel. It's a table with rows and columns — that's a 2-dimensional array. Now think of a stack of those spreadsheets, like a stack of papers on a desk. That's a 3-dimensional array — an array of arrays. A tensor is just the mathematical name for any multi-dimensional array of numbers. It can be 1-dimensional (a list of numbers), 2-dimensional (a table), 3-dimensional (a stack of tables), or even more. What makes tensors special in AI is that they carry a shape, which is a list of numbers that tells you how many items exist in each dimension. For example, a spreadsheet with 10 rows and 5 columns has shape (10, 5). A stack of 20 such spreadsheets has shape (20, 10, 5). The shape is not just a label — it's a promise about the structure of the data. Every time you perform a computation on a tensor (whether that's adding, multiplying, or applying a neural network layer), the shape of the output is determined by the shapes of the inputs. Tracking this is like checking that your screws fit before you tighten them — you need to know which dimension means what.

Think of a warehouse. The warehouse stores crates of identical items. A crate is a 2-dimensional box (length and width). To store many crates, you stack them on shelves. A shelf is a 1-dimensional row of crates. To organize many shelves, you line them up in aisles. Now you have a 3-dimensional structure: aisles (dimension 1) contain shelves (dimension 2) each containing crates (dimension 3) each with items (dimension 4? No, crates are like data elements, so let's keep it simple). Actually, let's use a more precise analogy: a library. A single book is a 1-dimensional array of pages (if you just list the pages in order). A bookshelf is a 2-dimensional array: shelves (first dimension) and pages within a book (second dimension, but it's nested). A room full of bookshelves is a 3-dimensional array. To find a specific page, you need to know the room, the shelf, and the page number — that's three indices. A tensor's shape tells you the size of each of those dimensions. When you do a computation, it's like taking books from one shelf and placing them on another — you need to make sure the shelf sizes match, otherwise you can't place them properly. The shape is the contract that ensures your operations make sense. The analogy breaks when you go beyond 3 dimensions — a tensor can have 4, 5, or even 50 dimensions. In a physical library, you can't have a sixth dimension of organization, but in a tensor you can. That's okay — the conceptual model of 'a list of lists of lists' extends naturally.

A tensor is a multi-dimensional array of numbers, and its shape is the tuple of integers giving the number of elements along each dimension.

You know that in everyday life, a list of numbers is just a list, and a table of numbers is a table. A tensor generalizes this: it is a container that can hold numbers in any number of dimensions. This is your first formal concept in linear algebra, which is the branch of mathematics that deals with vectors and matrices. In AI, data is always organized as tensors — a batch of images, a sequence of words, a table of features. The shape of a tensor is like the dimensions of a box — it tells you how many items fit in each direction. This will directly connect to matrix multiplication later, where the shape rules determine whether an operation is even allowed.