When one camera cannot count stacked objects

The first article counted separated objects with a threshold on a strip of pixels and no model at all. The second removed the separation assumption, and a detector earned its cost by finding the seams a single column of pixels could not see. Both articles kept one assumption quietly in place, and this article removes it: the objects lie flat on the belt, in one layer.

Now let them stack. This looks like the next step in the same progression - separated, then touching, then stacked, each a bit harder.

Touching objects were a hard inference problem: the seam between two objects is faint, few pixels wide, and the model has to learn what a boundary looks like. It is genuinely difficult, and difficulty is what a detector is for.

Stacked objects are not a hard inference problem. They are a missing information problem. The number you are asking for is not in the image, and no amount of image model capacity puts it there. That difference decides everything downstream, so the first section proves it rather than asserting it.

Why one camera cannot do this

Its not that it “struggles with.” It just cannot. The issue is geometric.

A camera maps a 3D scene to a 2D image. Every point in the world lands somewhere on the sensor, and the map throws away one coordinate: the distance along the ray from the lens. Two points on the same ray - one nearer, one further - land on the same pixel. This is projection, and its lossiness is not a defect of cheap cameras.

A top-down camera cannot distinguish one object from $n$ identical objects stacked on it, because both scenes produce the same image. The count is not recoverable from a function of the image, because the image is the same in both cases.

A detector - any detector, at any size, trained on any dataset - is a function of the image. If two different worlds produce one image, then every function of that image returns one answer for both, and it is wrong in at least one of them. Model capacity is irrelevant. A bigger model cannot separate inputs that are equal.

This is the same reason the previous article’s approaches do not extend. The threshold failed on touching objects because a single column could not see a seam running along the belt - a geometry of sampling problem, fixed by sampling an area instead of a line. The detector fails on stacked objects because the seam is not in the image at all. There is no sampling geometry that fixes it, because you cannot sample what was never recorded.

Note that we are assuming the objects to be identical and exactly aligned. Real stacks are not. A real stack of three boxes shows a slightly larger silhouette, a shadow at the overhang, a lower brightness because the top face is further from the light, and a change in defocus because it sits closer to the lens.

So the real statement is not “a top-down camera has zero information.” It is:

A top-down camera has no direct information about the count. What it has are second-order cues, and every one of them is weak, product-specific, and degrades toward zero as the stack becomes more regular.

That distinction matters commercially, because those cues are exactly strong enough to build a demo that works and a deployment that does not. Someone will label a few hundred images of one-high and two-high stacks, get 95% on a held-out split, and ship it. What the model learned was that two-high stacks are 8% larger in silhouette and slightly darker. Then the product changes, or the operators start stacking more neatly, or a new lamp flattens the shadows, and the accuracy falls off a cliff nobody can explain from the code.

A cue that shrinks toward zero is not something to build a counter on. It fails in exactly the conditions everyone is working to bring about.

Four ways to buy back the dimension

If the count needs a coordinate the image threw away, the fix is to buy that coordinate back. There are four ways to do it, and they are in the order I would actually try them - which is not the order of technical interest.

1. Do not let them stack

The same move as the speed-up belt in the previous article, and it is the right answer more often than the other three combined. A singulator, a vibratory feeder bowl, a step-down between two belts, a stack-height limiting bar across the infeed that simply sweeps off anything above one layer.

The economics are brutal in its favour. Destacking hardware is a few hundred to a few thousand dollars, once. Every other option on this list is a permanent piece of the system with a permanent owner. And it collapses this article back to the previous one: separated objects, threshold, $237.

The reason people skip it is that it changes the mechanical line, which needs a different team and a shutdown window, whereas mounting a camera does not. That is an organisational cost being paid with an engineering one, and it is worth naming out loud when it happens.

2. A second camera at a different angle

If the objects must be counted as they lie, add a viewpoint whose rays are not parallel to the stacking direction. Objects stack vertically, so a side camera at belt height sees the layers separated horizontally - the seam invisible from above is a strong edge from the side.

This is the direct answer to “what does a second viewpoint cost,” and the cost is not the camera.

The camera is $50–$75. It is the same module and lens as the first one, and this row is the one everybody budgets for.

The mounting is the hard part. A side view at belt height means a camera looking across the belt, which means the background is the rest of the factory - people walking past, a forklift, another line. The top-down camera had the belt as a controlled background for free. The side camera has to be given one: a plain backdrop panel on the far side of the belt, in a colour the product is not. Budget the panel, the bracket, and the argument about who owns the two feet of floor it stands on.

Lighting doubles, and it gets harder. The side view needs its own illumination, and now the two lights are in each other’s fields of view. A side light aimed across the belt puts glare into the top-down camera unless it is baffled.

The two counts have to be reconciled, and that is the actual engineering. Two cameras give two independent counts of the same objects. They will not agree. That disagreement is the whole problem, and the next section is about it.

3. Measure height directly

Rather than inferring depth from a second colour image, measure it. Three ways, in increasing cost:

A laser line profiler, which projects a line across the belt and views it at an angle. The line’s displacement in the image is directly the height. This is structured light, and for the specific job of “how tall is this pile” it is far more robust than a second colour camera, because it does not care about product colour, print, or belt cleanliness. A cheap implementation is a $30 line laser and the camera you already have.
A time-of-flight or stereo depth camera (RealSense and similar), $150–$400, giving a height map of the whole belt at once. More data than the profiler; more sensitive to shiny and dark surfaces, which absorb or scatter the projected pattern.
A photoelectric height gate - a cheap through-beam sensor a fixed distance above the belt, wired straight to the PLC. It answers exactly one question: is anything taller than one object passing? That is not a count, but as we will see it may be all you need.

Height is a much better signal than silhouette because it is linear in the count and independent of the failure modes above. A stack of $n$ objects is $nh$ tall, and unlike a silhouette that cue does not shrink as the stack becomes more regular - it gets cleaner. That is the opposite behaviour, and it is the reason to reach for depth before reaching for a bigger detector.

The catch is that height only counts objects whose thickness is consistent, and only when they stack in registration rather than at angles. A neat stack of three trays is $3h$ . Three trays in a jumbled pile is some height between $h$ and $3h$ that divides into nothing meaningful. Height converts the problem from impossible to the run-length division problem from the previous article, in the vertical direction - with the same $\epsilon k + \sigma\sqrt{k} < 0.5$ inequality governing when rounding survives, and the same warning that calibration error is what sets the ceiling.

4. Count somewhere else entirely

The most underrated option, and often the cheapest.

The stacking is happening at one point on the line. It is not happening everywhere. If there is any point upstream or downstream where objects are guaranteed single-file - a feed chute, a labeller, a wrapper, a robot pick station that by construction handles one at a time - count there and use the previous articles’ solution unchanged.

The requirement to check is conservation: everything that passes the counting point must reach the point you actually care about, with no reject gate, no manual removal and no accumulation between the two. If a reject gate sits between them, count after it, or count the rejects too and subtract.

I have seen this solve the problem outright twice: once by moving a camera four metres downstream to a point where the product went single-file into a wrapper, and once by taking the count from an existing PLC signal on a pick-and-place that was already, in effect, counting perfectly and telling nobody.

Reconciling two counts

Take option 2 and assume both cameras are running. Top-down gives $n_{\text{top}}$ - the number of footprints, correct in the plane. Side gives $n_{\text{side}}$ - the number of layers, correct in height. Neither is the count because the two cameras are not measuring the same quantity. They are measuring two factors of it.

For the clean case - a single pile of $k$ objects, all stacked, none side by side - the top-down camera sees one footprint and the side camera sees $k$ layers, so the count is $n_{\text{side}}$ and the top-down number is 1. For a single layer of $k$ objects side by side, top-down sees $k$ and side sees 1 layer. The general case is a grid, and the count is the product:

n = n_{\text{top}} \times n_{\text{side}}

Do note that it will give correct count only when the pile is a full rectangular block - every column the same height. If piles are ragged, two objects side by side with a third on top of the left one gives $n_{\text{top}} = 2$ , $n_{\text{side}} = 2$ , product 4, truth 3.

The multiplication is an upper bound, not a count. Which leads to the honest framing of this whole section:

Two cameras turn an impossible problem into a hard one. They do not turn it into a solved one.

The regime where two cameras genuinely solve it is narrow and worth stating so you can check whether you are in it: stacks that are regular, of known maximum height, and separated from each other along the belt. Trays in columns of at most three, arriving in discrete piles. There, the product rule holds, and the system works for years.

Outside that regime - a random heap of loose parts - two cameras do not give you the count either, and neither does a third. The occluded interior of a pile is not visible from any exterior viewpoint. If the objects in the middle of the pile cannot be seen from outside it, no arrangement of cameras counts them.

That is the case worth spending a section on, because it is the common one. Clean columns of trays are the exception; most lines that stack, stack badly.

The synchronisation problem

Two cameras counting the same objects have to agree on which objects. The top-down camera sees a pile at frame 1,200; the side camera sees it at frame 1,207 because it is mounted 80 mm further down the belt and its exposure started few ms later. Pair the wrong pile with the wrong pile and the product rule multiplies two unrelated numbers.

Three solutions things to get this right are:

Trigger both cameras from one source, so that “now” means the same thing to both. A shared hardware trigger is the standard answer, and it is what industrial cameras have a trigger input for.
Better, trigger on belt position rather than on time. A rotary encoder is a wheel riding the belt that emits a pulse every fraction of a millimetre travelled. Counting its pulses tells you how far the belt has moved, regardless of how fast it moved.

Why that beats a timer is worth spelling out, because it is the same lesson as “belt speed cancels” from the first article. Say the two fields of view are 80 mm apart. On a timer you must delay camera B by $t = d/v$ - at 500 mm/s, 160 ms. That number is only right at that one speed. Let the belt slow to 250 mm/s and the correct delay becomes 320 ms while your timer still fires at 160, so camera B photographs a point half-way to the pile and you pair one pile’s footprint with the next pile’s height. On an encoder you say “fire camera B 80 mm after camera A,” which is a statement about the geometry of the mounting and is true at every speed.

A timer encodes an assumption about belt speed; an encoder encodes the physical layout. Only one of those stops being true when a VFD ramps, a motor loads up, or someone turns the line down for a changeover.
Calibrate the offset between the views in belt distance, not in time. The physical distance between the two fields of view is fixed; the time between them is not, because it depends on belt speed. Store the offset in millimetres and convert.
Give each pile one identity across both views - that is, know which of camera B’s height readings belongs to which of camera A’s footprints. The product rule only means something when both numbers came off the same physical pile: multiply pile 7’s footprint by pile 8’s height and you get a confidently wrong answer with nothing in the data to reveal it.

This is the tracking problem from the previous article rotated by ninety degrees. There the question was is this the same object as last frame, and it stopped one object being counted fifteen times. Here it is which reading over there corresponds to this one over here, across two cameras rather than across time.

On a single lane it is nearly free: belts do not reorder things, so piles cross camera A in the order 1, 2, 3 and cross camera B in the same order 1, 2, 3, and a queue is the entire algorithm. On a wide belt it is genuinely hard - three piles cross A at once, and deciding which of B’s three readings matches which of A’s three footprints needs the two cameras to agree about position across the belt, from different angles. That is a calibration problem, not a queue, and it is the point where the honest recommendation reverts to option 1.

When the piles are haphazard

Everything above assumed stacks that are at least trying to be columns. Now take that away too: objects dropped loosely, at angles, in heaps of no fixed height. Layers do not exist, so there is nothing for the side camera to count, and HEIGHT_ONE divides into nothing.

The instinct at this point is that the problem got harder in the same direction, and it did not. It changed shape, and the change is worth naming before reaching for anything:

On regular stacks you are counting instances and can expect to be exactly right. On haphazard piles you are measuring a quantity and can only expect to be right on average. These are different jobs with different success criteria.

Why the top-down count degrades with load

Obviously you cannot see into the middle of a pile. The part worth a paragraph is how the error behaves as the pile grows, because that decides whether you can correct for it.

A heap holds its objects in three dimensions but only shows you its surface. Its volume has to grow in proportion to $k$ , and surface area goes as the square of linear size:

V \propto k \quad\Longrightarrow\quad \ell \propto k^{1/3} \quad\Longrightarrow\quad A \propto \ell^{2} \propto k^{2/3}

So the number of objects you can pick out grows like $k^{2/3}$ while the truth grows like $k$ . The error is not a fixed offset you can calibrate away; it grows with how full the pile is.

Measure bulk, not instances

If the objects cannot be individuated, stop trying. Measure something that is proportional to the count and divide.

Volume, from the height map. A depth camera gives height over the whole belt, and summing it gives the occupied volume $V$ . If one object has volume $v$ and the pile settles at packing fraction $\phi$ - the fraction of the occupied envelope that is actually object rather than air - then

\hat{n} = \frac{V \phi}{v}

Mass, from a belt weigher. The same estimator with mass in place of volume, and $\phi$ disappears entirely because mass does not care how loosely things are packed. If the objects have consistent mass, an in-motion check-weigher beats every camera in this article. It is the single most under-used answer to the haphazard case: no occlusion, no lighting, no labelling, no packing fraction, and it is a solved industrial product rather than a project. Mass variation is usually far tighter than anything vision could recover from a heap.

Where the weigher does not fit - objects too light, no room to break the belt, mass varies with moisture - volume is the fallback, and the rest of this section is about its one difficult parameter.

Packing fraction is the whole problem

$\phi$ is where the error lives, and it splits the same way everything in this series does: a random part that averages out, and a systematic part that does not.

Loose random packing sits around 0.55–0.65 for compact shapes, varying pile to pile with how the objects happened to land. Take $\phi = 0.62 \pm 0.06$ as a realistic line.

Since the estimate is $V\phi/v$ , a pile that settled looser or tighter than assumed is wrong in direct proportion, so the spread in $\phi$ passes straight into the count: $0.06/0.62 \approx$ ±10% on a single pile. Hopeless. But piles settle independently, so their errors partly cancel and the spread on a total of $N$ piles falls as $1/\sqrt{N}$ - the same identity as the previous article’s $\sigma\sqrt{k}$ . Over a thousand piles that is $10\%/\sqrt{1000} \approx$ ±0.4% on the shift total. That is the payoff of the reframing: useless per pile, genuinely accurate per shift.

The systematic part is the one to worry about. A 5% error in your assumed $\phi$ is a flat 6% error in the total, at every scale, forever - no amount of aggregation rescues a wrong constant. The same lesson as FRAMES_PER_ITEM two articles ago, and the same conclusion: calibration quality sets the ceiling, and calibration drifts after you leave. $\phi$ in particular drifts with drop height, belt vibration and how full the upstream buffer is, so it has to be measured on the line rather than taken from a handbook.

What this changes in the code

The earlier layers function returns None on anything that is not a clean multiple, which is right for regular stacks and wrong here. The haphazard path needs a different contract: always return a number, and always return how much to trust it.

V_OBJECT = 18.4        # volume of one object, cm^3 - from CAD or displacement
PHI = 0.62             # packing fraction - CALIBRATE on the line, never assume
PHI_SD = 0.06          # pile-to-pile spread, from that same calibration run

def estimate_pile(volume_cm3):
    """Objects in one haphazard pile. Never exact - carries its own error bar."""
    n = volume_cm3 * PHI / V_OBJECT
    rel_sd = PHI_SD / PHI                      # dominates every other term
    return n, n * rel_sd                       # (estimate, 1-sigma)

def shift_total(pile_volumes):
    """Aggregate. Random packing error shrinks as 1/sqrt(N); bias in PHI does not."""
    total = 0.0
    var = 0.0
    for v in pile_volumes:
        n, sd = estimate_pile(v)
        total += n
        var += sd * sd                         # independent piles: variances add
    return total, var ** 0.5                   # NOT a bound on PHI being wrong

Three things worth pointing at, and they are the reasons this is a different system rather than a tweak.

It returns a float and an error bar, never an integer. Rounding a single pile to a whole number implies a precision the measurement does not have. The estimate is 4.3 objects ± 0.4, and the moment it is rounded to 4 that uncertainty is gone from the record and someone downstream treats it as a count. Carry the uncertainty all the way to the dashboard.

Variances add, standard deviations do not - the same identity as the $\sigma\sqrt{k}$ argument in the previous article, doing the same work. It is what makes the shift total far more accurate than any pile in it, and it is the entire justification for this approach.

The returned error bar deliberately excludes PHI being wrong. It quantifies the random spread only. A systematic error in PHI shifts every pile the same way, so it cannot show up in a spread computed across piles - the number would look reassuringly tight while being 6% off. That is the failure this whole section is designed to make visible, and the comment is there so nobody reads the error bar as a guarantee it is not.

The honest summary for this case

Haphazard piles are countable as a rate, not as events. Concretely, what you can and cannot promise:

Shift and batch totals to a few percent: yes, with a calibrated $\phi$ and reconciliation against a downstream count.
Per-pile counts: no. Not to ±1, not with more cameras, not with a better model.
Anything that pays a person per piece: no, and this is the one to refuse out loud. The first article’s counter fed piece-rate pay, and a ±10% per-pile estimate must never be used that way. If the count settles wages, the objects have to be singulated. It is an obligation.

Which lands, once again, on option 1. The difference is that here it is not merely the cheaper engineering - for haphazard piles, singulation is often the only thing that produces a defensible number at all. If the count matters, the pile has to stop being a pile.

What I would actually build

Concretely, for a line where objects arrive in regular stacks of at most three on a single lane - the tidy case; the haphazard build is the volume estimator above, plus a weigher if one fits - here is the shape of the system, and note that most of it is not vision.

Belt-height limiting bar, so that anything above three high is swept off rather than counted. Top-down camera and detector, exactly the previous article’s build, giving footprints and their positions. A laser line profiler or a single height gate, giving layers. Then the counting logic, which is short:

HEIGHT_ONE = 24.0      # height of one object, mm - calibrate from the line
HEIGHT_TOL = 0.35      # fraction of HEIGHT_ONE allowed before flagging.
                       # Must be < 0.5: rounding already bounds the residual at
                       # half an object, so 0.5 disables the check entirely.
MAX_LAYERS = 3         # anything taller is a fault, not a count

def layers(height_mm):
    """Layers in one column, or None if the height is not a clean multiple."""
    k = round(height_mm / HEIGHT_ONE)
    if k < 1 or k > MAX_LAYERS:
        return None                                   # out of range: fault
    # fires when the column sits BETWEEN clean multiples - debris under a tray,
    # something tilted or half-seated, or a product whose thickness has drifted
    if abs(height_mm - k * HEIGHT_ONE) > HEIGHT_TOL * HEIGHT_ONE:
        return None                                   # ragged pile: fault
    return k

def count_pile(footprints, heights_mm):
    """Sum layers over footprints. Returns (count, faulted) - never a guess."""
    total = 0
    for h in heights_mm:
        k = layers(h)
        if k is None:
            return None, True          # do not guess; hand it to the operator
        total += k
    return total, False

Three things worth pointing at, and they are the same three every time.

layers returns None rather than a best guess. This is the single most important line in the file, for regular stacks. A stack that should be a clean multiple and is not; has something wrong with it, and the system’s job at that moment is to flag it. A counter that reports its uncertainty is worth far more than one that is quietly wrong, because the quiet wrong one poisons the yield report and creates distrust.

HEIGHT_TOL is the whole calibration, and it has a hard ceiling. It asks how far a column may sit from a clean multiple of one object before the reading is refused - at 24 mm objects and 0.35, anything more than 8.4 mm off a multiple faults. Too tight and every slightly-tilted stack faults out until an operator stops trusting the alarm; too loose and a debris-lifted two-high pile passes as three. The ceiling is the part that bites: because round already puts the residual within half an object, a tolerance of 0.5 or more can never fire - the check becomes dead code and every height is accepted, including the 36 mm reading that is equally likely one tray or two. Tune it against recorded runs.

Then a fault path: the faulted pile is photographed, logged, and either diverted or counted by a person. Budget for the fault rate being higher than anyone promised - that is the number that decides whether the system is used or bypassed after a month.

Cost

The three builds across the series, with the changed rows called out. Same caveat as before: the shape of the list is the useful part, not the numbers.

Item	Threshold	Detector	Stacked build
Compute	Pi 5 4 GB, $60	Jetson Orin Nano, $250	same Jetson, $250
Camera	IMX296, $50	same, $50	two, $100
Lens	$25	same, $25	two, $50
Depth sensing	none	none	line laser + mount, $60
Power	$12	$20	$20
Storage	microSD, $10	NVMe SSD, $40	NVMe SSD, $40
Lighting	$40	same, $40	two heads + baffles, $110
Mounting	$30	same, $30	+ backdrop, second bracket, $90
PLC output	$10	same, $10	+ height gate, $45
Hardware total	$237	$465	$765
Labelling	none	2–5 days	2–5 days, plus side-view set

The hardware roughly triples across the series, and as before that is not the important number. Compare it with a a singulator or a stack-limiting bar of $500–$3,000, paid once, and it deletes the entire right-hand column - the second camera, the profiler, the backdrop, the side-view labelling, the fault workflow - by taking you back to the $237 threshold build. If the piles are haphazard rather than regular, the equivalent is an in-motion check-weigher at $2,000–$6,000, which needs no labelling and no packing constant and simply works. Set against a $765 build that also carries a labelling project and a permanent owner, those win over any horizon longer than a few months.

That is the same conclusion as the previous article, one step further along, and it is worth stating as the rule the series has been building toward:

Each time the objects get harder to see, the vision system gets more expensive faster than the mechanical fix does.

Note also what did not change: the Jetson. The compute was already sized for a detector, and adding a second stream and a height calculation does not move it - the profiler’s arithmetic is trivial next to inference. If you already have the detector build, option 2 or 3 is an incremental hardware cost, not a new architecture. That is genuinely the best argument for doing it in vision rather than mechanically, and it applies only to lines that have already paid for the detector.

How do you know the count is right?

The reconciliation advice from both previous articles applies unchanged and matters more here than anywhere else in the series, because this is the first build whose failure mode is silent by construction. A miscounted stack looks exactly like a correctly counted single object. There is nothing in the frame to see, which is where this article started.

Three additions specific to this build:

Log the fault rate as a first-class number, next to the count. Not in a debug file - on the dashboard, beside the count, all the time. It is the health metric for the whole system. A fault rate that climbs from 2% to 9% over a fortnight is a product change, a lighting change or a lamp aging, and it is visible weeks before anyone notices the yield numbers are wrong.

Log the height histogram. Heights should cluster tightly around $h$ , $2h$ and $3h$ . When those peaks smear or a fourth appears, something physical changed - and unlike the count, the histogram tells you what. Peaks in the right places at the wrong ratio is a stacking-behaviour change; peaks at shifted positions is a product dimension change; smeared peaks is a calibration or mounting problem.

Reconcile against a count taken where objects are single-file, if such a point exists anywhere on the line - which is exactly option 4. If it exists and you are not using it as the primary count, it is at least the audit. Half a day of work to install a $237 Pi at a chute downstream, and it tells you every shift whether the expensive build at the stacking point is telling the truth.

Where the series ends up

Three articles, three assumptions removed one at a time, and the pattern is clearer at the end than it was at any point along the way.

Separated objects: a threshold on a strip of pixels, no model, $237. The information is in the image and it is easy to get out.

Touching objects: a detector and a tracker, $465 plus a labelling project. The information is in the image and it is hard to get out - which is what machine learning is for, and where it genuinely earns its cost.

Stacked objects, regularly: the information is not in the image. No model recovers it. You buy back the missing dimension with a viewpoint, a depth sensor, a different counting location, or a mechanical change - and the mechanical change is usually the correct engineering answer, exactly as it was one article ago, only more so.

Stacked objects, haphazardly: the question itself changes. You stop counting instances and start measuring a quantity - volume, or better, mass - accepting that any single pile is ±10% while a shift total is within a percent. That is a real answer for a yield report and not an answer at all for piece-rate pay, and knowing which of those you are being asked for is the entire job.

The through-line is that the three questions from the first article - how do the objects arrive, what is the throughput, how many are across the belt at once - were never preparatory work before the real work. They were the real work. The first question alone moves the build between $237 and $765, between no model and a permanent owner, and between a solved problem and a managed one. The most valuable hour on any of these projects is the one spent watching the belt before choosing anything.

And the most valuable question in that hour is not “which model” but “can we change how the objects arrive?”