Tree diameter estimation from RGB-D

00Summary

Can one photograph, taken by a field crew standing under the canopy, replace a diameter tape and a species guidebook? I trained a single RGB-D network that segments individual trees, classifies their species and estimates diameter at breast height (DBH), and measured honestly where it works and where it does not.

The final model reaches 62.29% test AP50 for species-aware segmentation, with no measurable segmentation cost from adding the diameter task. Evaluated end-to-end on its own detections, it returns a diameter for 27.1% of eligible test trees (1,690 of 6,233) with a 7.62 cm RMSE. In a controlled comparison where both models are trained from scratch and differ only in channel count, adding depth raises AP50 by 12.92 points. Detection recall falls with camera distance, from 82.4% within 5 m to 34.5% beyond 30 m, while species accuracy among detected trees stays at 98.9% or higher.

01The inventory problem

Forest inventory turns standing trees into per-tree records, chiefly species and trunk diameter, that carbon stock estimates, biodiversity indices and timber yield models are built from. DBH enters allometric biomass equations directly, so an error in diameter propagates into every downstream carbon figure. In Indonesian forests and agroforestry systems these records are still mostly collected by hand with diameter tapes. Cost scales with the number of trees rather than the area surveyed, and species identification needs skilled labour.

Terrestrial laser scanning recovers trunk geometry precisely but costs too much for routine inventory in much of the region. Image-based methods use cheap cameras, but most work close to a single trunk that the operator has chosen. The harder setting is a whole scene seen from where a field crew actually stands: trees occlude each other, and apparent size varies by more than an order of magnitude with distance. This thesis asks whether one network can deliver both measurements there, and where it fails.

02From raw data to results

Most of the work was not the network. It was making the data trustworthy and the comparison fair, in seven stages. The table says what each stage did and which scripts in the public repository carry it out.

The pipeline, stage by stage
Stage	What was done	Scripts
1. Acquire	SPREAD, a third-party synthetic below-canopy dataset rendered in Unreal Engine 5: 13 species in a plantation and a rainforest domain, with a colour image, depth map, instance masks, species and, for a subset, DBH. Obtained from its authors and not redistributed.	`register_combined.pyregister_plantations.pyregister_rainforests.py`
2. Audit	Before any training: each annotation's box compared with its mask to flag trees with a large box but a tiny mask; the union of the annotation masks compared with a semantic tree mask in every image and split, to find trunks left out; trunk pixel widths and the share of blank DBH values scanned per species.	`gt_bbox_seg_audit.pycoco_vs_semantic_full.pyscan_trunk_px_distribution.pygenerate_dataset_report.py`
3. Repair and select	Masks regenerated from the raw instance images into a new folder, never overwriting the original and guaranteeing each new mask contains the old one. Plantation masks needed it; rainforest masks were already exact and were copied unchanged. Missing main-tree annotations were added. The paper reports 416 masks (0.8% of instances) repaired: +1.7 AP50 for a pretrained RGB configuration, unchanged within noise for RGB-D from scratch. The species for diameter supervision were chosen from the data: those with more than half of DBH blank were excluded (Lemon 86.9% blank, LeechVine 66.4%, a liana), which leaves eleven.	`repair_coco_tolerant.pyrepair_coco_exact_v4.pyfix_main_tree_annotations.pycreate_clean_dbh_json_v3_universe11.py`
4. Prepare	Splits made by rendered scene, with a validator that each training script calls to confirm the stratified split is used; a 5% scene-aware sample for quick smoke tests; depth upsampled with nearest-neighbour rather than bilinear so edges between trunk and background do not get invented in-between values; and a script that widens a pretrained 3-channel stem to four channels for RGB-D.	`make_splits.pycreate_stratified_splits.pysplit_validator.pymake_sample.pyupsample_rf_depth_960.pyinflate_stem_rgbd.py`
5. Experiment	Staged sweeps on a SLURM GPU cluster over the annotation filter, backbone, learning rate and image size; six backbones plus a Mask R-CNN reference; augmentation variants (mosaic, copy-paste, class weights); and a diameter head rebuilt through several versions (v2 to v7 in the repository) before the final joint model.	`run_sweep_phases1to4.pytrain_combined.pytrain_combined_mosaic.pyfinetune_trunk_roi_dbh_hybrid_joint.py`
6. Evaluate	Diameter scored end-to-end on the model's own detections; every checkpoint evaluated, not only the last; recall split by camera distance and by occlusion ratio; a near-miss versus true-miss analysis to ask whether lost recall comes from occlusion; a test that degrades the depth channel to see how much the model relies on it; a score-threshold sweep; and parameters, FLOPs and speed per backbone.	`eval_end_to_end_dbh.pyrecall_per_jarak.pyocclusion_robustness.pynearmiss_occlusion_analysis.pyeval_depth_degradation.pythreshold_sweep_figure.pymeasure_flops_fps_v4.py`
7. Report	Every figure regenerated from cached predictions; a model-selection tracker kept up to date during the runs; and a paper that states what did not work.	`thesis_figures.pygen_final_model_tracker.pygen_qualitative_predictions.py`

Where the controlled comparison came from. The script that widens a pretrained stem records an early run in which the pretrained RGB-D model scored below the one trained from scratch (41.7 versus 50.4 AP50). That is the same effect the paper's controlled comparison later isolates, and it is why every depth comparison trains both arms from scratch.

Scripts are in code/scripts. Many hardcode the cluster paths used during the runs, and some comments are in Indonesian.

03The model

The architecture builds on MaskDINO, a query-based instance segmentation model, with a FocalNet-L backbone. The input convolution is widened from three to four channels so that depth enters as an input. A second head estimates diameter from backbone features at the image row that corresponds to breast height.

Architecture diagram. RGB and depth form a four-channel input to a FocalNet-L backbone producing four feature levels, res2 to res5. A pixel decoder and nine transformer decoder layers with 300 queries and 100 denoising queries output 13 species classes, boxes and masks. A trunk-region head reads the res2 feature at the 1.3 metre row, through a detached copy, and outputs diameter at breast height in centimetres. — **Figure 1.** Architecture of the final model, from the paper. The diameter head reads the same `res2` feature as the segmentation branch through a stop-gradient, so its loss never reshapes the backbone. Depth enters twice: as an input channel, and as the geometry that converts pixel width into physical width.

The diameter head uses depth to find the image row whose back-projected height is closest to 1.3 m, isolates the trunk columns nearest the camera in that row, and computes a purely geometric diameter from pixel width and distance. That value is both reported as a baseline and supplied to a small network that regresses the final diameter together with learned features and the species. The geometric baseline alone is about a factor of two worse than the learned head.

04Data and experimental design

Training and evaluation use SPREAD, a synthetic below-canopy dataset rendered in Unreal Engine 5 (Feng, She and Keshav, 2025). A virtual camera about 2 m above ground looks laterally into plantation and rainforest scenes, giving a colour image, a per-pixel depth map, instance masks, a species label and, for a subset, a recorded DBH. Thirteen species are used for segmentation; diameter is supervised on the eleven that carry recorded DBH. The split is by rendered scene, not by image, so the same tree never appears on both sides. The test split has 7,925 annotated tree instances. I trained and evaluated on a SLURM-managed HPC cluster with NVIDIA L40S GPUs.

Controlling for pretraining

RGB and RGB-D are usually compared with the RGB model initialised from ImageNet and the RGB-D model trained from scratch, because a four-channel stem cannot inherit three-channel weights. That mixes depth with initialisation. In the paper, pretraining raised the RGB model's AP50 by 4.29 points but lowered the RGB-D model's, because ImageNet features that help a three-channel input work against a four-channel one. A comparison with both arms pretrained would report a depth advantage of only 0.91 points. Only with pretraining removed from both arms does the 12.92-point effect appear, so every depth comparison here trains both arms from scratch and varies only the channel count.

05Results

Depth helps in every MaskDINO configuration

Figure 2. AP50 at 75,000 training iterations, with and without the depth channel, both arms trained from scratch (paper, Table 3). Depth adds between 9.68 and 16.93 points across the six backbones; a Mask R-CNN reference shows no measurable effect (+0.32) and sits far below the MaskDINO family. Ablation values sit below the final model's 62.29%, which uses 155,000 iterations.

Detection, not recognition or measurement, is the bottleneck

Because depth is an input, every ground-truth tree has a physical distance that does not depend on model weights, so results can be split by distance without retraining. The three abilities the system needs behave very differently.

Figure 3. Detection recall, mask IoU and species accuracy by camera distance on the test split (paper, Table 6). Species accuracy and mask IoU are conditional on the tree having been detected.

Data behind Figure 3
Distance	Trees	Recall (%)	Species correct (%)	Mask IoU (%)
0-5 m	1,889	82.4	99.6	79.1
5-10 m	1,375	79.3	99.7	75.8
10-15 m	744	70.4	99.8	69.8
15-20 m	472	59.5	100.0	65.4
20-30 m	501	46.7	99.6	63.4
over 30 m	267	34.5	98.9	61.0

Of 7,925 annotated test trees, 4,913 were detected and 4,891 of those were also named correctly. The 3,034 failures break down into 3,012 trees never detected and 22 detected but misnamed, so misclassification is 0.7% of all failure. A single recall figure of 62.0% is a mixture of 82% nearby and 35% far away, weighted by how far away the trees in a given dataset happen to be, which is why two studies reporting the same aggregate number on different stands are not reporting the same thing. Recall also falls steadily with occlusion, from 74.8% for barely occluded trees to 44.3% for heavily occluded ones.

Diameter

Evaluated end-to-end, on the model's own masks and species, the model returns a diameter for 27.1% of eligible test trees (1,690 of 6,233), with 7.62 cm RMSE, 4.89 cm MAE and a bias of -1.28 cm. Absolute error rises with distance, but so does mean tree diameter, because small distant stems are the ones that are missed; relative RMSE stays roughly flat at 20-28%. A plot of predicted against true diameter has a slope of 0.74, which is regression towards the mean: small stems are over-estimated and large ones under-estimated, the expected behaviour of a regressor whose input cannot separate neighbouring diameters at this resolution.

What did not help

Raising the diameter loss weight after segmentation had formed cost about 5.9% in segmentation loss and brought no measurable diameter improvement. The evidence points to interference between the two objectives in the shared features, not to a weak diameter signal. The final model therefore keeps a constant weight.

Input scene: a dense rainforest from ground level. — **Figure 4.** One rainforest scene from the test split, from a single forward pass at score threshold 0.30. Labels read n/a where the head found no valid breast-height row (typically an occluded trunk) or where diameter is not defined, as for a liana. Boxes of neighbouring stems overlap heavily while their masks stay separable.

The same scene with ten annotated tree instances in distinct colours. — **Figure 4.** One rainforest scene from the test split, from a single forward pass at score threshold 0.30. Labels read n/a where the head found no valid breast-height row (typically an occluded trunk) or where diameter is not defined, as for a liana. Boxes of neighbouring stems overlap heavily while their masks stay separable.

06Limitations

Synthetic data only. There is no independent real-world evaluation. No public under-canopy dataset shares SPREAD's species, so a transfer test today would confound domain shift with taxonomic shift.
Coverage. The 7.62 cm RMSE applies to the 27.1% of eligible trees for which the model returns a diameter. Most of the rest are trees it never detects, or detects but cannot resolve at breast height. The mean per-species R-squared is only 11.88% (8 species with at least 30 trees).
Resolution floor. At 480x270 pixels, one pixel spans 8.33 cm at 20 m, the same size as the reported error. A method cannot express a trunk width more finely than the image quantises it.
Diameter definition. Four of the eleven supervised species are small fruit trees that branch at or below breast height, so a single diameter at 1.3 m is not what a field crew would record for them.

07Why it matters for forest inventory and carbon accounting

The practical aim is a low-cost alternative to terrestrial laser scanning: a single commodity RGB-D camera and one network, so that resource-limited forestry agencies and smallholder programmes could track stand structure and carbon stock without specialised equipment or field taxonomists. Because DBH feeds allometric biomass equations, the work is aimed at the measurement that most directly affects carbon estimates. It is a feasibility study on synthetic imagery. Closing the sim-to-real gap is the first step before any operational use, and the distance and occlusion analysis shows where an operational system would need improvement first: detection.

08My role, credits and code

Thesis author, supervised by Dr. Muhamad Risqi Utama Saputra, who is co-author on the manuscript.
I designed the experiments and wrote the full training, evaluation and analysis pipeline in Python (PyTorch, Detectron2).
Builds on MaskDINO (IDEA-Research, Apache-2.0), FocalNet and the SPREAD dataset by Feng, She and Keshav; the repository contains only my patches and scripts, not those projects or the dataset.
The code is released as a record of what was run, not a plug-and-play package: many scripts hardcode cluster paths. Trained checkpoints are available on reasonable request.

Repository MaskDINO (upstream)

Back to projects Next: student retention

Multitask learning for tree instance segmentation and diameter estimation from RGB-D imagery