From the idea to the upstream implementation
These two contributions bridge the math a vision model uses and the operations Apple's tools can execute. Here is the background, the implementation, and a practical way to learn from the code.
1. A working model still needs a translation
Training and deployment use different software systems. A model written in PyTorch describes operations such as convolutions, sampling, and activation functions. Apple's CoreMLTools converter translates a supported model into a representation Core ML can use.
A lowering is one piece of that translation: replace a source operation with target operations that express its behavior. MIL, Core ML's Model Intermediate Language, provides those building blocks. An operator-level gap can stop the entire conversion before an app ever runs the model.
I added two Torch frontend lowerings. Deformable convolution issue #1889 and ROIAlign issue #1793 document the missing support. The ROIAlign report dates to March 2023 and shows a detector running in PyTorch before failing at conversion. This is a concrete integration barrier, even when the model itself is valid.
2. Deformable convolution: reconstructing a missing operation
A normal convolution reads an image or feature map at regularly spaced positions and combines the values with learned weights. Deformable convolution adds learned offsets to those positions. With a modulation mask, the model can also scale the contribution from each sample.
The sample locations can fall between pixels. Bilinear interpolation estimates a value by blending four neighboring pixels according to distance. That sampling behavior has to survive conversion, including what happens at the image boundary.
Sampling between pixels. Put values 0 and 10 on the top row of a 2×2 grid, and 20 and 30 on the bottom row. Exactly halfway between the four pixels, each contributes one quarter: (0 + 10 + 20 + 30) / 4 = 15.
My implementation builds that operation from existing MIL primitives:
- Construct the regular sampling grid from kernel position, stride, padding, and dilation. Add the learned offsets.
- Add an explicit zero border and adjust coordinates so interpolation at the boundary matches the intended sampling behavior.
- Normalize the coordinates and use
resamplefor bilinear sampling. Apply the modulation mask when present. - Arrange the sampled values and use a grouped 1×1 convolution to perform the weighted combination. Add the bias when present.
The final 1×1 convolution is doing the weighted combination after sampling; it is not a claim that an ordinary 1×1 convolution alone replaces deformable convolution. Grouping and the order of sampled channels matter to the result.
3. ROIAlign: sampling a selected region consistently
An object detector often needs to extract information from a proposed region of a feature map. ROIAlign divides that region into output bins, samples within them, and averages the samples. Interpolation lets it work with fractional positions instead of rounding every coordinate to a pixel.
My fixed-sampling lowering selects the correct batch image for each box, applies the spatial scale and alignment offset, and constructs sample positions for each bin. It uses resample to obtain the values, accumulates them, and divides by the number of samples.
For each output bin:
place a fixed grid of sample points
interpolate the value at each point
output the average of those valuesFor example, four sample values of 5, 10, 20, and 25 produce an output-bin value of 15. Unlike deformable convolution's learned weighted combination, this step is an average.
The implementation supports tensor-form boxes with a batch index and four coordinates, multiple regions, and both alignment settings. It requires a positive constant sampling ratio. Adaptive sampling is rejected explicitly, so an unsupported setting cannot silently become a different calculation.
4. The difficult part: agreement across representations and runtimes
A graph can convert successfully yet produce the wrong values on the target runtime. In the deformable-convolution review, initial native prediction failures led to a coordinate-mode correction. The merged implementation uses normalized coordinates in the range −1 to 1 with align_corners=True.
The ROIAlign discussion records the same coordinate-mode adjustment and precision-specific test tolerance. It also shows test revisions requested by the maintainer. A useful contributor lesson is to interpret feedback against the actual failing behavior and the project's testing conventions.
| Check | What it establishes |
|---|---|
| Conversion / graph construction | Can the frontend express the operation using supported MIL operations? |
| Numerical comparison | Do supported inputs produce results close to the PyTorch reference at the tested precision? |
| Native Apple execution | Does the converted representation behave correctly on the actual runtime? Linux conversion alone cannot establish this. |
| Rejection tests | Does an unsupported setting, such as adaptive ROIAlign sampling, fail clearly? |
The merged TestDeformConv2d tests vary masks, bias, grouping, kernel shapes, stride, padding, and dilation. The TestRoiAlign tests cover multiple regions and batch indices, scale and alignment choices, and the adaptive-sampling error path. Frontend and precision coverage help reveal differences a single example would miss.
5. How to approach a first converter contribution
- Reduce the conversion failure. Start with one unsupported operation and a tiny input. A full detector introduces too many unrelated places to fail.
- Write down the mathematical contract. Track input shapes, coordinate order, padding, alignment, and aggregation. Decide which options the first implementation will support.
- Find target primitives. Read the registered lowering in
ops.py. Follow how inputs become sample coordinates, then trace those samples to the output. - Build a test matrix around boundaries. Try a non-square input, a fractional coordinate, a boundary sample, a different batch index, and one deliberately unsupported setting.
- Separate conversion from runtime validation. Follow the CoreMLTools contributing guide and identify which checks your environment can actually perform. Report each result with its platform and precision.
Try it yourself: calculate the midpoint sample in the 2×2 example. Then move the sample onto a corner, and finally just outside the image. Before reading the padding code, write down which neighboring pixels should contribute in each case.
6. What this contribution demonstrates
I implemented two mathematical operations across a framework boundary, using existing Core ML primitives and tests against PyTorch. The work required sampling geometry, tensor layout, grouping, numerical precision, and platform-aware debugging.
PR #2752 merged July 15, 2026, and PR #2753 merged July 20, 2026. Those merges resolve two reported conversion gaps and put the implementations in Apple's upstream tooling, where other developers can build on them.