Developers · Vision

From an iPhone camera into a vision model.

If frames are going into code rather than onto a screen, pulling them back out of a broadcast stream is the long way round. A small Python receiver gives you decoded frames directly — and, unusually, a way to send the model's conclusions back to the lens.

Airlive team 6 min read

Why not just grab NDI

You can point a vision pipeline at an NDI source, and for a one-off that is fine. The reasons to receive directly are structural rather than aesthetic:

  1. Fewer conversions. Straight to your decoder instead of decode, re-encode, decode again — less latency and less CPU spent on work you undo.
  2. No extra machine. The republishing path implies something running Bridge. A direct receiver runs on the box your model already runs on.
  3. The control channel. This is the part no video transport gives you: the same connection that delivers frames accepts commands. A model can act on what it sees.

The pipeline, in four stages

  1. Discovery and connection. Advertise _airlive._tcp and accept the camera's TCP connection.
  2. Framing. Read the 18-byte header, then exactly the declared payload length. Buffer — a socket read is not a message.
  3. Decode. Configure your H.264 decoder from the format description, then feed access units. PyAV is the pragmatic choice in Python; anything wrapping FFmpeg works. Remember the video is AVCC — length-prefixed NAL units, not Annex B start codes.
  4. Hand off. Convert each decoded frame to an array and pass it to whatever you're running — a detector, a tracker, a classifier, a pose model.

Decouple decoding from inference. Put a small, bounded queue between them and drop the oldest frame when it's full. Models are rarely as fast as 30fps, and a pipeline that queues without limit turns into a growing delay and then a memory problem.

What people actually build with this

Sports and lecture capture want a subject followed. Venues want people counted without a camera that stores anyone's face. Installations want something to happen when a person walks in. And a quietly useful one: watching your own feed for drift — focus slipping, exposure wandering — and telling an operator before an audience notices.

Closing the loop

The control channel is what makes this more than a video tap. Your model's output can become a command on the same socket: setZoom and setLens to reframe, the focus and exposure commands to correct, setCue to flag a camera as interesting to a human director.

Two cautions from ordinary control theory. Damp your corrections — a camera that chases every frame of detection noise looks far worse than one that never moved; smooth the target and move in steps. And respect the shot: automatic reframing during a live take is visible in a way an operator's hand isn't. Many teams end up using the loop to suggest to a human rather than to drive the lens directly, and that is usually the better product.

Constraints to design around

  • No audio in the protocol — anything acoustic needs its own path.
  • Local network. Discovery is Bonjour; put the model on the same network or bridge deliberately.
  • The stream is a proxy of the viewfinder, not the camera's full sensor output. It is plenty for detection and tracking; it is not a substitute for the local recording if you need maximum detail.
  • The password is access control, not encryption. Treat the network as production infrastructure.

None of that blocks the interesting work. The protocol is small enough that the receiver is the easy part of this project — the model is where your time should go.

Related guides

All articles