A deep dive into RTP

For some years I’ve monitored RTP traffic at points in my network using span ports and tcpdump -T rtp. The latter shows the RTP sequence number, and piping into some perl allows me to look for missing or out of order packets as they traverse the network

I was quite keen on pulling the mpeg transport stream out of these captures though, to replay in VLC at leisure, or convert to thumbnails for monitoring purposes, etc, so I wrote a quick python tool to strip the RTP header from a pcap file and output to one (or many) RTP streams (as well as keep track of the RTP sequence numbers)

I then thought “it would be nice to extract the service name from the stream”, which meant a fascinating dive deep into RTP packet structures.

MPEG transport streams are a fascinating container with all sorts of goodies, so bashed some python together to investigate.

There are of course GUI programs like TSDuck to investigate streams, but they are GUIs, and thus far harder to use for normal people than a bit of linux command line, as they are difficult to integrate into a monitoring page.

By running tcpdump -i ens1 udp -w –, the entire of a udp sniff of a network tap can be output into python, and decoded, with all sorts of benefits.

Broadly speaking the RTP streams I work with contain multiple DVB MPEG services, one for video, one for audio, perhaps things like subtitles etc.

The RTP packet itself has a sequence number from 0 to 65535 and back to zero, but the (typically 7) 188 byte mpegts packets inside that have a 4 bit continuity counter, from 0 to 15 and back to zero. This is unique per PID, so we can track it, although if there is a continuity error there’s no confident way of knowing if you lost 1, 17 or 33 mpeg packets (you can probably infer if the rtp packets are also missing)

Occasionally an mpeg packet will go through which has things like the service and provider description, allowing easy identifying of what’s running through the network, so extracting that is worthwhile. Other bits of information can be helpful to diagnose decoder probelms too — RTP timestamps, varying PCR (program clock reference), interpacket arrival time, etc.

Sometimes you just want to dump the output.

The following is from sudo tcpdump -i ens1d1 -nn port 8100 -w – | ./dumpTS.py -s 1 sat across a passive fibre tap of traffic coming in from our current network provider, before it hits our network equipment.

It shows two RTP streams, one from 192.168.203.5 and one from 172.30.4.129, to 192.168.247.227 and .228. The contents is from an Appear X10, a fairly nice high density encoder.

The 192.168.203.5:8100>192.168.247.227:8100 stream is arriving with very little jitter (a maximum of 600μs, and an average of 340μs). There are 5 PIDs, PID101 is the highest bitrate. Pids 0, 17 and 100 are ‘helper’ pids in this DVB stream – 17 contains the SDT (Service Description Table) to tell you what the service name is, 100 is the PMT (program map table) to tell you what streams make the program up etc.

Generally the errors I tend to see are with the RTP stream – a packet goes missing (often because of Mikrotiks or Juniper SRXs), and things break. However occasionally the error is between the source of the mpeg transport stream and the RTP wrapper. I’ve seen this with AppearTV for example, which internally generates the mpeg stream in the encoding process, then ships it off to an IP interface. RTP analysis shows that the IP traffic passed without problem, however “Garbage in, Garbage Out” applies, and a CC error in the Video pid, jumping from 1 to 3, was detected.

One missing mpeg packet is unlikely to be noticable in the real world, but it’s a facinating level of visibility I haven’t seen before.

I’m not a software developer, I bang code together until it works well enough to solve my problem, then I move on. If you are interested in using the code though it’s at https://github.com/isostatic/rtp_reader

This level of monitoring is great, but it’s written in python, and when trying to process 600mbit of streams the CPU starts crying. What I really want is something to monitor all the traffic for potential RTP streams, and not eat my CPU while doing it, and that’s where I have to dust off my rusty C memories.