问题:
就我而言，比特流来自 h264 编码器，由实时摄像机提供。 Bitstream由live555库使用RTSP协议(protocol)发送。 RTSP数据包由rtsp客户端之一接收，例如vlc播放器(最重要的)，totem播放器，mplayer。 我需要知道如何设置较低的帧速率，因为我的相机产生 9 fps，但大多数 RTSP 客户端期望默认为 25 fps，这会导致视频流无法正常播放(缓冲、丢弃延迟的数据包等)。
问题:
我想知道RTSP客户端(播放软件)如何从比特流中读取(或评估)帧速率。这些信息存储在哪里以及如何发送到播放软件？
更多信息:
我正在使用 live555 中的 testOnDemandRTSPServer 示例代码。我已经从 live555 lib 编写了 DeviceSource 类的实现。我已正确设置演示时间:
gettimeofday(&fPresentationTime, NULL);
fDurationInMicroseconds = 1100000;
序列参数集 NAL 还包含正确的时序信息:
timing_info_present_flag : 1
num_units_in_tick : 1001
time_scale : 9000
fixed_frame_rate_flag : 1
RTSP header 包含帧速率:
a=framerate:9.0
不幸的是，这没有帮助。我认为视频播放器会平静地等待新帧，并且帧速率将与接收另一个 RTP 数据包(另一个 NAL)的频率相关。
视频播放器的行为方式:
VLC 播放器 - 视频断断续续，因为 vlc 预计下一帧会更早(25 fps 而不是 9 fps)。结果，vlc 将缓冲时间增加到例如10s，在此期间视频被卡住。
图腾播放器 - 来自 SPS Nal 的计时信息由图腾解析。帧速率显示正确，但视频仍然不稳定(延迟约 500 毫秒)。似乎正在等待更多帧(每秒 25 帧)。
Mplayer 识别出错误的帧速率值，显示 25 fps，但视频非常流畅且质量良好:
mplayer ffmpeg://rtsp://192.168.1.82:8554/testStream
.
.
.
VIDEO: [H264] 320x240 0bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
Live555 RTSP客户端测试应用 ./live555/testProgs/testRTSPClient rtsp://192.168.1.82:8554/testStream
Opening connection to 192.168.1.82, port 8554...
...remote connection opened
Sending request: DESCRIBE rtsp://192.168.1.82:8554/testStream RTSP/1.0
CSeq: 2
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Accept: application/sdp
Received 675 new bytes of response data.
Received a complete DESCRIBE response:
RTSP/1.0 200 OK
CSeq: 2
Date: Fri, Jan 01 1988 00:00:23 GMT
Content-Base: rtsp://192.168.1.82:8554/testStream/
Content-Type: application/sdp
Content-Length: 506
v=0
o=- 567993603275971 1 IN IP4 192.168.1.82
s=Session streamed by "testOnDemandRTSPServer"
i=testStream
t=0 0
a=tool:LIVE555 Streaming Media v2016.02.22
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "testOnDemandRTSPServer"
a=x-qt-text-inf:testStream
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801E;sprop-parameter-sets=Z0KAHpWgUHxA,aM44gA==
a=framerate:9.0
a=control:track1
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Got a SDP description:
v=0
o=- 567993603275971 1 IN IP4 192.168.1.82
s=Session streamed by "testOnDemandRTSPServer"
i=testStream
t=0 0
a=tool:LIVE555 Streaming Media v2016.02.22
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "testOnDemandRTSPServer"
a=x-qt-text-inf:testStream
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801E;sprop-parameter-sets=Z0KAHpWgUHxA,aM44gA==
a=framerate:9.0
a=control:track1
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Initiated the "video/H264" subsession (client ports 57660-57661)
Sending request: SETUP rtsp://192.168.1.82:8554/testStream/track1 RTSP/1.0
CSeq: 3
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Transport: RTP/AVP;unicast;client_port=57660-57661
Received 215 new bytes of response data.
Received a complete SETUP response:
RTSP/1.0 200 OK
CSeq: 3
Date: Fri, Jan 01 1988 00:00:23 GMT
Transport: RTP/AVP;unicast;destination=192.168.1.109;source=192.168.1.82;client_port=57660-57661;server_port=6970-6971
Session: 06B1DFA4;timeout=65
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Set up the "video/H264" subsession (client ports 57660-57661)
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Created a data sink for the "video/H264" subsession
Sending request: PLAY rtsp://192.168.1.82:8554/testStream/ RTSP/1.0
CSeq: 4
User-Agent: ./testRTSPClient (LIVE555 Streaming Media v2016.02.22)
Session: 06B1DFA4
Range: npt=0.000-
Received 96 new bytes of response data.
Received 94 new bytes of response data.
Received a complete PLAY response:
RTSP/1.0 200 OK
CSeq: 4
Date: Fri, Jan 01 1988 00:00:23 GMT
Range: npt=0.000-
Session: 06B1DFA4
RTP-Info: url=rtsp://192.168.1.82:8554/testStream/track1;seq=52833;rtptime=3161001994
[URL:"rtsp://192.168.1.82:8554/testStream/"]: Started playing session...
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 10874 bytes. Presentation time: 567993623.442288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12054 bytes. Presentation time: 567993623.482288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12227 bytes. Presentation time: 567993623.522288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12281 bytes. Presentation time: 567993623.562288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12384 bytes. Presentation time: 567993623.602288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12129 bytes. Presentation time: 567993623.642288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12141 bytes. Presentation time: 567993623.682288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12287 bytes. Presentation time: 567993623.722288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 16415 bytes. Presentation time: 567993623.762288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12360 bytes. Presentation time: 567993623.802288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12167 bytes. Presentation time: 567993623.842288
我通过工具 h264_analyze 解析了我的 NAL，下面是输出(只有几个前 NAL):
!! Found NAL at offset 4 (0x0004), size 18 (0x0012)
XX 00 00 00 01 67 42 80 1E 95 A0 50 7C 84 00 00 0F
==================== NAL ====================
forbidden_zero_bit : 0
nal_ref_idc : 3
nal_unit_type : 7 ( Sequence parameter set )
======= SPS =======
profile_idc : 66
constraint_set0_flag : 1
constraint_set1_flag : 0
constraint_set2_flag : 0
constraint_set3_flag : 0
constraint_set4_flag : 0
constraint_set5_flag : 0
reserved_zero_2bits : 0
level_idc : 30
seq_parameter_set_id : 0
chroma_format_idc : 1
residual_colour_transform_flag : 0
bit_depth_luma_minus8 : 0
bit_depth_chroma_minus8 : 0
qpprime_y_zero_transform_bypass_flag : 0
seq_scaling_matrix_present_flag : 0
log2_max_frame_num_minus4 : 4
pic_order_cnt_type : 2
log2_max_pic_order_cnt_lsb_minus4 : 0
delta_pic_order_always_zero_flag : 0
offset_for_non_ref_pic : 0
offset_for_top_to_bottom_field : 0
num_ref_frames_in_pic_order_cnt_cycle : 0
num_ref_frames : 1
gaps_in_frame_num_value_allowed_flag : 0
pic_width_in_mbs_minus1 : 19
pic_height_in_map_units_minus1 : 14
frame_mbs_only_flag : 1
mb_adaptive_frame_field_flag : 0
direct_8x8_inference_flag : 0
frame_cropping_flag : 0
frame_crop_left_offset : 0
frame_crop_right_offset : 0
frame_crop_top_offset : 0
frame_crop_bottom_offset : 0
vui_parameters_present_flag : 1
=== VUI ===
aspect_ratio_info_present_flag : 0
aspect_ratio_idc : 0
sar_width : 0
sar_height : 0
overscan_info_present_flag : 0
overscan_appropriate_flag : 0
video_signal_type_present_flag : 0
video_format : 0
video_full_range_flag : 0
colour_description_present_flag : 0
colour_primaries : 0
transfer_characteristics : 0
matrix_coefficients : 0
chroma_loc_info_present_flag : 0
chroma_sample_loc_type_top_field : 0
chroma_sample_loc_type_bottom_field : 0
timing_info_present_flag : 1
num_units_in_tick : 1001
time_scale : 9000
fixed_frame_rate_flag : 1
nal_hrd_parameters_present_flag : 0
vcl_hrd_parameters_present_flag : 0
low_delay_hrd_flag : 0
pic_struct_present_flag : 0
bitstream_restriction_flag : 0
motion_vectors_over_pic_boundaries_flag : 0
max_bytes_per_pic_denom : 0
max_bits_per_mb_denom : 0
log2_max_mv_length_horizontal : 0
log2_max_mv_length_vertical : 0
num_reorder_frames : 0
max_dec_frame_buffering : 0
=== HRD ===
cpb_cnt_minus1 : 0
bit_rate_scale : 0
cpb_size_scale : 0
bit_rate_value_minus1[0] : 0
cpb_size_value_minus1[0] : 0
cbr_flag[0] : 0
initial_cpb_removal_delay_length_minus1 : 0
cpb_removal_delay_length_minus1 : 0
dpb_output_delay_length_minus1 : 0
time_offset_length : 0
!! Found NAL at offset 26 (0x001A), size 4 (0x0004)
XX 00 00 00 01 68 CE 38 80
==================== NAL ====================
forbidden_zero_bit : 0
nal_ref_idc : 3
nal_unit_type : 8 ( Picture parameter set )
======= PPS =======
pic_parameter_set_id : 0
seq_parameter_set_id : 0
entropy_coding_mode_flag : 0
pic_order_present_flag : 0
num_slice_groups_minus1 : 0
slice_group_map_type : 0
num_ref_idx_l0_active_minus1 : 0
num_ref_idx_l1_active_minus1 : 0
weighted_pred_flag : 0
weighted_bipred_idc : 0
pic_init_qp_minus26 : 0
pic_init_qs_minus26 : 0
chroma_qp_index_offset : 0
deblocking_filter_control_present_flag : 0
constrained_intra_pred_flag : 0
redundant_pic_cnt_present_flag : 0
transform_8x8_mode_flag : 0
pic_scaling_matrix_present_flag : 0
second_chroma_qp_index_offset : 0
!! Found NAL at offset 33 (0x0021), size 339 (0x0153)
XX 80 00 00 01 65 B8 04 04 3F FF F8 7A 28 03 EF BE
==================== NAL ====================
forbidden_zero_bit : 0
nal_ref_idc : 3
nal_unit_type : 5 ( Coded slice of an IDR picture )
======= Slice Header =======
first_mb_in_slice : 0
slice_type : 2 ( I slice )
pic_parameter_set_id : 0
frame_num : 0
field_pic_flag : 0
bottom_field_flag : 0
idr_pic_id : 0
pic_order_cnt_lsb : 0
delta_pic_order_cnt_bottom : 0
redundant_pic_cnt : 0
direct_spatial_mv_pred_flag : 0
num_ref_idx_active_override_flag : 0
num_ref_idx_l0_active_minus1 : 0
num_ref_idx_l1_active_minus1 : 0
cabac_init_idc : 0
slice_qp_delta : -16
sp_for_switch_flag : 0
slice_qs_delta : 0
disable_deblocking_filter_idc : 0
slice_alpha_c0_offset_div2 : 0
slice_beta_offset_div2 : 0
slice_group_change_cycle : 0
=== Prediction Weight Table ===
luma_log2_weight_denom : 0
chroma_log2_weight_denom : 0
=== Ref Pic List Reordering ===
ref_pic_list_reordering_flag_l0 : 0
ref_pic_list_reordering_flag_l1 : 0
=== Decoded Ref Pic Marking ===
no_output_of_prior_pics_flag : 0
long_term_reference_flag : 0
adaptive_ref_pic_marking_mode_flag : 0
!! Found NAL at offset 376 (0x0178), size 1897 (0x0769)
XX 00 00 00 01 41 E0 20 3F 36 5F FF 87 A2 80 03 7E
==================== NAL ====================
forbidden_zero_bit : 0
nal_ref_idc : 2
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
======= Slice Header =======
first_mb_in_slice : 0
slice_type : 0 ( P slice )
pic_parameter_set_id : 0
frame_num : 1
field_pic_flag : 0
bottom_field_flag : 0
idr_pic_id : 0
pic_order_cnt_lsb : 0
delta_pic_order_cnt_bottom : 0
redundant_pic_cnt : 0
direct_spatial_mv_pred_flag : 0
num_ref_idx_active_override_flag : 0
num_ref_idx_l0_active_minus1 : 0
num_ref_idx_l1_active_minus1 : 0
cabac_init_idc : 0
slice_qp_delta : -15
sp_for_switch_flag : 0
slice_qs_delta : 0
disable_deblocking_filter_idc : 0
slice_alpha_c0_offset_div2 : 0
slice_beta_offset_div2 : 0
slice_group_change_cycle : 0
=== Prediction Weight Table ===
luma_log2_weight_denom : 0
chroma_log2_weight_denom : 0
=== Ref Pic List Reordering ===
ref_pic_list_reordering_flag_l0 : 0
ref_pic_list_reordering_flag_l1 : 0
=== Decoded Ref Pic Marking ===
no_output_of_prior_pics_flag : 0
long_term_reference_flag : 0
adaptive_ref_pic_marking_mode_flag : 0
!! Found NAL at offset 2277 (0x08E5), size 2989 (0x0BAD)
XX 00 00 00 01 41 E0 40 3F 37 0F 0C 3E D8 A0 00 4E
==================== NAL ====================
forbidden_zero_bit : 0
nal_ref_idc : 2
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
======= Slice Header =======
first_mb_in_slice : 0
slice_type : 0 ( P slice )
pic_parameter_set_id : 0
frame_num : 2
field_pic_flag : 0
bottom_field_flag : 0
idr_pic_id : 0
pic_order_cnt_lsb : 0
delta_pic_order_cnt_bottom : 0
redundant_pic_cnt : 0
direct_spatial_mv_pred_flag : 0
num_ref_idx_active_override_flag : 0
num_ref_idx_l0_active_minus1 : 0
num_ref_idx_l1_active_minus1 : 0
cabac_init_idc : 0
slice_qp_delta : -15
sp_for_switch_flag : 0
slice_qs_delta : 0
disable_deblocking_filter_idc : 0
slice_alpha_c0_offset_div2 : 0
slice_beta_offset_div2 : 0
slice_group_change_cycle : 0
=== Prediction Weight Table ===
luma_log2_weight_denom : 0
chroma_log2_weight_denom : 0
=== Ref Pic List Reordering ===
ref_pic_list_reordering_flag_l0 : 0
ref_pic_list_reordering_flag_l1 : 0
=== Decoded Ref Pic Marking ===
no_output_of_prior_pics_flag : 0
long_term_reference_flag : 0
adaptive_ref_pic_marking_mode_flag : 0
!! Found NAL at offset 5270 (0x1496), size 4074 (0x0FEA)
XX 00 00 00 01 41 E0 60 10 CD B7 C1 BF B8 36 00 6A
最佳答案
I would like to know how frame rate is read from bit stream (or evaluated) by RTSP client (playback software). Where this informations are stored and how are sending to playback software?
客户端不需要知道播放接收到的视频的帧速率。播放是根据根据 RTP 和 NTP 时间戳计算得出的演示时间进行的。
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 10874 bytes. Presentation time: 567993623.442288
Stream "rtsp://192.168.1.82:8554/testStream/"; video/H264: Received 12054 bytes. Presentation time: 567993623.482288
您似乎没有正确设置演示时间。这可能就是每秒 25 fps 的情况。如果设置正确，testRTSPClient 将每秒打印 9 个样本，它们之间的差异大约为 1/9 秒。您的差异约为 40 毫秒，即 25 fps。
a=framerate:9.0
我已经使用各种帧速率内容开发了 RTSP 服务器，并且我从未需要设置该 SDP 属性。我不确定哪些客户端软件实际使用它。
VLC player - video is choppy because vlc expects next frame earlier (25 fps instead of 9 fps). In results, vlc increases buffering time to e.g. 10s, during this time video is frozen.
根据我过去几年的经验，VLC 不擅长播放实时内容。我测试了各种 RTSP 服务器，VLC 经常遇到问题，即使使用更大的网络 jitter buffer 也是如此。
您可能还想了解的其他一些内容:
fDurationInMicroseconds = 1100000;
如评论中所述，对于实时源，fDurationInMicroseconds 应设置为 0。
b=AS:500
比特率似乎设置不正确:您的流绝对不是 500kbps。
关于video-streaming - live555如何计算或读取通过rtsp协议(protocol)发送的h264比特流的帧率，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/36332455/