azure - 聚合多批记录

Question

我想将流分析记录分批聚合 n 条记录，每条记录满足以下条件:

• 每条记录只能属于一个批处理。
• 每个批处理最多包含 n 条记录。
• 当批处理的第 n 条记录到达(或发生超时)时，窗口将关闭并评估查询。

我还没有找到使用窗口函数来实现此目的的方法(因为它们是基于时间的，而不是基于计数的)。此外，像 CollectTOP 这样的函数也不起作用，因为它是针对每个记录而不是每个批处理进行评估的。

Answer 1

正如评论中提到的，Azure 流分析没有事件计数的概念。

为了能够对每批特定大小的事件进行重新分组，第一步是对它们进行排名 ( CollectTop )。与任何流处理器一样，ASA 需要 time window 来定义该等级。

这不符合要求，也在评论中进行了澄清。抱歉，ASA 似乎不是这里的解决方案。

如果您可以容忍时间窗口 - 意味着容忍延迟(记录在窗口末尾输出)并容忍不完整的批处理(大多数时间窗口不会有 n 个事件的倍数) - 那么这可以通过多步查询部分实现。

在输入中使用此 input file 以及此查询:

WITH Ranking AS (
    SELECT
        System.Timestamp() as ts,
        COLLECTTOP(1000000) OVER (ORDER BY EntryTime, TollId) as c
    FROM [entry] AS e TIMESTAMP BY EntryTime
    GROUP BY TumblingWindow(minute,5)
),
Unfolding AS (
    SELECT
        System.Timestamp() AS windowEnd,
        FLOOR((d.ArrayValue.rank-1) /3.0) as batch3,
        d.ArrayValue.rank,
        d.ArrayValue.value.*
    FROM Ranking AS r
    CROSS APPLY GetElements(r.c) AS d
),
Batching AS (
    SELECT
        System.Timestamp() as ts,
        Collect() as batch
    FROM Unfolding
    GROUP BY System.Timestamp, batch3

)
SELECT * FROM Batching

我们将得到:

{"ts":"2014-09-10T12:05:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:01:00.0000000Z","TollId":1,"LicensePlate":"JNB 7001","State":"NY","Make":"Honda","Model":"CRV","VehicleType":1,"VehicleWeight":0,"Toll":7,"Tag":null},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:02:00.0000000Z","TollId":1,"LicensePlate":"YXZ 1001","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":123456789},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:02:00.0000000Z","TollId":3,"LicensePlate":"ABC 1004","State":"CT","Make":"Ford","Model":"Taurus","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":456789123}]}
{"ts":"2014-09-10T12:05:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:03:00.0000000Z","TollId":1,"LicensePlate":"BNJ 1007","State":"NY","Make":"Honda","Model":"CRV","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":789123456},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:03:00.0000000Z","TollId":2,"LicensePlate":"XYZ 1003","State":"CT","Make":"Toyota","Model":"Corolla","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null},{"windowEnd":"2014-09-10T12:05:00.0000000Z","batch3":1.0,"rank":6,"EntryTime":"2014-09-10T12:05:00.0000000Z","TollId":2,"LicensePlate":"CDE 1007","State":"NJ","Make":"Toyota","Model":"4x4","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":321987654}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:06:00.0000000Z","TollId":2,"LicensePlate":"BAC 1005","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":567891234},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:07:00.0000000Z","TollId":1,"LicensePlate":"ZYX 1002","State":"NY","Make":"Honda","Model":"Accord","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":234567891},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:07:00.0000000Z","TollId":2,"LicensePlate":"ZXY 1001","State":"PA","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":987654321}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:08:00.0000000Z","TollId":3,"LicensePlate":"CBA 1008","State":"PA","Make":"Ford","Model":"Mustang","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":891234567},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":2,"LicensePlate":"CDB 1003","State":"PA","Make":"Volvo","Model":"C30","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":765432198},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":1.0,"rank":5,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":2,"LicensePlate":"DCB 1004","State":"NY","Make":"Volvo","Model":"S80","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":654321987}]}
{"ts":"2014-09-10T12:10:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":7,"EntryTime":"2014-09-10T12:09:00.0000000Z","TollId":3,"LicensePlate":"YZX 1009","State":"NY","Make":"Volvo","Model":"V70","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":912345678},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":8,"EntryTime":"2014-09-10T12:10:00.0000000Z","TollId":1,"LicensePlate":"CBD 1005","State":"NY","Make":"Toyota","Model":"Camry","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":543219876},{"windowEnd":"2014-09-10T12:10:00.0000000Z","batch3":2.0,"rank":9,"EntryTime":"2014-09-10T12:10:00.0000000Z","TollId":3,"LicensePlate":"BCD 1002","State":"NY","Make":"Toyota","Model":"Rav4","VehicleType":1,"VehicleWeight":0,"Toll":5.5,"Tag":876543219}]}
{"ts":"2014-09-10T12:15:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:11:00.0000000Z","TollId":1,"LicensePlate":"NJB 1006","State":"CT","Make":"Ford","Model":"Focus","VehicleType":1,"VehicleWeight":0,"Toll":4.5,"Tag":678912345},{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:12:00.0000000Z","TollId":3,"LicensePlate":"PAC 1209","State":"NJ","Make":"Chevy","Model":"Malibu","VehicleType":1,"VehicleWeight":0,"Toll":6,"Tag":219876543},{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:15:00.0000000Z","TollId":2,"LicensePlate":"BAC 1005","State":"PA","Make":"Peterbilt","Model":"389","VehicleType":2,"VehicleWeight":2.675,"Toll":15.5,"Tag":567891234}]}
{"ts":"2014-09-10T12:15:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:15:00.0000000Z","batch3":1.0,"rank":4,"EntryTime":"2014-09-10T12:15:00.0000000Z","TollId":3,"LicensePlate":"EDC 3109","State":"NJ","Make":"Ford","Model":"Focus","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":198765432}]}
{"ts":"2014-09-10T12:20:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:18:00.0000000Z","TollId":2,"LicensePlate":"DEC 1008","State":"NY","Make":"Toyota","Model":"Corolla","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null},{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":2,"EntryTime":"2014-09-10T12:20:00.0000000Z","TollId":1,"LicensePlate":"DBC 1006","State":"NY","Make":"Honda","Model":"Civic","VehicleType":1,"VehicleWeight":0,"Toll":5,"Tag":432198765},{"windowEnd":"2014-09-10T12:20:00.0000000Z","batch3":0.0,"rank":3,"EntryTime":"2014-09-10T12:20:00.0000000Z","TollId":2,"LicensePlate":"APC 2019","State":"NJ","Make":"Honda","Model":"Civic","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":345678912}]}
{"ts":"2014-09-10T12:25:00.0000000Z","batch":[{"windowEnd":"2014-09-10T12:25:00.0000000Z","batch3":0.0,"rank":1,"EntryTime":"2014-09-10T12:22:00.0000000Z","TollId":1,"LicensePlate":"EDC 1019","State":"NJ","Make":"Honda","Model":"Accord","VehicleType":1,"VehicleWeight":0,"Toll":4,"Tag":null}]}

这保证了每个批处理不会有超过 N=3 的事件。但我们在 5 分钟窗口内强制执行这一保证。当一个时间窗口内的记录较少时，我们将得到一个少于N条记录的批处理。