Modern video encoding is based on interpolating from the last frame, that is, only storing a real frame occasionally, and most of the time just having diffs from the last frame. ^+ This means that the data is inherently a stream -- a decoder needs to have completed decoding the previous frame before it can start on the next.

(^+ and B-frames, but let's not overcomplicate things)