This lack of uniformity in playback controls is my biggest complaint about Android TV in general. While the OS allows for the inclusion of dedicated playback remote control buttons, it doesn't require or expect it. If you look at Google's own Android TV developer devices, they only have the D-pad with center button. And Google doesn't require all apps to adhere to some degree of uniformity in terms of how they respect to D-pad clicks. They should force all apps to pause when the center button is clicked, and also bring up the on-screen UI at the same time, then resume play when it's clicked again. A single left-click or right-click on the D-pad during playback should skip backward or forward several seconds while keeping left or right pressed down during playback should rewind or FF. When video is paused, the pressing up or down on the D-pad should allow you to navigate to other on-screen controls (e.g. closed captions, other videos/episodes, playback speed, etc.).
Unfortunately, not all Android TV apps work that way. You just have to learn a slightly different set of behaviors for each one.