RMem: Restricted Memory Banks Improve Video Object Segmentation

With recent video object segmentation (VOS) benchmarks evolving tochallenging scenarios, we revisit a simple but overlooked strategy: restrictingthe size of memory banks. This diverges from the prevalent practice ofexpanding memory banks to accommodate extensive historical information. Ourspecially designed "memory deciphering" study offers a pivotal insightunderpinning such a strategy: expanding memory banks, while seeminglybeneficial, actually increases the difficulty for VOS modules to decoderelevant features due to the confusion from redundant information. Byrestricting memory banks to a limited number of essential frames, we achieve anotable improvement in VOS accuracy. This process balances the importance andfreshness of frames to maintain an informative memory bank within a boundedcapacity. Additionally, restricted memory banks reduce the training-inferencediscrepancy in memory lengths compared with continuous expansion. This fostersnew opportunities in temporal reasoning and enables us to introduce thepreviously overlooked "temporal positional embedding." Finally, our insightsare embodied in "RMem" ("R" for restricted), a simple yet effective VOSmodification that excels at challenging VOS scenarios and establishes new stateof the art for object state changes (on the VOST dataset) and long videos (onthe Long Videos dataset). Our code and demo are available athttps://restricted-memory.github.io/.