This one tracks the original positions on top of just the items so that it can remove items. It now takes twice as much memory but removes half of a million items in 50ms.