A full 32bit 4k screen is 31mb
There are approximately 70 full 4k screens in 2GB, which is about the memory of an ancient PC or graphics card.
My PC performs unbuffered memory writes at 10GB/s according to a benchmark I just ran, and read performance is over 13GB/s
That means my PC can write the full 2GB in 1/5 of a second and my PC is currently under a lot of load, I have loads of stuff open.
More details including GPU to CPU and CPU to GPU memory performance can be found here:
http://www.bealto.com/gpu-benchmarks_mem.html
The devices listed are relatively old, a PCI Express V2 16 BUS is listed with a performance of 8GB/s
It would take 1/25 of a second to transfer the full 2GB at theoretical maximum performance.
That means you could sustain a transfer rate of 201MB at a speed of 40 frames per second, or about 6 and a half full 4k screens or 25 full-HD screens 40 times a second.
A modern PCI Express bus can sustain 126 GB/s
Having a vector with 32bit x and y coordinates for every pixel of a 4k screen shouldn't pose a performance issue. Old PCI Express 2 systems can achieve that at 120 fps.
Masks and stencils won't be the whole screen and even if they were older systems without 4k screens (you need a modern graphics card to be able to operate one) could have roughly 350 full-HD screen stencils before filling up 2GB of RAM.
So, the memory requirement of using masks and stencils and then texturing and filtering on the GPU shouldn't be an issue.
Well, that's all the theory out of the way, and it certainly seems viable. Just to see if any of it works in practice.