AFAIK, the interlace filter causes that. This also happens on the Saturn emulator SSF when automatic frameskipping starts to drop frames (since many Saturn games use high resolution interlaced video modes), the interlacing starts to produce visual artifacts such as the one you're seeing. For MAME, you need the romset as well as the CHD file for the game. As you start it the first time, the CPS-3 hardware rewrites the NVRAM. This takes around 70 minutes on the actual hardware (talk about inefficient data transfer). Depending your computer speed, and whether you use fast forward it naturally takes less time.