Amid a glut of painting and portraiture, a few artists are experimenting with video, photography and animation