It might be simple to take the amount that each note is ahead or behind where it would be quantized to and model that parameter exactly the same way that dynamics are modeled.