Sample selection within the 'Fit' function

I've been using the fit function to play stacks of drum patterns recently. It's great, but now i'm looking for a way to change the samples within the fit function, using an integer - which I intend to assign to a midi controller.

Usually 'n' would be perfect for this, but it doesn't work within the fit function. Below is the code of what i'm effectively trying to do. I should point out that I have tried doing this using 'fix', which does work, but won't work for me as it uses a lot of CPU for some reason and I'm trying to do this for quite a lot of drums.

d1 $ sound (fit 3 [("drum" # n "5"), ("hh" # n "3"), ("sn" # n "2")] "1 0 [1 1]  2 2 0 [2 2]")

Can anyone think of a way I might be able to make this work? Thank you very much. You guys have helped me out so much recently.