Ah. The FrontPanel GUI cannot communicate through Modelsim. You will need to use the API simulation commands in your Modelsim test bench.
While this is an interesting idea, it's mostly impractical due to the disconnect between simulation speeds and real-time interaction speeds of a GUI. The simulation would run so slow in relation to the GUI that I'm not sure what you'd be able to do with this.