I am having troubles with my internet. It seems like my connection would drop every so often (When I play on my Xbox 360 online in Madden NFL 12, my connection keeps getting lost. It would say "Lost connection to host" after 3 minutes or so).

Also, every so often, my internet wireless bar thing on the bottom right of my screen (The bars) would have a yellow exclamation mark on them, indicating no internet access. Currently, I am using a Linksys G-router model number : WRT54GR. And I recently got a new modem from AT&T, so I am good on that. Also, my computer doesn't have the 'limited access' label when I plug my ethernet cable directly into my computer. It works perfectly fine, so I know it's not my modem. I know it has something to do with my router, but I am not sure what.

This is very frustrating, any help is greatly appreciated. Thank-you in advance!

Try giving your computers a fixed IP. You can do this by logging into the router and assigning an IP to to each device. If you are not sure how to do this, Google will help you.