What is the brown dusty stuff on the bottom of my aquarium?

I have a 10 gallon aquarium that has been in use and cycled constantly for 2 years. It has a hang on back filter that is twice as powerful as is necessary for my tank and it is working properly (100GPH). There is still a lot of brown dusty stuff sitting on top of my gravel (I expect it is fish excretions). However the levels in my water are fine (ammonia, nitrite, nitrate), and fish have been living in it just fine for the last 2 years. The question is, should the filter be picking this up? why isn't it? and is it a problem? and how should I deal with it? By the way my last fish in the tank just died so I have an oppurtunity to really mess around before I get new fish (I just did a 50% water change).