There is no designated numbered altitude in the regulation for when you're flying "anywhere." Rather, this altitude is dependent on the area that is being overflown. FAR 91.119(a) states that "no person may operate an aircraft below...an altitude allowing, if a power unit fails, an emergency landing without undue hazard to persons or property on the surface." In other words, the pilot must fly high enough to be able to make an emergency, engine-out landing without placing persons and property on the ground in "undue" jeopardy of harm. This is an overall minimum that supplements the more specific minimums for flying over congested and noncongested areas.
The regulation stipulates that "over any congested area of a city, town, or settlement, or over any open air assembly of persons, an altitude of 1,000 feet above the highest obstacle within a horizontal radius of 2,000 feet of the aircraft" must be maintained. The FAA does not define "congested area" in the FAR or in the Aeronautical Information Manual (AIM). And, FAA interpretations and decisions issued by the National Transportation Safety Board in enforcement cases are not consistent enough to allow us to draft a precise definition. Such a determination is usually made on a case-by-case basis and congested has been interpreted rather broadly. For example, a highway with moderate traffic was found to be "congested," as was a seaside area where 200 to 300 persons were sitting on the beach or bathing in the water.
In areas that are not considered congested but may still contain some population, you must operate at an altitude of 500 feet above the surface. If you are over open water or sparsely populated areas, you may operate your aircraft at any altitude, but you must not fly closer than 500 feet to any person, vessel, vehicle, or structure (including electrical or telephone wires). This 500-foot minimum distance may be measured horizontally, vertically, or at an angle.
Helicopters "may be operated at less that the minimums prescribed...if the operation is conducted without hazard to persons or property on the surface. In addition, each person operating a helicopter shall comply with any routes or altitudes specifically prescribed for helicopters by the [FAA]." This regulation was drafted to recognize the unique flight characteristics of helicopters and accommodate their special flight utility while maintaining a standard for safety.
The regulation allows you to operate your aircraft at altitudes lower than those prescribed when it is necessary for takeoff or landing. The key to complying with this exception is determining what may be necessary, and the determination will depend on the circumstances of your flight. For example, it will be considered necessary for you to fly below the minimum prescribed altitudes while executing a normal approach to an approved runway. But, it will probably not be considered necessary for landing to fly below those altitudes on approach to an unsuitable landing area, such as a taxiway, a closed runway, or a field too small to safely accommodate your aircraft.
The general regulatory requirements for the minimum altitudes were adopted for safety reasons, not because of noise concerns. Note that the minimums are measured above the surface, not in mean sea level. In some circumstances the minimum altitude is determined by the height of a building or a tower or the location of a person or a vessel, rather than the height of the underlying terrain. There are also specific regulations that identify minimum altitudes for particular areas, such as the Grand Canyon.
Low flight can be fun, but any flight close to the ground must be conducted in accordance with the rules and in a safe, prudent manner.