I would say, yes, the city has an excellent historical defense.
Its primary defense was in conquering its neighbors. Having neighbors that can't war with you is the single best method of defending your own territory and cities. We see this strategy used over and over again. Not only does conquering your neighbors deny them the ability to attack, the conquered enemies add their economic and military power to your own, further deterring others who seek a fight. Just to start a siege, an invader would need to eliminate multiple Imperial armies and have successfully cleared a supply route to the capital.
The city itself is defended by the Emperor. While we see some of what the Emperor can do at the end of Oblivion, we surely don't see everything. The city also has a well established school of magic, giving the city quite a supply of able mages. Having a big bottle of magic whoop-ass that you can use against your enemies surely makes attacking the city that much harder.
The walls themselves look perfectly adequate to keep others out while keeping everyone inside safe. Successfully prosecuting a siege against the Imperial City would take considerable time and resources. That's not saying that the walls are impregnable, because given enough time and enemies, nobody is impregnable. However, the walls are more than sufficient to hold out while awaiting the arrival of intact Imperial legions.
If the Empire has reached the point where it has no armies, is under siege, and the opposing army is sufficiently well managed that it can supply itself and build a causeway over the walls, the city won't stand. As far as I'm concerned, that's a mighty high bar to reach, which makes that a mighty good defense.
On the other hand, Whiterun defenses look good, but they're nothing compared to the Imperial City. They don't have a magic bottle of whoop-ass as a last ditch defense, they don't have field armies capable of returning, they have few sympathetic allies, only one resident wizard, and their defenses are in poor repair. How much punishment can those walls take? Who knows? In the short run, their walls do their job, but against a long siege, especially by determined attackers, their walls would fall.