It's strange that in all the literature on the subject of urbanism and city living, very few people ever ask themselves why humans would build cities at all. It seems to me that in order to truly know what to do with the city, we must start out by knowing why it's there in the first place. Somehow cities are so deeply rooted in humanity's history that we never get around to asking why we live in them.
I suppose the best way to start out is to look at what life would be like if there were no cities. Let's suppose that we all lived on our own homesteads, much as some of our ancestors did (truer in America than elsewhere). We would get up in the morning, work our homestead, eat our meals at home, provide for our entertainment at home, then go to bed at night, without ever leaving the homestead. But could we organize into advanced industries? We could build "factory homesteads" where we would get up in the morning in the factory, work in the factory during the day, and sleep in the factory at night, with all the thousands of other workers. (This is what the first socialists-planners, like Charles Fourier, proposed to do with their Phalanstère. It failed miserably.) But that still would be a very bare lifestyle, and it's not very nice to live in a factory. So instead we would probably want to live in a house, and then walk to the factory and back. So would our fellow factory hands want each their own house.
But what if we were bored of entertaining at home? After the factory, we might want to get dinner out from a place that cooks better than us, and go to a show with people who are better entertainers. That's two different locations that we need to add to our web of destinations. These locations matter to us because we want to form a relationship to them. Since they can do something different than us, they are complementary to us.
By now our model landscape of homesteads has been split up into a sprawling landscape of different zones, somewhat like Frank Lloyd Wright had proposed Broadacre City should be. But since everything is so far away from everything else, it is costly to form many relationships. The closer together we build the different (differentiated) spaces, the more relationships can form between them at the same cost. And so we cluster everything together as tight as feels comfortable, and we now have a wide selection of different factories, eateries and entertainment to choose from. Cities ultimately make possible diversity. Why are the densest cities in the world, like Paris' left bank or New York' Manhattan, also the most expensive? Why would the richest people choose to live there in close proximity to others, when they can afford to live anywhere? Because that is where there is the greatest diversity of relationships they can form.
So having decided that it's a good idea to be close together, are there any alternative means of achieving that? We could commission a great architect to build us a gigantic arcology that would condense us all into one big huge building. But what would we tell him we need to build? And where would we live until it's done? This project would run into the inevitable problem of programmatic chaos. An architect can only begin his work when we know what we need him to build. We may need a house with 3 bedrooms and a large kitchen, and therefore we tell this to an architect and he builds this house for us. But can this apply at the scale of an entire city? In reality, we are not quite certain what it is we need at this moment, and we certainly don't know what our needs will be a decade from now. A million-people arcology is going to take that long to build. And where would live until that thing is ready to inhabit?
Cities solve these problems by splitting up this huge system into individual cells that can be added or removed at pretty much every scale. If population goes up, we keep up by building more homes. If it goes down, we shrink. If we decide that in fact bowling alleys are not cool anymore, we remove the bowling alleys and build schools in their place. But it is always in the present moment that we know what is sorely needed, and we can only know this by evaluating what the city consists of in its current state. The program, the features that a building is designed to support, can only be determined for very small scale projects. A house, a school, a shop can be described in great details based on our previous experiences with these and their interaction in the complex mesh of the city. But the bigger the building is, say a big skyscraper, the more difficult it is to define it clearly, so what we see is that skyscrapers usually are made to be "bare bones" generic space that will be later filled up with concrete activities once the skyscraper is complete, several years after being designed.
The programmatic chaos that we must live with in a complex society means that we can't build huge structures in one step, we must build them in feedback loops by building chains of small structures that connect together, each new structure correcting the imbalance that was perceived in the last loop. So if I move to a new city and I can't find a home to my liking, I can buy an empty lot and build one, or build one on an abandoned or crumbling building, correcting two imbalances at the same time. And while I get a new job or start a business the city's industry will change as well, and may need different buildings and different places to put them. But this is work that is entirely different from the work of architecture.
This means that the 20th century model of urban planning introduced by Le Corbusier and the CIAM has not only been a huge mistake, but it has attempted to destroy the very thing that makes cities useful in the first place, the fact that they can deal with the uncertainty of our future by transforming themselves while remaining functional through this transformation. This is what makes them complex systems like the organic structures of the natural world. They are multicellular, they are capable of growth and adaptation by changing these cells, and since they transform themselves to meet uncertainty, they are unpredictable, emergent structures.
When we look at science fiction, we often see pictures of huge city-sized space stations, shining smoothly in perfect circles or some such. But these gigantic space stations would inevitably also run into programmatic chaos, so what we would see in reality is probably stations with a lot of "suburban growth" attached to them as space pioneers built extensions to make up for the lack of omniscience of the station's designers. The best thing to do, of course, would be to build space cities, space stations that could work at any scale and grow and shrink, and the construction of these space cities would have to employ many of the techniques used by city builders here on Planet Earth.
So what we learn from all these philosophical exercises is that urbanism is quite an important discipline, one that happens to be completely different from architecture, and that it represents in itself a very important form of technology for humankind.
Comments
There is in fact considerable literature as to why cities emerge. It's generally, as you point out, accepted that there is a benefit in physical proximity for certain activities (ranging from physical protection, to marketing, etc.).
Re. the granularity of buildings and organic vs. planned city issue, I would submit that there is a need for both and that both can complement each other. Generally, granular/organic works better and is more flexible but there are important programmatic needs that cannot be met purely through organic/small scale, gradual growth.