Lately I've been wondering a lot about this whole Agile debate, where it's applicable and where not. I'm trying to absorb as much as I possibly can about the art and science of software development, from its roots in human thinking and philosophy and management science, through to its underlying technical componentry, in the hope that it'll make me a better craftsman. When looking at this debate over whether or not Agile "works", I can't help but feel a little overwhelmed. Do I invest my time and effort into learning processes and building habits that will ultimately not profit me? The fact that this debate still rages on in the first place says to me that we've possibly gotten a whiff of value coming from the Agile direction, but can't quite place our finger on what that tasty new smell is.
When confronted with large quantities of information, I try to go back to what I think are the "basics" (in my case, deeper levels of abstraction and a focus on values and value). Broadly speaking, Agile is a class of processes, and processes by their very nature are structures (in our thinking, habits, communication patterns, etc.) that are meant to facilitate the extraction of value in a somewhat repeatable way. That is, we want to to be able to take those processes that facilitate value extraction (e.g. working software with happy clients and happy engineers) in one environment, and apply them to another environment with similar results.
Management science in general has this somewhat dubious appearance to me in that we try to apply the kind of rigorous scientific thinking that we apply to the domains of physics, chemistry and biology, which all have had pretty regular processes and structures in place for many billions of years, to the social domain, which is barely in its infancy right now when looking at humanity in context. And for what? To control it? To guarantee some level of predictability and repeatability?
Of course, we can't throw all critical thinking on the management science front out - I think some of it definitely does benefit us, when relevant. But we need to keep our context in mind, and a sense of humility and openness to experience and correction. We're all effectively still experimenting here, trying to grope around in the dark in search of these potentially repeatable processes that continue to elude us.
I think the best summary I've seen so far of where Agile succeeds and where it fails is probably Michael O. Church's article on why Agile and Scrum are terrible. One of the deep reasons why it fails so horribly is that people try to apply Agile processes to social environments that are "Waterfall" by their very nature (i.e. the "fun" or "interesting" work gets progressively picked off as the work trickles down the bureaucratic hierarchy, leaving engineers with nothing but lifeless implementation work and little to no meaningful connection with the end customer/client). Church calls this "business-driven" business, as opposed to "engineer-driven" business. If you apply Agile processes to a "business-driven" business, all you get is sloppy, undisciplined Waterfall.
In short, it looks as though the simple rule is, if your company's existing approach to making things for customers already aligns really well with the 12 Agile Principles, then Agile processes are a good fit for your environment. Otherwise, you're going to need to keep searching. Or change your company (in either or both senses of the expression).