Emergent Architectures

I was slightly alarmed to read recently in a document describing a particular adaptation of the unified process that allowing architectures to ‘emerge’ was a poor excuse to avoid hard thinking and planning and that emergent architectures, and anyone who advocates them, should be avoided.The term ’emergent architecture’ was, I believe, first coined by Gartner (see here) and applied to Enterprise Architecture. Gartner identified a number of characteristics that could be applied to emergent architectures one of which was that they are non-deterministic. Traditionally (enterprise) architects applied centralised decision-making to design outcomes. Using emergent architecture, they instead must decentralise decision-making to enable innovation.

Whilst emergent architectures certainly have their challenges it is my belief that, if well managed, they can only be a good thing and should certainly not be discouraged. Indeed I would say that emergence could be applied at a Solution Architecture level as well and is ideally suited to more agile approaches where everything is simply not known up front. The key thing with managing an emergent architecture is to capture architectural decisions as you go and ensure the architecture adapts as a result of real business needs.

Advertisements

Skills for Building a Smarter Planet

This is the transcript of a talk I gave to a group of sixth formers, who are considering a career in IT, at a UK university this week. The theme was “What do IT architects do all day” however I expanded it into “What will IT architects be doing in the future?”.What I want to do in the next 30 minutes or so is not only tell you what I, as an IT architect, do but what I think you will be doing should you choose to take up a career as an IT architect and what skills you wil need to do the job. In particular I’d like to explain what I mean by this:

Today’s world is full of wicked problems. Solving these problems, and building a smarter planet needs new skills. I believe that IT architects need to be a versatile and adaptive breed of systems thinkers.

Here’s the best explanation I’ve seen of what architects do:

Architects take existing components and assemble them in interesting and important ways. (Seth Godin)

As an example of this consider something that we use everyday, the (world-wide) web. Invented by Tim Berners-Lee just 20 short years ago, Tim basically assembled the web from three components that already existed: hypertext, internet protocols and what are referred to as markup languages. All these things existed, what Tim did was to assemble them in an “interesting” way. So what I do is to use IT to try and solve interesting and important business problems by assembling (software) components. I’m not just interested in any problems though, the type of problems that interest me are the “wicked” variety. What do I mean by these?

Wicked problems are ones that you often don’t really understand until you’ve formulated a solution to it. It’s often not even possible to really state what the problem is and because there is no clear statement of the problem, there can be no clear solution so you never actually know when you are finished. For wicked problems ‘finished’ usually means you have run out of time, money, patience or all three! Further, solutions to wicked problems are not “right” or “wrong”. Wicked problems tend to have solutions which are ‘better’, or maybe ‘worse’ or just ‘good enough’. Finally, every wicked problem is essentially novel and unique. Because there are usually so many factors involved in a wicked problem no two problems are ever the same and each solution needs a unique approach.

But there’s a problem! Here’s a headline from last year Independent newspaper: “Labour’s computer blunders cost £26bn”. What’s going on here? This is your and my money being wasted on failed IT projects. And it’s not just government projects that are failing. Here’s an estimate from the British Computer Society of how many IT projects re actually successful. 20%! How poor is that? It projects ‘fail’ for many reasons but interstingly it’s rarely for just technical reasons. More often than not it’s due to poor project and risk management, lack of effective stakeholder management or no clear senior management ownership. So we have a real problem here. As we’ll see in a minute,  problems are not only getting harder to fix (more ‘wicked’) but our ability to solve them does not seem to be improving!!

So what are these wicked problems I keep talking about? They are many and numerous but many of them are attributable to inefficiencies that exist in the “systems” that exist in the world. Economists estimate that globally we waste $15 trillion of the worlds precious resources each year. Much – if not most – of this inefficiency can be attributed to the fact that we have optimized the way the world works within silos, with little regard for how the processes and systems that drive our planet interrelate. These complex, systemic inefficiencies are interwoven in the interactions among our planet’s core systems. No business, government or institution can solve these issues in isolation. To root out inefficiencies and reclaim a substantial portion of that which is lost, businesses, industries, governments and cities will need to think in terms of systems, or more accurately, a system of systems approach. This means we will need to collaborate at unprecedented levels. For example no single organization owns the world’s food system, and no single entity can fix the world’s healthcare system. Success will depend upon understanding the full set of cause-and-effect relationships that link systems and using this knowledge to create greater synergy. Basically many of the problems the world faces today are cause by the fact that our systems don’t talk to each other. What do I mean by this? Here’s a simple example to illustrate the point.

Imagine you are driving your car around town trying to find a parking space. You can be sure that somewhere in town there’s a parking meter looking for a car to park in it. How do we marry your car with that parking meter? Actually the technology to do this pretty much exists already. However the challenge of actually fixing this problem stretches beyond just technology. A solution to this problem includes at least: intelligent sensors, communications, public and private finance, local government involvement, control and policing as well as well established open standards.

Like I said, from a pure technology point of view we are in pretty good shape to solve problems like this. We now have an unprecedented amount of: instrumentation, interconnectedness and intelligence
such that organisations (and societies) can think and act in radically new ways. However in order to solve problems like the parking one as well as significantly more ‘wicked’ ones I believe we need skills that stretch beyond the mere technological. If you are to help solve these problems then you need to be a versatile and adaptive systems thinker. A systems thinker is someone who not only uses her left-brained logical thinking capabilities but also uses her right-brained creative and artistic capabilities. Here are six attributes (from Dan Pink’s book A Whole New Mind) that a good systems thinker needs to adopt which I think will help in solving some of the worlds wicked problems:

  • Design – It is no longer sufficient or acceptable to create a product or service that merely does the job. Today it is both economically critical as well as  aesthetcially rewarding to create something that is beautiful and emotionally engaging.
  • Story – We are living in a time of information overload. If you want your sales pitch or point of view to be heard above the cacophony of background noise that is out there you have to create a compelling narrative.
  • Symphony – We live in a world of silos. Siloed processes, siloed systems and siloed societies. Success in business and in life is about breaking down these silos and pulling all the pieces together. Its about synthesis rather than analysis.
  • Empathy – Our capacity for logical thought has gone a very way to creating the technological society we live in today. However in a world of ubiquitous information that is available at the touch of a button logic alone will no longer cut the mustard.In order to thrive we need to understand what makes our fellow humans tick and really get beneath their skin and to forge new relationships.
  • Play – In a world where we are all having to meet targets, pass tests and  achieve the right grades in order to get on it is easy to forget the importance of play. There is a lot of evidence out there of the benefits to our health and general well-being of the benefits of play, not only outside work but also inside.
  • Meaning – We live in a world of material plenty put spiritual scarcity. Seeking meaning in life that transcends above “things” is vital if we are to achieve some kind of personal fulfilment.

A Gartner report published in 2005 predicted that by 2010, IT professionals will need to possess expertise in multiple domains. Technical aptitude alone will no longer be enough. IT professionals must prove they can understand business realities – industry, core processes, customer bases, regulatory environment, culture and constraints. Versatility will be crucial. It predicted that by By 2011, 70 percent of leading-edge companies will seek and develop “versatilists” while deemphasizing specialists.

Versatilists are people whose numerous roles, assignments and experiences enable them to synthesise knowledge and context in ways that fuel business value. Versatilists play different roles than specialists or generalists. Specialists generally have deep technical skills and narrow scope, giving them expertise that is recognized by peers, but it is seldom known outside their immediate domains. Generalists have broad scope and comparatively shallow skills, enabling them to respond or act reasonably quickly, but often at a cursory level. Versatilists, in contrast, apply depth of skill to a rich scope of situations and experiences, building new alliances, perspectives, competencies and roles. They gain the confidence of peers and partners. To attain versatilist skills, IT professionals should..

  • Look outside the confines of current roles, regions, employers or business units. The more informed a professional is about a company, its industry segment and the forces that affect it, the greater the contextual grasp.
  • Lay out opportunities and assignments methodically. Focus on the areas and challenges that fall outside the comfort zone; those areas generally will be the areas of greatest growth.
  • Explore possibilities outside the world of corporate business. Not-for-profit ventures, startup companies, government agencies and consumer IT service providers offer powerful ways to bolster experiences, behavioral competencies or management skills.
  • Enroll in advanced degree programs or in qualified education courses to expand perspective.
  • Identify companies, projects, assignments, education and training that will increase professional value.

I believe we’ve only just begun to scratch the surface of what’s possible on a “smarter planet”. However if we are to really address the truly wicked problems that are out there in order to make our world a better, and maybe even a fairer place, we need people like you to make it happen.

Finally you might be tempted in these hard economic times when you are being asked to pay outrageous amounts for your education not to bother with university. However bear this in mind:

“Unskilled labor is what you call someone who merely has skills that most everyone else has. If it’s not scarce, why pay extra? Skills matter. The unemployment rate for US workers without a college education is almost triple that for those with one. Even the college rate is still too high, though.  On the other hand, the unemployment rate for skilled neurosurgeons, talented database designers and motivated recombinant DNA biologists is essentially zero, despite the high pay in all three fields. Unskilled now means not-specially skilled”.

The only real investment you have for the future is the piece of grey matter between your ears. Make sure you continue to nurture and nourish it throughout your life by stimulating both the left and right sides.

Thank you and good luck with whatever path you choose to take in life.

Forget T-Shaped, We Need V-Shaped Architects

A recent blog from the Open Group discusses the benefits of so called “T-shaped people”. According to this blog, T-shaped people are what HR are looking for these days. To quote from the blog a T-shaped person is someone who: “combines the broad level of skills and knowledge (the top horizontal part of the T) with specialist skills in a specific functional area (the bottom, vertical part of the T). They are not generalists because they have a specific core area of expertise but are often also referred to as generalizing specialists as well as T-shaped people“. The picture below shows this.

Traditionally for software architects the specialism that T-shaped people usually have has come from their entry-level skills or the ones that got them into the profession in the first place. This is usually a skill in a particular programming language,  development approach (agile, scrum or whatever) or other areas related to software development such as test or configuration management. As you progress through your career and begin to build on your skills (learning more programming languages, understanding more about design etc) you may add to the verticals in your T’s with these other specialisms. This, at least, has traditionally been the approach. The problem is that in some organisations in order to “progress” (i.e. earn more money) you almost need to know more about less. You need to generalise more and more quickly. No one is going to employ you to be a Java programmer if your salary is ten times that of what a Java programmer in India or China earns. This is not meant to be a criticism against software professionals in India or China by the way. It’s just the way of things. Soon people in India and China will be out-sourcing to lower cost regions and so the cycle will go on. It does however raise an interesting problem of how those core specialisms will be developed in people just entering the profession. I spent a good 15 years as a programmer before I moved into architecture and would like to think that what I learnt there gave me a good set of core, fundamental skills that I can still apply as an architect. I firmly believe that the fundamentals I learnt from programming (encapsulation, design by contract, the importance of loose coupling etc) never go out of fashion.

As I have blogged before, I believe that whilst good “generalizing specialists” can also make good architects there is another dimension to what makes a true architect who has the skills necessary to solve the really hard business as well as socio-political (e.g. global warming, global terrorism, resource shortages etc) problems that the world faces today. Gartner coined the term “versatilist” back in 2005 and whilst this does not seem to have really taken off (there is a versatilist web site but it seems to be little used) I like the fact that the ‘V’ of versatilist makes a nice paradigm for what 21st century IT architects need to be. V-shaped people are not just ones who have deep skills in specific functional areas but also have skills in other disciplines. Further a good V-shaped person is one who has skills not just in technical disciplines but also business and artistic disciplines. So why does this matter?

The concept of bringing interdisciplinary teams together to break down boundaries in solving difficult or wicked problems is not a new one. It is recognised that pooling different academic schools of thought can often throw up solutions to problems that any of the individual disciplines could not. It follows therefore that if an individual can be well rounded and at least have some level of knowledge in an area completely outside his or her core discipliines then they to may be able to shed new light on difficult problems. This is what being a versatilist is about. As shown below its not just about specialising in different functional areas within a discipline but also across disciplnes. If these disciplines can be a mix of the arts as well as the sciences that exercise both right and left brains then so much the better.

So how should versatilists develop their skills? Here are some suggestions I give to IT students when discussing how they might survive as professionals in the 21st century world of work:

  • Objectively view experiences and roles – When you have finished an assignment note down what you learnt from it, what you could have done better and maybe ask others what they thought of your performance.
  • Look further than current roles. Today you are working on a particular project however always have in mind what you want to do next and an idea of what you want to do after that. Don’t become stereotyped, prepare to move on even if you are in an area you know well.
  • Plan opportunities and assignments – This follows on from the last one. make sure each assignment really builds on and develops your skills. Step out of your comfort zone in each new assignment.
  • Explore other possibilities. Never assume there is only one option. Think differently and look at alternatives. Like Paul Arden said, “Whatever You Think, Think The Opposite“.
  • Pursue lifelong learning – What it says. never stop exploring!
  • Identify companies that will increase professional value. Companies are out to get what they can from you. make sure you do the same with them.

So as we enter the second decade of the 21st century can we not look for more T-shaped people but start the search for V-shaped people instead? These are the ones who will really make a difference and be able to address the really wicked problems that are out there.

Interprise Architecture and Ultra-Large-Scale Systems

In a previous post I introduced the term “Interprise Architecture” to describe how the internet is breaking down the traditional boundaries of the enterprise and thus requires a new approach to Enterprise Architecture that’s not just about describing what’s inside the enterprise but also what’s on the outside. No longer can Enterprise Architects create blueprints for some future state that the enterprise will one day reach with roadmaps for how that state will be achieved. There are too many disruptive influences and new technologies that are impinging on the enterprise that will not only mean the roadmap is sending you in the wrong direction but that you are probably using the wrong mode of transport to get there as well.

I received a few comments on this from folk at the Software Engineering Institute (SEI)as well as Gartner. The work on Ultra-Large-Scale (ULS) Systems from the SEI particularly drew my attention and resonates nicely with some of my own thoughts. Here are some of the key ideas from the SEI report Ultra-Large-Scale Systems – The Software Challenge of the Future plus some additional musings of my own on what constitutes Interprise Architecture. First, ULS:

  • The SEI report on ULS systems was funded by the United States Department of Defence (DoD) which asked the SEI to consider future systems that could not only contain of billions of lines of code but also exhibit some, possibly all, of the following characteristics: decentralisation; conflicting, unknowable, and diverse requirements; continuous evolution and deployment; heterogeneous and changing elements; erosion of the people/system boundary; and normal failures of parts of the system.
  • ULS systems are likely to mean that traditional software and systems engineering approaches will no longer be adequate or can be the primary means by which such systems are designed (architected) or built.
  • ULS systems can be compared with cities whereas traditional systems can be compared with buildings. Buildings can be designed and built to a blueprint whereas cities emerge and are continuously adapting over time.
  • ULS systems are comprised of a dynamic community of interdependent and competing organisms (in this case, people, computing devices, and organizations) in a complex and changing environment. These are referred to as socio-technical ecosystems.
  • ULS systems are ones that are continuously evolving with new behaviour constantly emerging. In this respect they have the attributes of wicked problems where the problem is never definitively solved (or indeed understood).
  • ULS systems expect failure to be the norm and that unusual situations and boundary conditions will occur often enough that something will always be failing.

The SEI report is primarily aimed at allowing the US military to develop new systems however I believe the key ideas that challenge the development of such systems also have wide applicability in business systems, the sort I’m most interested in. I see that what I have characterised as Interprise Architecture could therefore be applied to developing ULS business systems. Here are three examples of ULS business systems that might benefit from an Interprise Architecture approach:

  • Electronic Trading Systems. These are systems that trade securities (such as stocks, and bonds), foreign currency, and exchange traded derivatives electronically. They use IT to bring together buyers and sellers through electronic media and create a virtual market place. Such systems are typically built using proprietary software that has grown and evolved over many years. Investment banks have extremely complex technology requirements, as they have to interface with multiple exchanges, brokers and multi-dealer platforms, as well as their own pricing, profit and loss (P&L), trade processing and position-keeping systems. The challenge here then is not only the large numbers of systems but also the increasing complicated regulatory environment.
  • Electricity Generation and Metering. The generation and consumption of electricity faces a number of unique challenges including improved and more efficient use of green technologies as well as smart metering. Traditional electrical meters only measure total consumption and as such, provide no information of when the energy was consumed. Smart meters provide an economical way of measuring this information, allowing price setting agencies to introduce different prices for consumption based on the time of day and the season.
  • Retail Systems. As retailers look for ever more cunning ways to get consumers to part with their hard-earned cash, traditional (i.e. high street) and electronic retail will merge more and more. For example not only can I use my 3G enabled smart-phone from the store I happen to be in to quickly compare prices in other stores in the area, the store itself can potentially detect I am shopping there using location based services and make me an enticing offer to shop there.

So here are the seven challenges that I see Interprise Architecture must deal with in developing a ULS business system:

  1. Requirements are unknowable. Sometimes the very act of capturing a requirement (in whatever form) changes the nature of that requirement. Interprise Architecture must not only allow for continuously changing requirements but must also recognise that some uses of the system cannot be known up-front; hence the need to build more adaptable systems.
  2. The boundary between people and systems is at best blurred and at worst never established. Sometimes people will be users of the system, sometimes they (or at least the devices they own) will be part of the system.
  3. Development never stops but is a continuous cycle. Development processes as well as the projects that deliver such systems must therefore support this never-ending cycle.
  4. Systems continuously adapt and exhibit emergent behaviour. As new uses of the system are “discovered’ by users the components that make up the system need to be able to adapt to satisfy those new behaviours.
  5. Failure (of parts of the system) is inevitable. Just like a fire in a building in a city can be localised and extinguished without by and large affecting the whole of a city then so to must Interprise Architecture allow for partial failure and reconfiguration of some components.
  6. Development tools and languages need to take account of the unpredictable and maybe even unspecifiable aspects of systems development. Traditional development tools favour left-brain thinkers where logic and reasoning can be applied to develop systems that move from abstract ideas to physical implementations (from models to code if you like). New tools for developing and describing Interprise Architectures need to be able to inject a bit of right-brain thinking by allowing creative multi-disciplinary elements to be added.
  7. Governance needs to be de-centralised. Strong, top-down governance (the sort favoured by Enterprise Architects) cannot work in a system where all the parts may not even be known. Interprise Architecture needs to recognise that some components are outside its control or immediate sphere of influence and have policies in place that allow new components to be added which don’t harm or damage the whole system.

As an interesting post-script to this the Financial Times recently published an article on Facebook and the plans that CEO Mark Zuckerberg has for advancing his brainchild. Zuckerberg had just announced a new feature on Facebook called Deals which allows smartphone users who have downloaded the Facebook application to check in at a physical location such as a coffee shop and get a reward. Zuckerberg says:

If you look five years out, every industry is going to be rethought in a social way. You can remake whole industries. That’s the big thing.

Facebook is one example of how external applications that allow users to impinge on the enterprise are changing how Enterprise Architects must think.

Next, a story for what a ULS business system might look like and how it might work.

V is for Versatilist

On the IT architecture class that I teach in IBM we have a thing about saying IT architects should be T-shaped. What we mean by this is shown below.

Ideally an architect should have a good range of general skills and at least one deep skill. So an architect might be a good Java programmer for example but also have a broader range of skills including project management, negotiating skills, SOA or whatever.

A Gartner research note I discovered recently called The IT Professional Outlook: Where Will We Go From Here? (from 2005) predicted that by 2011 “70 percent of leading-edge companies will seek and develop “versatilists” while deemphasizing specialists.” It defines a versatilist as someone “whose numerous roles, assignments and experiences enable them to synthesize knowledge and context in ways that fuel business value”. This diagram better shows the versatilist skills therefore:

I like this because not only is the versatilist a V-shaped sort of guy, denoting a broad range of skills at a greater depth of understanding and practice, I believe these skills should be cross-discipline and yes, maybe even consist of right-brain as well as left-brain skills.

As mentioned in a previous post I believe that we architects need not only breadth, to quite a level of depth, but also a good range of skills across all disciplines if we are to come up with new ways of thinking to solve some of the wicked problems out there. Architects should therefore by V(ersatilist), not T-shaped.

Architecture for a New Decade

Predicting the future, even 12 months ahead, is a notoriously tricky past-time. Arthur C. Clarke the English scientist and science fiction writer who sadly died in March 2008 said:

If we have learned one thing from the history of invention and discovery, it is that, in the long run – and often in the short one – the most daring prophecies seem laughably conservative.

As we move into a new year and a new decade the world, in my memory at least, has never seemed a more uncertain place. As the doom and gloom merchants predict an even more troubled economy and the ongoing threats from global warming, increasing pressure on dwindling natural resources and yet more wars do not make for a happy start to 2010.

Whilst solving these truly wicked problems is slightly beyond me I am left to wonder what this new year brings for us architects. According to Gartner the top 10 strategic technologies for 2010 include a number of things which we as an architect community need to be getting our heads around . Of the list of ten, there are three technologies in particular that interest me:

  • Cloud Computing
  • Advanced Analytics
  • Social Social Computing

Whilst it is easy to get consumed by the technology that these new architectural “styles” bring to the table I think the key things we as architects need to do is:

  1. Gain sufficient understanding of these architectural styles to be able to articulate their benefits (and of course their risks) to clients.
  2. Understand what the real difference between these technologies and the one that went before it are so we can build solutions that take advantage of these differences rather than more of the “same-old-architecture” in a slightly different guise.
  3. Figure out how we sell these benefits to the really important stakeholders (the RIS’s).

I reckon that in 2010 being able to identify the RIS’s and convincing them of the business benefits of going with solutions based on technology X is going to be the absolute number one priority. Most businesses in 2010 are going to be struggling to survive and not thinking about IT spends. However survival needs businesses to be both agile and also have the ability to swallow less fortunate companies as efficiently and quickly as possible. Thankfully I think the really good architects that can do this and span the business-IT gap will still be around this time next year. I’m not sure about the rest though?