Organizational Entropy, the Problem of Choice, and the Effect of Organizational Structure on Performance (part 1 of 4)

Have you ever wondered why your company is “stupid,” and why it just seems to get stupider as it grows? Shannon Entropy and Productivity: Why Big Organizations Can Seem Stupid (paper) by Dr. Richard Janow sheds some light on this phenomenon and shows that organizations really do get stupid as they get big (though I personally would call it something more akin to “scatter brained” than “stupid”). Here’s the description of the paper from Dr. Janow’s CV:

Accepted for publication in Journal of the Wash D. C. National Academy of Sciences.

The first application of Shannon-like entropy to decision-making in organizations suggests that there is a fundamental upper limit on per capita decision efficiency that decreases as organizations grow larger. The result can be impaired efficiency in utilizing intellectual capital (knowledge workers) unless re-structuring is consciously designed to limit entropy effects. Quantitative tools that can help manage organizational entropy and productivity in business firms and in command and control applications are suggested (patent pending).

There are many interesting implications of the analysis. For instance, “the maximum per capita management decision flow rate … actually shrinks as the number of decision-makers in the network grows.” This is somewhat counterintuitive at first glance (you might think that as more people can make decisions those decisions would be made faster). But this is actually the “stupidity” that Janow mentions in his title – as the organization grows, it has a harder time keeping up with the required decisions, thus appearing a bit “slow”.

The “proof” of all this involves quite a bit of detailed math (Dr. Janow is, after all, a physics Ph.D.), but the results are fairly straightforward. In addition to describing the implications, Dr. Janow provides some concrete ideas on how to manage organizational entropy. The best one-line recommendation, from the abstract, is “[t]he smallest size organization that has the resources to handle a task is preferable.” In practical terms, we see this every day. Large companies are broken up into functional units, business units, corporate staff, etc. But even doing this can create its own set of problems.

For instance, say you have a large organization with a great deal of organizational (Janow) entropy. You re-structure the organization so that instead of a small number of large, high entropy sub-organizations that communicate widely throughout the organization, you have a large number of small, low entropy sub-organizations that communicate through a single point to other parts of the organization. You have improved the performance of the new small sub-organizations, but now you have created a whole new layer of highly entropic communications between the sub-organizations. You also now must deal with the prospect of overload at this single communications point, resulting in the possibility of loss in communications between sub-organizations.

Janow acknowledges that many of his recommendations have been used intuitively by organizations, but believes the quantitative formulation he introduces in the paper “can be the basis for modeling tools that will make the tuning of organizations for high performance a much less chancy and ad hoc process.”

A stitch in time saves nine

Recently, I listed 4 high-level parts of an information and knowledge management continuum: personal / individual, group / team, organizational, and external interaction. The first three can be seen as a “path” that many (most?) organizations follow in their creation and growth. (The last – external interaction – is a key function for the other three that I won’t get into here.)

First, an individual has an idea. Using various personal information and knowledge management type techniques (e.g., basic research, remembering where they wrote down notes and contact information, etc.), the individual refines the idea until it has potential value for others besides the individual.

At this point, the individual finds like-minded individuals to form a “team” to further develop the idea and start the process of getting it to market. Not only does each individual need to do some personal IM / KM, but the two (or three or four) must now work together and figure out how to manage the info and knowledge they are generating as a team.

Once a product is to market, or even before, the team will expand into an “organization.” Even if it is not a large organization, certain key functions will become “standardized” – such as payroll, HR, etc. In addition to the personal and team IM / KM going on, an organizational infrastructure develops to handle the needs of the organization as a whole. Sometimes (probably far too often) this infrastructure develops ad hoc and by default, with little thought given to a design or long term goals. Individuals and teams must now work within this infrastructure as well as within their own.

Ideally, the organizational infrastructure will support not just the IM requirements required for the smooth operation of the organization but will also easily support and accomodate the IM/KM needs of the individuals and teams that actually do the organization’s work. For well established organizations, especially large organizations, this is likely a very difficult (though not impossible) goal to achieve.

Smaller companies looking to grow or individuals just starting out with an idea, however, are in a much better position to build (grow?) an effective IM/KM structure and culture from the ground up. Unfortunately, there seems to be little short term benefit in long term IM/KM planning for start-ups who are focusing on simply making it work.

Blogs as Organizational Tool

As you’ve probably noticed, this blog is a bit of stream of consciousness, a bit of pre-planned thought, and a lot of wandering around. I truly use it as a personal wastebook (wasteblog?) and you will likely see many topics come round and round. I am also in the process, though, of putting together a plan for using blogs as a means of content management (loaded term, I know) and info/knowledge sharing for organizations.

Looking for some ideas on how to use blogs as an organizational tool, I came across several good ideas, such as Blogs in Business: The Weblog as Filing Cabinet from Dave Pollard’s how to save the world. Especially useful, and a good jumping off point for getting more good ideas, is More Corporate Blogging Resources from Amy Gahran’s …Contentious.

In addition to the increasingly common corporate uses for blogs (internal sharing, face to the public, etc.), I’m looking at using blogs as a key part of a web site for a K-12 school for everything from school announcements to PTO shared blogs. Any ideas or examples are welcome.

Should I build or should I grow?

Though Knowledge Management is much (much much) more than information management, the fact is that information management is a key element (you could say requirement) of effective KM. Information technology just happens to make info management easier and harder all at the same time.

One of the challenges in coming up with an effective information strategy is figuring out what works best for the type of organization that you have*. There is no single strategy that will work for every type of organization and, best practices and knowledge management and all that stuff aside, even organizations that are very much alike will likely use their information in vastly different ways. This is because organizations are made up of individuals and every individual does things his or her own way. (I thought about qualifying that a bit, but I don’t think that would be appropriate. Even in the most structured environment, individuals will find little tweaks and tricks that make the work go faster, better, more enjoyable, whatever.)

“Old style” organizations, especially those in the manufacturing arena, were very structured in their design and operation. From the outside looking in you might say that the organizations were built, likely using a very structured blueprint and plan based on best practices, scientific management theories, etc. Like a building or machine, the foundation was set, and everything was developed according to the plan. When you are finished, you typically end up with what you started out to make.

“New economy” organizations, particularly knowledge based organizations, are more fluid in their design and operation. From the outside looking in, it may seem that the organization was grown more than it was built. Much like a garden, a general plan is established (in the form of boundaries and goals) and the appropriate seeds planted and nourished. What you end up with may not be exactly what you were planning on, but chances are you will end up with something much better than what you imagined in the first place.

While there may not be any one right way to create and maintain an organization, it is important that you make a plan and stick with it. In other words, strategy by design not strategy by default (which, using the garden analogy above, will likely result in a wild field of weeds with a few pretty flowers mixed in randomly throughout).

*(Of course, this implies that you know what type of organization you have – mechanistic or organic, scientific management or new economy.)

Thoughts on knowledge work as craft

From Knowledge Jolt with Jack: KMPro with Jim McGee:

What does it mean to be responsible for one’s own knowledge management? How does one go about doing this? How do we approach this question knowing that we will move from position to position in the work world? How do we manage if our organizations do not formally support this knowledge craft?

and Knowledge Work as Craft Work:

One useful direction for some additional poking around would be to investigate how to apply the lessons learned in software development around version control and source code library management to more general forms of knowledge work. Wouldn’t you like to have that level of tracing over your powerpoint presentations or correspondence files? Doing that today takes a level of sophistication that the average knowledge worker or knowledge work group doesn’t have.

Like many (most?) knowledge workers, I take great pride in my work. Though I’d never really thought of it in the terms of a “craft” before, it does make sense. As knowledge workers, we take raw materials and use our skill, knowledge, and experience to transform them into something of value. Or we take a problem and figure it out to provide a solution. There is no one right answer, though some answers are better than others.

Just as in the “physical” crafts, such as carpentry, etc., you can categorize knowledge workers at different levels, from apprentice through journeyman up to master. As you start in the trade, you are in a learning mode, primarily focused on figuring out how to do things while also “getting the job done.” For the most part, you are told what to do and how to do it, with some feedback from the more experienced craftsman on the process you used and the quality of the end product.

If you are a union craftsman or tradesman (carpentry, electrician, plumber), chances are there is also a formal education process where you get the “book” learning you need to improve. Even so, it is your responsibility to learn what you need and to apply it.

As you advance in skill, knowledge, and experience to the Journeyman level you become more responsible for your work. You may still be told what to do, in terms of assignments and tasks, but how you accomplish the assignment is more up to you. At this point, you may also be responsible for supervising an apprentice’s day to day performance, imparting some of your skills, knowledge, and experience to the “new” guy. And then on to Master.

Craftsman in the physical crafts are responsible for their own knowledge and, importantly for this discussion, their own tools. While I took the “knowledge worker” route, my brothers are both tradesman/craftsman. One thing I learned from them, which I never would have guessed myself, is that for the most part they are responsible for owning, maintaining, and knowing how to use their own tools. Obviously, the company they are working for will provide the big, infrastructure things, but when it comes down to the actual tools needed to get the job done, each individual will have their own tools and their own way of using them. A craftsman with mediocre tools may have the skill needed to get the job done, but may be limited by the tools. A craftsman with the best tools and mediocre skills may look good getting the job done, but will likely produce a mediocre result.

It seems to me that it is much the same in the world of knowledge work – workers can be limited by both their tools and their skills. It is up to the individual worker to find the best tools for the job and to understand how to use them. Some organizations will see the value in building an in-house knowledge work-force and encourage and provide access to training and allow individual workers to use the tools that best suits them. Other organizations will not see the value in nurturing this type of environment, but will still expect the individual knowledge workers to be able to perform at high levels. As in the trades/crafts, in many ways it depends on where you work and what you want out of your professional life.

As in anything else, once you’ve figured out where you want to go you need to make a plan on getting there (allowing, of course, for detours and unexpected opportunities, along the way).

Unfortunately, this doesn’t address Jim McGee’s bigger question of “visibility” in knowledge work and how to establish an effective training program along the lines of “apprentice – journeyman – master.” Definitely something to think about….