Bad Night for Big Data

I have a nightmarish pet scenario that as we as a society gain non-stop access to ever-increasing data, there is a risk that we actually get progressively dumber — as we lose the ability to process and analysis that data sufficiently.

My idea got a workout this week during election night when the polling industry, most of whom had predicted a single or even double percentage point Clinton victory, got it monumentally wrong.

When we hear on TV every ten minutes about how Watson is curing cancer, among other breathless hype about Big Data, an error of this stunning magnitude seems at first paradoxical.

But the more you think about it, the more it makes a perverse kind of sense.

“Dewey Defeats Truman”

Embarrassing election errors are nothing new — witness the iconic photo of President-elect Truman gleefully displaying the newspaper headline “Dewey Defeats Truman” the day after the 1948 election.dewey-defeats-truman

People claimed then that the error was due to a combination of slow reporting and the print-era need to prepare headlines hours in advance of publication.

What IS new is that polls are now easier and cheaper to field, and as a natural consequence there is a proliferation of them. And, as they are invariably deemed newsworthy, they feed the hungry news-cycle monster. They generate eyeballs and click-bait — and they’re fun, especially when your own pick is ahead.

Especially toward the close of this 18-month campaign, it seemed like a new poll was appearing every other day. We became so collectively absorbed in the twitching poll dashboards that we neglected the fleeting opportunity to discuss in any depth the serious challenges facing our country and our society.

Read the rest of this entry »

More News from the Dark Side

I pay close attention to feedback I receive on the KVC and other analytic frameworks we are developing.  Many times I make revisions based on this feedback — that’s why the KVC Handbook is now on its fourth major edition.

One of the things I’ve heard is that the KVC model is too idealistic.  Even as I confess to being idealistic by nature, I think that’s a fair criticism.  And thanks to your feedback I use this blog (and my Knowledge Clinics) to address things not in the current edition of the book.

In the private sector, examples of damaging deviations from the ideal are as easy to spot as this morning’s Wall Street Journal.  Last month, for example, I outlined the issue of what happens when the Knowledge Value Chain is broken by chance — or corrupted by intention.

Intelligence in war

This month we examine a case that has been in play for a while, from public affairs in the US.  (Though even our readers in South Africa and elsewhere should take note — things like this could happen there too!)

In general the issue is the reliability of intelligence in an active war theater — here the ongoing actions against ISIS.  Does this sound familiar?  It should — read my earlier post about General Michael Flynn’s criticism and subsequent reshaping of the intelligence effort in Afghanistan.

And those of you who (like me) are baby boomers will remember this issue as it played out in Viet Nam.

Read the rest of this entry »

The Research Matrix

The other day I received an email from “Susan”, an alumna of the Columbia IKNS program whom I had the good fortune to work with as one of my students there. Susan’s question to me was on research, which that program touches upon but doesn’t cover in great depth, and in which I have lots of experience.

Susan’s client is exploring the feasibility of entering a new industrial services market related to energy.  Susan was looking to me for guidance about how to structure and price her research proposal to them.  For my purposes here, it doesn’t matter what the industry is — though I can say it’s B2B, and not one of those industries (like technology or health care) that is often in the news.

The value question

I prefer to demystify things wherever possible, so I proposed that Susan start by determining the client’s value question: What do they want to find out, and why (i.e., what do they plan to do with the answer)?  Susan had studied the KVC model at Columbia, so she knows how value is created from knowledge by making decisions and taking actions based on that knowledge — and that Planning (“Step 0”) starts at the top.

Columns A and B

Then I went into TV detective mode. Working down the chain:  what do we know, and what do we want to know?  We essentially have two columns, A for what we know, B for what we don’t know.  A is essentially our “value-relevant knowledge inventory”, and B is our “open to buy” knowledge purchase order.

Our mission

Simply put, our mission is to move things systematically from Column B to Column A.  That’s the essence of the research agenda, represented in the diagram by the orange arrow.

Except that it’s not often that simple.  (Bet you could see that coming.)  Knowledge users/clients are people, and people are essentially and eternally curious.  So Column B starts life as a set of Minimally Required Knowledge (MRK) to solve our business problem — our “need to know” items.  (Philosophers call this paring-down approach Occam’s Razor.)

Read the rest of this entry »

Meetings Will Make You — or Break You

Most of us work in virtual meetings often, some of us almost exclusively.  People call in using Google Hangouts, Skype, GoToMeeting, WebEx, JoinMe, Free Conference, and so on.  (I’m speaking here of “virtual meetings for the rest of us,” not the high-end meeting rooms costing hundreds of thousands.)

The hybrid meeting

I’ve been part of and/or hosted lots of physical meetings, and lots of online meetings.  In a meeting I hosted the other day, I encountered a variation — the digital-analog hybrid, where some of the people are remote and some are in the room.  (I’m in New York City, where everybody passes through at one time or another.)

It’s amazing what happens around the table that people on the phone do not have access to.  Off-mic side comments, glances, smiles, tones of voice — a panoply of meta-meaning that provides richness and context, and that only those physically in the room benefit from.

This led to a misunderstanding with one of my three co-presenting colleagues who was not present in the room.  I should mention that presence or absence had nothing to do with how important each person was to the meeting, nor how important the meeting was to each of them.  It was based simply on their availability to come to NYC at that time.  (Some were connected from as far away as Europe.)

The problem

For my purpose here, it’s not so important what exactly transpired.  Let’s just say it was a miscue deriving from an agenda item that was modified at show-time by consensus of those physically around the table — but not made clear — and here’s where as meeting leader I fell short — to those calling in. In other words, it was a problem enabled by the hybrid nature of the meeting.

What I’d like to share with you — because many of you may experience this too, with regard to meetings, or other conflicts with peers — is how we resolved it.

Read the rest of this entry »

Forward to the Past

I used our recent office relocation to review some files I had not visited in a while. Though an arduous undertaking, it provided some surprising rewards. Among other things, I came across one of my first major projects, done with the great firm of Peat, Marwick, Mitchell — which soon after became KPMG.

Screen Shot 2016-02-10 at 1.41.56 PM

Out client was the Department of Taxation and Finance for New York State. I worked under a business economist, Don Welsch — a brilliant and fun guy, a real visionary.

My job was to develop a model for the economy of New York State, segmented into more than 100 sectors. The State wanted to have a computer model ready to go so they could change sales tax rates (read:  raise taxes) in selective categories if they needed to rapidly plug a revenue gap.

I had in effect become the firm’s reigning guru in sales tax during a previous engagement, and before that had studied data analysis and linear modeling at Yale. This was a unique opportunity to leverage both cognitive streams.

Things were different then

It hardly needs saying that many aspects of this assignment were much different then than they would be today. It was the Digital Dark Ages! (Though of course we didn’t know that at the time. We were just trying to get our work done in the best way possible.) The Internet was still a dozen years in the future. Personal computers were just barely on the horizon, and nowhere to be seen even at well-funded firms like KPMG.

Don had a relationship with Chase Econometrics, a timesharing service that we used for time-series data by dialing in on a TI Silent 700 terminal through an acoustic coupler, into which we plugged a landline phone. Results came back to us dot-matrix printed on proprietary thermal paper at a blazing 300 characters per second.Screen Shot 2016-02-10 at 1.53.17 PM

That was a quick sprint through the infotech graveyard — but, for its time, our process was pretty high-tech. The Silent 700 was one of the first portable terminals to achieve major commercial penetration, so we could work from an office (ours or those of our client), rather than have to go to a data center. Many of our consulting colleagues regarded us as futuristic space cadets.

Read the rest of this entry »

Piercing the Enterprise Bubble


Harkness Tower, Yale

A few weeks ago I attended a reunion at my alma mater, Yale University.  As they always do, Yale offered up some of its most articulate faculty and administrators to describe the current state of affairs at the University.

The array of talent, initiative, and innovation on display was dazzling.  By the end of the two days, many of those of us who attended college decades ago were ready to sign up for another round — things have changed that much in the interim.

The Yale bubble

One surprisingly interesting session featured current administrators and faculty commenting on the current state of the University.  One dean mentioned what she calls the Yale bubble. It seems that students expect, and routinely receive, such high levels of performance from themselves and from the institution that they experience culture shock when they step outside its boundaries.

As one current student put it, “At Yale, it can be easy to get absorbed in our work, our activities, and friends. It can be easy to surround ourselves with a nice little Yale bubble.”  She goes on to describe how she and some of her friends broke out of that bubble to raise money for a disaster relief effort after a hurricane in the Philippines.

More recent events could be interpreted as showing the downside of the Yale bubble — a potential loss of balance and perspective as to what really matters.

Corporate culture — for better and for worse

Every enterprise creates its own nexus of practices, protocols, traditions, mythologies, and values — strands that together weave the fabric we call corporate culture. When you count over 300 years of history, $24 billion in the bank, and US presidents and other world leaders among your alumni, as Yale does, it’s easier than average to pull this off.

But every enterprise builds this cultural bubble, whether intentionally or not, and whether successfully or not.  It’s an essential part of what binds people to the enterprise — and thereby to each other — in collective pursuit of some common goal.

In some cases the enterprise bubble is “bubble-istic” — fluid, transparent, and porous.  It alternately expands and shrinks to fit new circumstances.  It is welcoming and inclusive.

In other cases, the enterprise bubble is made of steel and concrete.  It is hard, inflexible, exclusionary, and restrictive.  (North Korea might fit this model, for example.)

Read the rest of this entry »

The Competitive Runway

I read the following headline recently in the Wall Street Journal:  “Consumers crave [PRODUCT], but [PRODUCERS] enjoying their best profits ever are reluctant to switch.”  (The words I’ve bolded here were specified in the article, but I’ll get to that in a minute.)

Headlines reminiscent of this have been written many times in business history.  They are often prelude to disaster in the form of self-imposed obsolescence.


Regarding Kodak, for example, one might in the late 20th century have written [digital technologies] and [film manufacturers] in the respective slots.  The profitability on film was so great that Kodak persisted in making and selling it, and famously did not invest soon enough in a switch over to digital. This was a titanic strategic blunder from which the company never recovered, eventually filing for Chapter 11 bankruptcy in 2012.

Willful ignorance

It happens constantly, in all industries, that consumer preferences migrate — sometimes so slowly that it’s hard to notice — until the change has become the new normal.  It’s more noticeable in B2C industries than B2B, but it happens in the latter too.

What makes these changes especially difficult to respond to is our near-universal tendency to gloss over and ignore that which could be unpleasant — or even fatal.  Our “all good news, all the time” corporate cultures make it tempting to look the other way and hope such problems will resolve themselves before metastasizing.

On the other hand, companies that have an innovation philosophy that demands that they “Obsolete ourselves before someone else does” have the upper hand.  Intel and Jobs-era Apple were famous for thriving under such regimes of continual, relentless self-betterment.

Read the rest of this entry »

Frameworks and Lenses

We recently had a discussion at Columbia’s Information and Knowledge Strategy program about consolidating, or at least coordinating, the various analytic frameworks that many of the faculty use in their work and teaching.

The consensus seemed to revolve around the idea that there is an optimal number of frameworks — that while too few leaves you with gaps in your perceptions, too many leaves you confused about which is “better”.

What is a framework?

A framework is a perceptual filter through which we see the world — in this case, the world of organizational behavior.  It’s a narrative overlay that integrates and orders disparate and dynamic elements of reality.  At best, it makes tangible that which is inherently intangible (“organizations” and “knowledge”, for example.)

A framework is like a lens that enables us to see what would otherwise be invisible.  Without such a lens, the world is a fuzzy undifferentiated mass — it’s nearly impossible to see clearly what is going on.  With a good framework, it’s possible to discern patterns — “shapes,” if you will — to diagnose what is awry, and even to predict what is likely to happen under certain conditions.

What about that lens analogy?

As a serious amateur photographer, I know the value of lenses.  When you’re starting with photography, you concern yourself most with the features of cameras themselves — how many megapixels does it display, does it have GPS and WiFi, and so on.

After you log substantial hours behind the glass, you realize that what really matters is the lenses.  A beginner will buy a lens almost as an afterthought.  An expert will notice subtle differences among similar lenses, and may even be able to tell which lens — but probably not which camera —  captured which image.  (In case you didn’t know, serious photogs don’t snap photos, they “capture images.”)

Canon 50mm lensAs you develop as a photographer, you often find that your needs and tastes change — even for what in some respects are interchangeable lenses.  A 50mm lens, for example, is a popular lens — it “sees” similarly to the human eye, and is so prevalent that it’s called a “normal” lens. It serves a range of needs, and is usually among the lenses you acquire first.  Most manufacturers make them, and some make several flavors.  Canon (my favorite vendor) currently makes four different 50mm lenses, each of which has its own characteristics and capabilities — and prices ranging from $125 to more than $1500.  As with so many things, you get what you pay for (at least to some extent.)

Read the rest of this entry »

Knowledge Leadership

I recently had the pleasure of meeting the class of incoming students at Columbia’s Information and Knowledge Strategy program and conducting an introductory learning session with them in tandem with Kate Pugh, the program’s academic director.

Take it from the top

This master’s degree program is designed in an ingenious way.  Unlike many programs in the knowledge disciplines that begin with the “What” of how to execute them, at Columbia we start with the “Why” — which in this case means the organizational strategic imperatives that drive all enterprise activity, including (but not limited to) knowledge activities.

I flashed the following table on the screen. The first dimension is familiar to most of us. STRATEGY is where we’re going; TACTICS is how we’re going to get there.

STMLThe second dimension is perhaps less familiar, but also of great importance.  Where management as a discipline is nearing its half-century mark, leadership is both less-understood and more-discussed lately.

Read the rest of this entry »

Drivin’ that TRAN(E)

The opportunities revealed by the Knowledge Value Chain derive mainly from taking data, enriching it, and applying it to support decision making in more innovative and value-enhancing ways.

But are there inherent characteristics of certain data that make it more valuable than other data? Absolutely.

two locomotives

Five factors come to mind as distinguishing one set of data from other, even before considering what further processing it has undergone:  timeliness, relevance, accuracy, novelty, and exclusivity.  In order more of mnemonic expediency than importance, we’ll call these TRAN(E).


Like radioactive atoms, all data have a “half-life” — a period beyond which their usefulness becomes progressively much more limited. They say that yesterday’s newspaper is good only for wrapping fish — because the “news” it contains is no longer “new”. As conditions change, informational descriptions of such conditions must change at a corresponding rate — or lose their potency.

If you’re a stock trader, stock pricing data is essentially useless by the time it’s published “for the rest of us” 20 minutes after the fact. By that time, any market-moving information is already reflected in the stock price. You’re too late to push the button.

Read the rest of this entry »