Big Data’s Dirty Little Secret

In my youth we lived on the corner of a tiny street in suburban Philadelphia called Shepherd’s Lane.  At the other end of the street, less than half a city block away, the sign read Shepards Lane. The discrepancy bothered me at the time (though not enough to compel me to find out which was the “correct” spelling.)

A store by any other name

Years later I thought of this while analyzing a massive table of client data from several thousand retail stores.  By eyeballing the data I realized that we had a pretty serious “data thesaurus” issue — with physical sites listed under several slightly different alternative spellings/misspellings in many instances. Before we could do any meaningful analysis, we needed to concatenate all those records that called the same physical store by different names in the database.

This is intuitively obvious to a human analyst, but surprisingly challenging for a computer, primarily because of the potential number and randomness of the variations.  We could have developed a reference sub-table of each potential alternative spelling — but we had neither the budget nor the time for that.  And since this was a non-recurring analysis, it wouldn’t have been worth it.  We cleaned the data manually, iteratively, on the fly.

How data scientists spend their time

A recent article in the New York Times “For Data Scientists, ‘Janitor Work’ is Hurdle to Insights” discusses a key processing step called data wrangling or data munging — basically all the work you have to go through after you collect the data, but before you start to actually analyze it for meaning and implications.

Data scientists are said to spend between 50 and 80 percent of their time collecting and cleaning data before it can be explored to yield useful nuggets. Anyone who has worked professionally with data knows this — but, as the article points out, “It is something that is not appreciated by data civilians.” (Love that term!)

Read the rest of this entry »

Ebola Reveals the “Unwieldy” State of our Medical Records

In late September 2014 a man showing symptoms of the Ebola virus came into the United States from West Africa, was examined by doctors at a hospital in Dallas — and was then released back into the community.  There he came into contact with other people before finally being readmitted to the hospital, where he died within a matter of days.

Our radar failed

If the entry of the Ebola virus into the US wasn’t itself shocking enough, one doctor claims that this will not be the last time such an error occurs — due to what she calls “a simmering crisis in medical data management.”

ebolaWriting in the New York Times on October 14, Dr. Abigal Zuger warns, “Even scarier than that mistake is the certainty that similar ones lie in wait for all of us who cope with medical information stored in digital piles grown so gigantic, unwieldy, and unreadable that sometimes we wind up working with no information at all.” (My emphasis).

Dr. Zuger claims that “hospital servers store great masses of trivia mixed with valuable information and gross misinformation, all cut and pasted and endlessly reiterated.  Even the best software is no match for the accumulation. When we need facts, we swoop over the surface like sea gulls over landfill, pick out what we can, and flap on.  There is no time to dig and, even worse, no time to do what we were trained to do.”

Read the rest of this entry »

Knowledge Leadership

I recently had the pleasure of meeting the class of incoming students at Columbia’s Information and Knowledge Strategy program and conducting an introductory learning session with them in tandem with Kate Pugh, the program’s academic director.

Take it from the top

This master’s degree program is designed in an ingenious way.  Unlike many programs in the knowledge disciplines that begin with the “What” of how to execute them, at Columbia we start with the “Why” — which in this case means the organizational strategic imperatives that drive all enterprise activity, including (but not limited to) knowledge activities.

I flashed the following table on the screen. The first dimension is familiar to most of us. STRATEGY is where we’re going; TACTICS is how we’re going to get there.

STMLThe second dimension is perhaps less familiar, but also of great importance.  Where management as a discipline is nearing its half-century mark, leadership is both less-understood and more-discussed lately.

Read the rest of this entry »

Knowledge Strategy Simplified

I’ve been preparing to help Guy St. Clair teach his course Management and Leadership in the Knowledge Domain. This is part of Columbia University’s exciting new master’s degree program in Information and Knowledge Strategy, which I am pleased and honored to be part of.

This for me is a perfect opportunity to link my ideas about competitive strategy with my ideas about information management.  If you’ve read any of these posts, you’ll know that I believe strongly that the latter should serve the former, with few exceptions.

Guy and I will be presenting a high-level view of how to develop “knowledge strategies” in a range of industries. We’re using as our examples investment banking, pharmaceuticals manufacturing, advertising, and higher education. All are relatively knowledge-intensive as industries, yet each faces significantly different challenges at both strategic and tactical levels from each of the others.

Three basic questions

In approaching these industries, as I would approach any new client, I find the answers to the following qualifying questions essential in setting a direction for further diagnosis.

VM-CS-KS1. What is their value model? How do they produce revenue? Is the value produced solely economic, or are other value components (like societal benefit) also in play?

2. What are their competitive differentiators? How do they make themselves more attractive than their rivals? How are these factors weighted in importance and investment?

3. What knowledge/information do they require to support these value-producing and competitive factors, at both strategic and tactical levels?

Once I have reasonably complete answers to these three questions, I can begin to get a handle on where they need to be in terms of knowledge strategies.

You may realize that this is essentially a truncated version of the Knowledge Value Chain model. It drives down from the top — the organizing principle is the value model, which is then supported by — and even “given life” by — the knowledge model and infrastructure.

Read the rest of this entry »

Drivin’ that TRAN(E)

The opportunities revealed by the Knowledge Value Chain derive mainly from taking data, enriching it, and applying it to support decision making in more innovative and value-enhancing ways.

But are there inherent characteristics of certain data that make it more valuable than other data? Absolutely.

two locomotives

Five factors come to mind as distinguishing one set of data from other, even before considering what further processing it has undergone:  timeliness, relevance, accuracy, novelty, and exclusivity.  In order more of mnemonic expediency than importance, we’ll call these TRAN(E).

Timely

Like radioactive atoms, all data have a “half-life” — a period beyond which their usefulness becomes progressively much more limited. They say that yesterday’s newspaper is good only for wrapping fish — because the “news” it contains is no longer “new”. As conditions change, informational descriptions of such conditions must change at a corresponding rate — or lose their potency.

If you’re a stock trader, stock pricing data is essentially useless by the time it’s published “for the rest of us” 20 minutes after the fact. By that time, any market-moving information is already reflected in the stock price. You’re too late to push the button.

Read the rest of this entry »

“Knowledge” is a National Security Issue

The headline in today’s Wall Street Journal caught my attention:  “Bringing Jobs Back to the U.S. Is Bruising Task.” The article describes the Reshoring Initiative, a nonprofit group that helps companies return production to the US. 80% of these companies are relatively small, with revenues of $200 million or less.

The article describes that a key barrier to bring work back is the declining quality of the labor force in the US.  One of the issues is the lack of a good work ethic.  One company reports that when the announced that a drug test would be held for plant workers the following day, 20% of the work force didn’t show up for work.

2 + 2 = 5

But an even more fundamental problem — as has been widely reported before — is that American workers often don’t have the skills needed to run even basic manufacturing machinery.  The owner of a scented candle company that originally sourced in China and Vietnam reports that even elementary school math skills are absent in many of her US employees — including both line workers and supervisors.

As I understand it, making candles ain’t brain surgery.  If US workers can’t run candle making machines, how would they do in a factory making advanced electronics, or even cars?

This brings to mind the various surveys that continually report the low and falling test scores of US students relative to those in Asia, Europe, and elsewhere. (See, for example, data below from the 2012 Programs for International Student Assessment that originally appeared in the Washington Post.) This is not just some abstract global game of Jeopardy — it ultimately matters to how people put food on their table.Screen Shot 2014-06-26 at 5.23.30 PM

And to the future of the US economy as a whole.  While we in the US pride ourselves on our consumer-led economy, it’s clear that people can’t continue to consume stuff indefinitely if they don’t have adequate jobs.  The consumer-led economy only works when the people who buy stuff do so with money made by working in factories that make stuff.

Read the rest of this entry »

Strategic Knowledge – What’s New is Old

I take pride in bringing the latest techniques and insights to my clients.  But it turns out the origins of what I do go back at least 2500 years.  I discovered this while reading Lawrence Freedman’s new sprawling history of all manners of strategy, called (fittingly) Strategy:  A History.Strategy cover

Professor Freedman traces the modern use of the term “strategy” back to the 19th century, when it was used by von Clausewitz and others to describe the application of empirical science and reason to the conduct of military campaigns.

Strategy = leadership

But the term “strategy” ultimately derives from Greece in the fifth century BC, when a military leader or strategos was expected to “lead from the front, fight with the best, and show total commitment.”

Thus strategy is, at its core, about leadership.  The leader, be it in military or business affairs, is the one (to quote leadership expert Robert Reiss) whose job it is to guide the group from HERE to THERE — however those start and end points are defined.

Read the rest of this entry »

KVC Handbook 4.0 Launches

We’re pleased to announce the publication of the latest Version (4.0) of our Knowledge Value Chain Handbook.  Here’s an excerpt (with added emphasis) from the Introduction.

KVC Handbook Cover.bmp CROPKnowledge is a fundamental resource of our economic lives. If you were going to enter the business of, say, manufacturing airplanes, you would want to hire people with substantial applied expertise in the sciences of metallurgy, electronics, and even polymers—all the basic building blocks of your product.

Most of us work and compete in what is widely acknowledged to be a knowledge-based economy. But we do not have a science of knowledge—because there isn’t one yet. Philosophy has a branch called epistemology that discusses the origins and characteristics of human knowledge—but that doesn’t qualify as a science in the sense that technologies and management practices can be consistently derived from its principles.

Even the basic economics of knowledge are open to study and debate. Classical economists like Adam Smith generally did not regard knowledge as a key economic component. The fundamental  “factors of production” were land, labor, and capital—with knowledge and information barely mentioned as being essential to production.

Read the rest of this entry »

Up Close and Personal: Our $2.7 Trillion Cottage Industry

I recently had the experience of intensively studying an industry as a consultant — then subsequently (and unrelatedly) becoming a client of that industry.

The industry, as you probably have guessed, is health care.  There are things you learn when you are a patient that you never see when you’re not. (I’ve heard doctors report this as well.)

In the past year I spent a few hours as a “client” in each of two major New York hospitals.  The net results from a clinical perspective were excellent, and I’m relieved to report that in both cases their work is now finished.

One of these hospitals was ranked near the top in NYC hospital quality by the US News and World Report polls, the other somewhere down in the low 40’s.  But my observations apply to both.

Systemically speaking, hospitals as seen from the inside seem almost artisanal in nature.  Very hands-on, customized, laid-back — and, well, expensive.  It’s as if you were a guest in the city’s best private club, where no expense or courtesy is spared.

This has both positive and negative consequences.  On the one hand, the leisurely pace and attention are welcome, especially when one is in a state of discomfort and/or anxiety.  The most obvious negative — the cost of doing things this way — is often hidden, at least until later.

The quest for cost

Inside “the club”, of course, money is never discussed.  It’s just not done — and if you try, you don’t get very far.  In one case I asked my doctor’s office assistant what the cost of a routine, elective outpatient surgical procedure was going to be.  I was honestly trying to decide if it would be “worth it” at this time.  I was told that I would not pay more than 20% of the cost, due to my excellent private insurance plan coverage.  “Sounds good…but 20% of what number?”  was my obvious question.  “Well, let’s see,” she responded, clearly taken aback by my question. “The surgeon’s fee is $7500, the anesthesiologist is probably about $1500…then there is a hospital fee, I don’t know what that is.”

My positive feelings for my doctor notwithstanding, I couldn’t stop myself from mentioning that $5000 per hour sounded like a lot.  “Of course you won’t pay 20% of $7500 — it will be based on whatever your plan has negotiated with the surgeon.  It could be much lower. You’d have to call your plan to find that out.”

Not wanting to give myself sticker shock, and only having so much time in my day to spend on this, I did not pursue her recommended option.  When weeks later the Explanations of Benefits (“the tab”) arrived, she was shown to be nearly 20% low on the professional fees — which were actually $9000 for the surgeon and a little over $2000 for the anesthetist.  But the part she didn’t know — the hospital charge, was over $26,500 — more than twice the combined fees of the three doctors involved.  My afternoon in the hospital cost $37,900 “list price”, of which my insurance plan paid the “discount price” of $17,200 (a discount of nearly 55%).

Hospital costs a

Her advice to me not to worry about it too much, though, was sound.  The cost to me was nothing.  (Thanks, excellent health care plan!)

Read the rest of this entry »

Calibrating Your Value Model

The holiday break provides us an ideal time to calibrate and refocus our efforts and build for the year ahead.  Given the dynamic nature of the business environment, I recommend you consider these two simple questions:  If I were starting my enterprise today, what would I be doing different from what I am doing now?  And what (other than inertia) is holding me back from doing those things now?

The changes we don’t see

Changes in the competitive business environment come slowly.  In the heat of battle — producing and selling what we produce and sell — we often don’t see change…until it has already passed us by.  We tend to do the same things we did last year — maybe a little faster, a little better, a little more efficiently.

But we forget that the market, the competition, the entire business environment is different from what it was a year ago, or even a quarter ago.  Ever so slightly different, perhaps — but then, big changes typically begin life as small changes.

Consequently, the value model that is central to our product or service offering may not be as, well, valuable as it once was.

Read the rest of this entry »