Ground Truth and Social Sensors

Ground truth is the unfettered and unfiltered relating of people’s experiences within a human social system. It can be associated with a specific event at a particular point in time, such as personal interviews with survivors on December 26, 2004 in Sri Lanka shortly after the devastating tsunami struck. It can be the review of a series of experiences across a period of time such as follow-up interviews with those same Sri Lankans who were interviewed originally to understand how their circumstances are changing. It can be the stories told about how members of a social system, who experienced a catastrophe like the tsunami, adapted their social, political, and economic structures in response; namely, what worked, what didn’t, and what changes to make. Ground truth is given when people speak for themselves. It is the ONLY way a human social system knows what is REALLY going on.

People who deliver their ground truth are acting as well-functioning “social sensors” in a broader social system. They draw upon a wide range of information and communication technologies (ICT) e.g., websites, email, blogs, wikis, with land-line, cellular, and satellite connections from locations throughout the world, no matter how remote, to relate their experiences instantaneously and continuously. Social sensing parallels similar functions within mechanical and biological systems. In fact, there is a point of convergence between scientific and social sensor development paths that establishes the possibility of two working together in a highly interrelated manner that enables large, complex systems to be better managed.

Sensor technology, too, has it roots in ICT. The earliest application of ICT during its commercial development was data collection. People would make manual entries into databases of data they and others collected utilizing various measuring devices at certain points within factory or office operations. The computer would use programs to analyze those data and put them into an informational format that could help interpret what was occurring and develop responses to improve the processes or procedures.

One of the first areas where ICT quickly developed in the 1970s and 1980s was in sensor technology. With increased capability and reliability, sensor technology contributed significantly to the replacement of humans as the means through which data was collected and entered into databases. Further developments over the last 15-20 years greatly reduced the size and power requirements for sensors, and increased the sophistication and range of type and capability of sensor technology. Now, sensors are pervasive; they influence almost every aspect of our lives and endeavors. And they continue to displace people from those activities where consistency, repetitiveness, quality, and reliability are essential for effective and efficient operations.

Of particular significance in these developments and displacement is the degree of integration and compatibility between what is being sensed and what is doing the sensing. Today, extensive sensor networks are carefully nested within all manner of systems: mechanical, chemical, optical, biological, and social. Regardless of application, these sensor networks monitor and evaluate conditions which become feedback in larger, adaptive systems that devise corrective strategies and take appropriate actions in response. The key to their success is the accuracy and timeliness of their input as well as the pervasiveness and comprehensiveness of their coverage.

In many ways, the human body is a complex web of sensor networks. Millions of nerve receptors of different types and functions are distributed throughout the body and send continuous signals through the central nervous system to the brain where they are processed and given responses. And like any sensor network, the quality of the response is tied directly to the quality of the input.

Despite highly evolved and elaborate redundancies that function effectively the vast majority of the time, our senses can be fooled: hot can feel cold and vice-versa; we see mirages we believe are real; we hear sounds when there is silence; odors we smell and taste evoke memories that do not accurately reflect what we are experiencing in the moment. And as in the interplay between sensor networks and the larger systems they help regulate, there are different ways of analyzing and processing input with each eliciting different responses. Furthermore, we can ignore sensory input or respond in ways that override evidence suggesting a more appropriate course of action. So, regardless of how well-designed the system and how well-refined the processes, the arbitrariness and irrationality of our decision-making have the potential to bring it to naught.

Like the human body, human social systems are vast sensor networks. Each member of the system is a “sensor” who “reports” on conditions as they are experienced. The system – comprised of hierarchical political, economic, and social structures that operate according to sets of self-serving rules – sorts, aggregates, and analyzes data entries from sensory members in an effort to understand, interpret, determine response possibilities, consider alternatives, and decide on a course of action. Of course when considered on a global scale there are myriad social systems in play simultaneously. Members of one social system can concurrently be members of others. Interpretation of sensory input in one social system can elicit a different response compared to what happens in response by another social system. The key determinants are rank, status, and position in the formal structure and presence, voice, and passion in the informal structures.

Human social systems are analogous to the human body in other ways. There are over 6 million people in the world. The human body consists of billions of cells. Thousands of people die and thousands more are born every day. Millions of cells in the human body die daily and millions more are regenerated. Of the thousands who are born, live, and die each day, I have the opportunity to know only a handful. I know my body, in general, through its organization by function, role, and relationship of one part or system to another. Most of it I will never see and I don’t have to; I trust that it will do what it should without my deliberate attention if I follow simple rules of good health in terms of diet, nutrition, exercise, and rest. Similarly, most people in the world obey the rules of the social system to which they belong. These rules present choices and people decide in ways that permit them to adapt to current circumstances, but preserve the integrity of the system. Behavior is managed and people stick by the intent of their roles, responsibilities, and relationships.

What happens when taking care and following the rules is not enough? Even when we do our best to prevent it, inevitably, our bodies get sick. Sensory cells we seldom hear from send messages that indicate they or the systems to which they belong are in trouble. Depending on the nature of the condition they are signaling there is a wide array of prescriptive treatments from which we can select. These can be non-invasive wherein normal functions of the cells and systems are restored through medications; or invasive through the repair, removal, or replacement of tissue. The same phenomenon occurs in social systems. People in their “sensory roles” relate experiences wherein the system – no matter how well-designed the rules and how noble the principles and ideals that frame them – fails to respond within an acceptable range. Functions break down; remedies are required. In some instances a simple reinterpretation of an existing rule is all that is needed. Other times, though, more radical steps are in order such as rescinding laws and enacting new ones, closing operations and opening others, and eliminating products or canceling services and offering of others.

Oftentimes, we do not heed the early warning signals from our bodies indicating something is amiss and what was once easily restored must now be repaired, removed, or replaced. The sensory networks did not fail, but we chose through our heads or hearts to ignore the input, e.g., “I don’t feel any pain” or “I don’t see any bruises” or to not give the input appropriate attention, e.g., “It will go away” or It’s nothing.” Because the human body is marvelously adaptive, this approach works to some degree, but the performance of the whole and the cellular arrangement and functioning that comprise it are compromised. We live with it in a compromised state or we take more radical steps to correct or reverse the damage.

Again, there is a clear parallel within human social systems. Billions of people in the world have a nearly infinite variety of experiences daily. How do these experiences fit the frameworks of the social systems to which people are members? Where are there anomalies between expectations and experiences? Do these differentials drive responses? Is the system stretched beyond its limits to adequately respond and more deliberate and protracted strategies are needed to spur deeper adaptation? To know the answers requires being attentive to the “sensors.” It means getting to “ground truth” with people in the system about their circumstances.

Establishing ground truth is a three-step process:

Ask people for the truth about their realities and encourage them to tell their stories openly
Hear their truth, once offered, understand it; and commit to respond with appropriate action
Follow-up afterward to confirm that the responses were, indeed, appropriate and that the current situation is corrected and steps are underway for longer term changes in the system preventing recurrence of the problems

Just as we do not heed messages within our bodies we do the same in social systems. To know what is really going on requires ground truth. To not ask, listen, comprehend, and take action are just as effective in shutting down responsiveness and adaptation in the social system as it is with our bodies. Much of time it is for the same reasons: “don’t confuse me with the facts” and “if I am ignorant I cannot be held accountable.” Typically, we do not like change even though circumstances warrant it. In addition, we do not like to know about circumstances where change is needed because we will be challenged to take action – in other words, make it happen. Either way we claim we will lose focus and be distracted from the mission we are locked into at the time.

Just as the health of our bodies is compromised when warning signals are ignored or overridden, social systems become corrupted when the ground truth of members is not heard or heeded. Social systems can continue to function, in general, despite certain levels of corruption, albeit their effectiveness and efficiency are significantly reduced depending on the type, degree, and pervasiveness of the corruption. Change is particularly problematic in established social systems. Power concentrates in the tops of the ruling hierarchies, corruption increases, and along with it an aversion to change that might disrupt the structure, grows. As a result, these hierarchies uphold tenets and “rules” that support the dominant culture remaining dominant.

Ruling minorities become increasingly distant from their ruled majorities. In so doing they become increasingly cut-off from what is really happening within the social systems they are charged to “protect and serve.” Ground truth exists in the heads, hearts, and souls of social system members whether it is sought after and cared for or not. People and their truth, like life itself, will find a way to express itself, even it means setting in motion disruptive patterns of behavior that threaten to totally transform the system in which they exist. David Brooks, in his editorial entitled, “Trade, Oppression, Revenge,” published in the NY Times on December 25, 2005, illustrates this point through a very recent example. The native Indian people of Bolivia, who comprise 65% of the population, dominated for years by a ruling white elite representing 3% of the population and controlling almost all of the resources in the country, used the democratic process to elect an Indian president. What is in store for the ruling minority of Bolivia and their repressive, exploitative policies? Something not nearly as pleasant as it could have been had the ground truth been spoken, heard, understood, heeded, and the outcomes confirmed.

Therein lays the challenge with respect to ground truth: some have to want to hear, some have to be willing to say, and others still have to respond to what the truth means about the design of the system and make changes accordingly. For any sensor to work effectively, regardless of type or application, its input signal must be captured, processed, and acted upon. This certainly pertains to people as social sensors in human social systems: their input is in the form of valuable stories to tell and their experiences constitute important feedback in regulating the function and adaptation of these same systems. Are you asking…and listening?

Originally posted to New Media Explorer by Steve Bosserman on Monday, December 26, 2005

The Problem of Power

Since humanity began its odyssey out of Central Africa millennia ago, hierarchy is the only consistently adopted structure for distributing power within a social system. This structure holds intact the social system in which it exists. To do so, it exercises three roles:

Define “boundaries” – territorial, birthright, and behavioral – that determine who’s in and who’s out
Provide security that protects the boundaries, preserves the lineage, and maintains the behavioral guidelines
Respond to changing circumstances so that the primary social system persists

Hierarchy is easy to install since it begins when one person assumes a dominate position relative to another. It is efficient. The dominant one sets the boundary conditions and subordinates operate within them. When coupled with fear of consequence if one does not stay within the boundaries and the promise of reward if one meets or exceeds expectations the vast majority of members comply with the dictates of the hierarchical social system.

Members of hierarchical social systems participate in governance, free enterprise, and non-governmental affiliations. The rulers set the parameters in which members operate so that government has the resources to assure security, enact rules of conduct, ownership, and commerce, and provide a system of jurisprudence in the event that members do not follow the rules or violate the rights of members. Property owners, business owners, and those who contribute their skills, time, and energy are compensated, pay some percentage of their profit and income to government so that it can function, and voluntarily give to community-based organizations that benefit the commonweal and care for those who cannot contribute. It is a system of cooperation, choice, and commitment that is simple and elegant.

Hierarchy is a structural “tool” for managing social systems. It is the integral framework upon which all institutions – governmental, business, or non-governmental / not-for-profit – are built. As such, it becomes a common denominator that cuts across society in all its endeavors. Like any tool, it is neither good nor evil. How people use hierarchical structure within their institutions and throughout their social system is what imbues it with certain characteristics of morality, ethics, and fairness and places it on a continuum of social justice ranging from barely evident on one end to being a matter of course on the other.

The driving force that shifts a social system along the scale of social justice is the use, or abuse, of power. Hierarchical social systems concentrate power in the hands of a minority number among the overall population. As Lord Acton stated, “Power tends to corrupt, and absolute power corrupts absolutely.” In other words, when holding a dominant position, some are seduced by the power they have over others. This prompts them to do the following:

Stay in their positions of power
Keep the hierarchical social system intact so that the positions of power they hold are preserved
Curtail affiliation among those who are disenfranchised by the system so that the threat of insurrection is quelled.

Basically, people who are so affected are drawn by the possibility of extracting even more profit or garnering more control. They limit choices for subordinates and force compliance to rules and conditions that are not beneficial to members. In addition, they attempt to keep the system as it is so that their gain is consolidated and ideally, continues to grow.

As an example, approximately 50% of the world’s population earns less than $2 / day per capita through subsistence agriculture. The hope of a higher quality of life is dashed each day eking out an existence under the harshest of circumstances. Leadership in these countries is averse to mechanize and modernize agriculture because those displaced would venture to cities in an effort to find work. This would overburden the already congested and over-stressed infrastructures in these urban areas elevating discontent and unrest. Better to keep people where they are so they are contained having to care for themselves. Meanwhile, the economies grow and the top echelons gain wealth.

Another example: according to an article in The Open AIDS Journal entitled, “HIV Infection and AIDS in Sub-Saharan AfricaUNAIDS report,” Sub-Saharan Africa is home to 70% of all people afflicted with HIV in the world. This area holds some of the world’s richest natural resources, yet the population is in grave peril, health-wise, including leaders. In fact, that is the point. The world has known about AIDS / HIV in Africa for nearly 30 years and has only in the last five mobilized efforts to do something. All indications suggest it is too little, too late. Millions have died from AIDS already and millions more will do so over the next two generations as the endemic unfolds. Sub-Saharan governments are rendered ineffective due to a lack of competent, experienced leadership, citizens are not able to care or fend for themselves, economies slow their growth, and property ownership and control shifts to foreigners. Colonies once lost due to revolution are reclaimed – without firing a shot.

As yet another example, consider genocide. In his 1994 book, Death by Government: Genocide and Mass Murder since 1900, author R.J. Rummel defined and documented “democide,” e.g., atrocities committed by governments on their citizens. Of the nearly 170 million who died at the hand of their governments during the 20th Century, 130 million, over 75%, were victims of four regimes: Soviet, Communist Chinese, German, and Nationalist Chinese. This does not include Rwanda in 1994. And genocide continues as the people from Darfur have borne witness every day this year. A weakened population becomes weaker. Those who hold the most power in the world act as though they are powerless as these horrific injustices go unchecked. The reality is there is nothing to be gained by intervening and much that could be lost if circumstances go awry. The people of Darfur, like the millions before them, are collateral damage in an economic power struggle of global proportions.

In his unsettling essay, “Waiting for the Lights to Go Out ,” published in the October 16, 2005 edition of The London Times, Bryan Appleyard posits a rather grim future wherein civilization is doomed to return once again to the Dark Ages. The central theme of his essay is the meltdown of society as we know it due to our insatiable addiction to oil, the inevitable depletion of oil reserves, and our woefully inadequate and untimely response to that eventuality. At the heart of his thesis is that human nature has not progressed very far since we began our migrations throughout the world from our African origins. Appleyard states, “Our aggressive, tribal nature is hard-wired, unreformed and unreformable. Individually we are animals and, as animals, incapable of progress. The trick is to cage these animal natures in effective institutions: education, the law, government. But these can go wrong.” Not particularly encouraging!

Rulers enact laws that are increasingly restrictive, militaristic, and draconian or become arbitrary in their decisions. Owners pull more profit, become more risk averse, and offer fewer benefits. Having less voice and impact, non-governmental organizations become more fanatical and close-minded; their ratios of administrative overhead to pay-out increase, and beneficiaries receive less real assistance.

In summary, the thirst for power and the specter of losing it becomes a corruptive force that undermines the fundamental tenets of a system that is given to efficiency and effectiveness if used honorably. People suffer for lack of true justice. Is there any way institutions, no matter how well-intended, can escape the slippery slope into corruption and injustice? Or is this simply the dark underbelly of hierarchy which must be accepted until people have more distance in time from their early violent struggle for survival as Homo sapiens? What is your opinion?

Originally posted to New Media Explorer by Steve Bosserman on Monday, December 12, 2005

Pareto and the Pyramid of Power

The year 2006 marks 100 years since Vilfredo Pareto noted that 20% of the population owned 80% of the property in Italy. This conclusion combined with the analysis methods that support it led to the formulation of the “Pareto Principle.”

The tools and techniques associated with this principle have widespread application in circumstances where a small subset of one category causes a significant effect within a large subset of another category. During his work with industry, Dr. Joseph Juran coined the phrase, “the vital few and trivial many,” to describe the effect of the Pareto Principle in the business setting. For example, a company has a staff of ten sales representatives. Of the ten, the three highest performers generate 60% of the sales made during a reporting period, the five middling performers bring in 35%, and the two lowest performers contribute only 5%. Armed with this information, responses can be developed that are tailored differently for each of the three groups in an effort to increase overall sales performance for the least investment or cost. Another example is in a set of 100 manufacturing operations where the overall defect rate is .1%. There are 10 operations that cause 90% of the defects. Addressing those 10 are going to have a much more positive effect on the performance of the whole than focusing attention on combinations of the remaining 90 operations. Like most rules of thumb, the Pareto Principle can be misused; but in general it helps prioritize activities, separate the important from the pesky, and focus limited energy on the items that are going to make the most difference.

The Pareto Principle had it birth in economics, a social science. Given this background, there is another application for the Pareto Principle that covers additional ground. When this basic postulation – a small percentage of the population owns a large percentage of the property – is bracketed by two corollaries – a small percentage of the population enacts and enforces a large percentage of the rules that govern the behavior of the overall system and a small percentage of the population receives a large percentage of the compensation awarded by the total system – the resulting triad describes a fundamental truth about social systems: a small percentage of the population controls a larger percentage of the power within the whole system.

The exercise of power in a social system establishes an individual or group in a dominate role and subordinates the larger population of individuals or groups within that system. The population size can range from two – one person in relationship with another as in a marriage – to one over millions as in a country ruled by dictatorship. Regardless of population size, structure is required to maintain a requisite level of control over myriad dominate-subordinate relationships in the system so that the system persists. This structure is hierarchy.

Hierarchical social systems impact the people within them in three key ways:

Each person belonging to a hierarchical social system has hierarchical relationships with all others in that system
Each person has concurrent membership in multiple hierarchical social systems and can hold positions at different levels from one hierarchical social system to the other
All hierarchical social systems concentrate power in the hands of a select few.

The universal symbol for a hierarchical structure is the triangle. Authority is held at the top, then distributed in varying degrees, level by level from the top to the bottom. However, a more appropriate geometric symbol for a social system is a three-sided pyramid (reference image below). One side of the pyramid is the hierarchical structure of governance. People participate in governance by making / changing the rules, enforcing the rules, and obeying the rules. Of course, there are always those who choose not to obey the rules. They are subject to some consequence levied by those who enforce the rules in the interest of what is called justice. The net result is that only a few operate at the top of the governance triangle to set and manage the rules while the clear majority obeys.

Another side of the pyramid is free enterprise. People participate by doing work that adds value for which they are compensated. They exchange that compensation for other goods and services they need and want. Wrapped up in free enterprise are concepts of property and ownership, money and capital, business and entrepreneurship, markets and customers. Wealth, in the form of assets, rests in the hands of a few.

The third side of the pyramid is affiliation. People participate by joining different groups and organizations which represent shared beliefs and ideals, customs and traditions, principles and values. These groups carry out activities that promote “causes” shaped by their worldviews. They provide forums for members to have voice and presence concerning their perspectives and interests. Those having the greatest access and influence are those who have the highest positions in the hierarchies of these organizations and represent the strength in numbers or vantage point of their memberships.

These three arenas: governance, free enterprise, and affiliation define the landscape in which organized human endeavor is conceived and carried out. The pyramid they form is a pyramid of power that develops and deploys human intelligence, energy, and skill to build, adapt, and sustain civilizations. While the three are vitally important as standalone systems, the interrelationships among them determine the effectiveness and efficiency of the whole. People vote for their leaders in governance. People make and buy based on supply and demand. People form non-governmental organizations to give body and shape to their views and interests. People derive power from the pyramid in unequal portions from the three triangles but regardless of the combination, the power they get is sufficient to stay in the system and work together so that the system persists.

Originally posted to New Media Explorer by Steve Bosserman on Saturday, December 10, 2005