Wednesday, January 11, 2006

Think Tank Meeting 5 Jan 2006

Summary:

Prior to the 05JAN2006 Think Tank meeting, members reviewed two articles: “Combating Cognitive Biases in Information Retrieval” by Rafael Alonso, and Hua Li and “Observations on ‘Cognitive Factors in Design’” by Thomas T. Hewitt. The group discussed the method by which the Anti-Bias Mechanism (ABM) (in the former article) worked to develop an alternative query in Internet searches conducted by analysts. The group agreed that a “Google” styled evaluation of information, found to be commonly relevant by other analysts (unions), has some utility. However, the group was cautious in claiming that this system can effectively fight analyst bias. The group felt this would be ineffective at combating and in fact may promote groupthink.
The group also discussed “problem-solving schemas”, from the latter article. As a person encounters problems, and more importantly successfully solves problems, they develop “schemas” (or standardized structures for solving problems). Try this equation: A+2b+a=? The average person would attempt to solve this problem using an algebraic problem-solving schema. A person may ignore the anomaly that there is no integer or value on one side of the equation and therefore the problem is in its most reduced form. A person may try to fix the problem to fit their schema by adding an integer: A+2b+a=X, however this is not the original query and they have merely “patched up” the problem to fit the schema. The answer is Abba. We would attempt to argue that this problem is incorrectly stated, and that although it is not a complete algebraic equation it contains algebraic language (2b). However, this only emphasizes the point. Although some of the question fits the algebraic model for problem solving, the anomaly tells us we are using the wrong schema. Here in lies the problem: if we misidentify the query and apply the wrong schema to solve it we will inevitably have errors in our analytical processes, regardless of their accuracy. From these concepts, the group identified what it considered to be one of the most dangerous biases: the “paradigm”. It was the belief of the group that the ABM, or other known anti-bias techniques could not redress this bias.
Our answer to this bias: imagination. In past meetings the group has discussed what makes good analysts. The analysts stated that analytical sciences and techniques could be taught to make acceptable analysts, however certain qualities of exemplar analysts are innate. Many historical “failures of intelligence” (9/11, Pearl Harbor, Iran [Shah/Ayatollah], etc.) are characterized as “failures of imagination.” The think tank believes it may be extremely beneficial to identify some “jumping off points” between empirical analyses and imaginative analyses, doctrinal and counter intuitive thinking. The group poses the question: “Can we identify turning points or cues in the analysis process when the current use of a paradigm should be questioned?” One possible answer to this is the systematic search and identification process of anomalies. The think tank concluded with the assertion that, perhaps, analysis should be “revolutionary” in nature. That is to say, it should be fearless, and ready to question the very foundations of its own reasoning, and cast off paradigms, which have in the past proven fruitful. This again brings into question the validity of “experts”, and experienced analysts. The think tank wonders what effects long-term dedications to paradigms have on the analyst. We pose a series of questions for further inspection using primarily psychological principals: Were analysts so engrossed in the era of counter-Soviet intelligence ready for 9/11? Who is the more effective analyst: the Lance Corporal (entry level analyst), or the Colonel (experienced analyst)? What weight should “expert” opinions have on analysis? Can “expert” knowledge be presented in a digestible manner, which a non-expert analyst may use effectively? How can we know when and how to be imaginative? How do we learn to be revolutionaries, and how do we know when it is appropriate?
Professor Hoff was careful to mention that biases have evolved in the human psyche as survival tools or techniques, which improve cognitive abilities. There is a question of what extent human intuition can be trusted. The group has resolved itself to consider biases to have positive and negative qualities based on their context.
We hope to answer these questions by reviewing existing work in the field (such as Heuer’s book Psychology of Intelligence Analysis, the afore mentioned articles, etc.) and new applications of existing psychological principals. To this end we are resolved.

MINUTES OF THINK TANK MEETING 06JAN2005

1) MEET AND GREET

2) ATTENDANCE:

Prof Hoff
Halen Allison
Lance Ramsey
Mark Blair

3) OPENING SHOTS:

a) (Mark Blair): I wanted to remind everyone that Heuer’s book Psychology of Intelligence Analysis is available on line at http://www.cia.gov/csi/books/19104/ In the future this book will be a subject of review for us, and you may wish to do some preliminary reading or print it out ahead of time.

b) (Mark Blair): I would also like to point out that the Mercyhurst Intelligence Studies Dept as published a new book elaborating on Heuer’s concept: Analysis of Completing Hypothesis (ACH), entitled Structured Analysis of Competing Hypothesis. I have acquired a copy for Prof Hoff.

c) (Mark Blair): Gentlemen, Halen Allison and I have created an online blog for the “Mercyhurst Psychology of Intelligence Analysis Think Tank.” It can be found at http://www.psychintel.blogspot.com/. Please go online and review it. Ensure that there are no issues that we have discussed absent, and ensure that it properly represents our meetings to your recollection.

i. Once I have a thumbs up from all of you I would like to send out invitations to the faculty of the Psychology Dept and the Intelligence Studies program to review it and give feed back.
ii. Furthermore I would like to invite Col Walters of III Marine Expeditionary Force and Col Millen of the Strategic Studies Institute to review our blog and leave feedback.
iii. Finally, I believe Dr Owoc of the Anthropology Department could provide us with unique viewpoints and exposure to paralell concepts such as human agency. I also believe any curriculums that may result from our work may have interest to anthropology students. An early interaction with the Anthropology Department will allow us to address their concerns and interests in this subject. For these reasons I would like to invite Dr Owoc to join our online dialogue.
iv. I would like unanimous consent for these invitations. (Unanimous consent is given).
v. I would also like to encourage anyone else with suggestions or possible contributors to bring them up in this or future meetings.

4) The two text were discussed simultaneously.

a. (Prof Hoff) What did each of you think about the “Combating Cognitive Biases in Information Retrieval?”

b. (Halen Allison) I think this has great potential for document exploitation. I would like to get a hold of this program to test it. I think it would greatly benefit our efforts in the Venona Project. But, I’m not sure it would combat some of the more standardized biases analysts face, such as mirror image biases etc.

c. (Prof Hoff) Maybe someone can explain this system a little better for me. It reminds me of Amazon.com. When you purchase a book it comes up with a list of books that people bought who also bought the book you are purchasing. Conceivably it then monitors what books from that list are bought by others or even just “click on.” Then it refines the list accordingly.

d. (Mark Blair) Yes that was my take. I noticed, to my understanding, striking similarities to how Google works. Google monitors search parameters, again to the best of my understanding, and then records what people select or click on. Through this learning process Google learns what results are relevant to search parameters. The Anti-Bias Mechanism (ABM) records queries which individual analyst rate in terms of relevance. The common findings, or unions, are assumed to be unbiased. These unions form the bases for the development of alternative queries, with which the ABM returns findings on the original query along with findings of the alternative maps. These findings are considered “immunized.” The ABS uses the search parameters to identify other related and relevant search parameters to theoretically provide results of alternative questions. That is to say the system assumes the question/queries may be biased or exclusive and not include other possibilities. Like, asking, “Where are the terrorists”, this question does not provide for the possibility that there are no terrorists present.

e. (Prof Hoff) In the Amazon example there may be a book you would like that no one has purchased and the system cannot provide for that contingency. Plus there is the potential that these findings may serve as social pressure. (Especially with the less experienced analyst.) By providing the most viewed or best rated (in terms of relevancy) a person may be subject to a social pressure to assume this view. I could serve as a computerize groupthink or ban wagon effect.

f. (Mark Blair) I agree, but I’m not sure the alternative results are identified as being different from the result based on the original query. I think that the ABM may also have a monitoring utility. If people can monitor the findings of the ABM it may be possible to identify groupthink or common fundamental errors. Especially if it has a wide dissemination. It may be able to identify “regional”, that is interoffice or interagency, etc. beliefs. By studying and comparing this data we may be able to identify and resolve differences in analysis in the intelligence community as a whole. It may also reveal failures of communication between agencies. For instance, one agency may have info or intel that they are using in their analysis that another agency is without. A system that could identify discrepancies in analysis may also identify discrepancies in information. I think also that the process involved may be helpful to understand and could possibly be utilized outside of the computer realm.

g. (Halen Allison) Yes, concepts like “Devils Advocate” should be part of analysis standard operating procedures (SOPs).

h. (Mark Blair) This is of key importance. The other paper “Observation of Cognitive Factors” mentions problem-solving schemas. People develop these problem solving schemas from encounters with problems in their past. The danger of biased analysis lies in the misidentification of a problem, and the application of an inappropriate problem-solving schema to a non-related problem. Consider the equation A+2b+a=. The majority of people, presumably, would, at least initially, adopt an algebraic problem-solving schema. Many may ignore the lack of an value after the equation (i.e. =2 or X). A person may try to fix the problem by adding an integer such as A+2b+a=X. However, they are misapplying an inappropriate problem-solving schema to a problem. The answer is, of course, Abba your favorite and mine. So the question, in this case, is not over the nonsensical form of the question, yet a question of what do you bring to the table?

i. (Prof Hoff) I think we should be careful here. Biases evolved for a purpose. Biases are adaptive and not necessarily bad. Systems designed to remove biases could inevitably be harmful in some cases. We should consider positive and negative transfer. (Psych definition from Contemporary Behavior Therapy by Spiegler & Guevremont: Transfer- Process in which what is learned and practiced in one setting [such as therapy] carries over to other settings [such as home].) We may gain from considering the Blink Approach, by Malcolm Gladwell? His studies ask, “To what extent we can trust human intuition.” One danger we must take a look at is the existence of paradigms. Take scientific paradigms for instance. When some one subscribes to a paradigm, they not only become rationally tied to it, but also emotionally tied to it. Paradigms are used to define the world around you, and the way you conduct yourself. Paradigms may start to have flaws or anomalies. There may be things or instances that are the exceptions to the paradigm. Those who are considerably invested in the paradigm may begin to “patch up” the paradigm. But pretty soon you have patches everywhere as the paradigm continues to fail in other situations. The young guys, new to the field, don’t have as many problems with this. It’s the old guys who can’t let go of the old paradigms who push these paradigms beyond their validity. It’s the young guys, who based on the old guys work, break the mold and bring on new paradigms.

j. (Lance Ramsey) I agree it is often the ones who are fresh to a field to precipitate the evolution of the way we think.

k. (Prof Hoff) But it’s more than that. Its not evolution, its revolution man! The old way is completely cast off in favor of a new and better paradigm that is not in conflict with the anomalies of the old paradigm that accumulate over time. Take Einstein of instance: How did Einstein cast of the old Newtonian view of physics and radically redefine the universe. It’s that fundamental change of the basics that constitutes revolution. The old rules didn’t work anymore so he reinvented the concepts of physics, for instance: relativity. It’s imagination that allows you to go beyond a currently excepted paradigm. History has proven that it’s difficult to go beyond these paradigms. Imagine the first time it was suggested that the Earth was round or the Earth revolved around the Sun and not vice versa.

l. (Mark Blair) If it is a question of revolution, well we find ourselves on the precipice of revolution. After the 9/11, the WMD question, etc. anomalies have present themselves in the old paradigm of intelligence and everyone is scrambling to patch it up. I believe that the “good stuff” lies in the details. That is to say, often the bricks at the bottom of the wall are the ones most likely to weaken the wall. But how do we learn, and teach others to have imaginative processes that are able to reach to the very foundations of thought and problem solving to identify these anomalies?

m. (Prof Hoff) This is the question that many ask. You hear it all the time. Bosses in a board meeting ask, “Lets think outside the box”, only to be met with silence. Paradigms are often a question of comfort. The ABM gives us the terrain we are comfortable in, but it can’t lead to radical anti-biases mechanisms. It may generate ideas or directions we haven’t thought of but I don’t think it gives us new eyes.

n. (?) We have new interests in critical thinking, which I think is good, but its missing a creative quality.

o. (Mark Blair) How do we regard this critical vs. imaginative thinking? Where is the line, and how do we identify it? Can we identify a point where one type of thinking is more appropriate than the other?

p. (Prof Hoff) I think these two types of thinking can be complimentary. We may consider anomaly centric thinking or imaginative thinking. That is we may identify anomalies and then try to go into new, unexplained territory through an imaginative process.

q. (Mark Blair) Often this is how analysts think. We ask ourselves: “What do I know?” “What don’t I know?” etc. There was an example Prof Grabelski gave in one of his presentations which demonstrates this point. A young girl went to her mother's funeral. Though sadden she met a man she had never seen before. She became convinced the he was the man for her, however she did not get his phone number before leaving. For the next couple weeks she fell into depression. Two weeks after her mother’s funeral, the young girl murdered her sister. Why? When I heard this story I immediately zeroed in on the strange man. For me the relationship of the strange man to the family was an anomaly. It was what “I don’t know.” I was but a hairs distance from figuring out the entire case. However, there was a failure of imagination. I assumed there was an answer to this question. Someone knew this man, or knew why this man was at the funeral. However, if someone knew the man the young girl would have, conceivably, asked everyone and gotten in touch with him. So the young girl has the same issue as I do. Yet, the man for some reason knew the family or at least the mother, and cared enough to come to her funeral. So the young girl deduced that he would also come to her sister’s funeral. So she murdered her sister. Perhaps, if we can identify anomalies then that, in itself, is an indicator to turn on the imagination, and let go of the paradigms. I guess that seems rather obvious.

r. (Prof Hoff) I think it is clear that old paradigms are not only captivating of those who have used them for an extended period of time, but it renders the users incapable of understanding news ideas. (Provided they can’t let go of the old paradigm.)

5) Next week, Mercyhurst Psychology of Intelligence Think Tank Meeting 12Jan2006: The group decided to review “Cognitive Rigidity: methods to overcome it” by David W. Robson for next week.

6) Closing Comments: Thank you Gentlemen for attending this meeting. Thank you to everyone who may review our minutes. We would like to invite feedback from all who feel so inclined.

Tuesday, January 03, 2006

Think Tank Meeting 6 Dec 2005

Summary:

Prior to the meeting, members reviewed an article by Ross Stapleton-Gray, Ph.D., CISSP entitled “Watchin’ the Analyst.” The full text of this article is available in appendix A, at the end of this text. Discussion Points are as follows:

1. Right to privacy and clearances: What expectation of privacy should a person granted a clearance have?
a. The analysts agree that persons who are awarded clearances necessarily give up some expectation of privacy. The questionare (EPSQ- SF 86) of a clearance investigation is by nature a declaration that the subject of the investigation has nothing to hide.

2. Compartmentalization: Restrictions across compartmental lines may encourage unofficial venues of communication (i.e. Blogs, Chat rooms).
a. Compartmentalization has traditionally and in contemparay times been viewed as intrinsic to securty. In post 9/11 days compartmentalization has been frowned upon. Decenters have sighted lack of inter-agency communication and accessibility as a significant contributor to intelligence professionals' failure to predict the attacks of September 11, 2001. As the Stapleton article points out compartmentalization also promote back channel communication by intel professionals.



Minutes of Think Tank Meeting 6 Dec 2005:

1. Meet and Greet

2. Members in attendance:

a. Professor Hoff, Director of Mercyhurst Psych Dept
b. Halen Allison, Student Director of Research, Intel Studies Dept
c. Lance Ramsey, Intel Studies Dept
d. Mark Blair, Intel Studies Dept


2. Discussion: Members discussed the problems and logistics of teaching a class derived from a curriculum for the psychology of intelligence analysis.

a. (Prof Hoff) The Psych Dept has a limited faculty.
b. (Mark Blair) Although at this early juncture the only official feedback we have gotten for the potential curriculum is that it will definitely be considered for an elective. However, from the feedback I have gotten from some of the faculty, I feel there is a definite potential for such a curriculum to become a core class of future Intel Studies Degree Programs. If it is an elective or if it becomes a core curriculum of the intel program, there will be a lot of students, and we may have an easier time justifying dedicating limited faculty to teach the course.
c. (Halen Allison) ~This is not the only utility for a class such as this. A five-day seminar over the summer may prove fruitful. Such a seminar could be taught to, not only, Intel students, but may also attract analysts from the agencies and other organizations.
d. (Mark Blair & Prof Hoof [in a later conversation]) (Hoff) ~We may bear more fruit to our efforts if we are able to appeal to other programs. If the title of the curriculum “the Psychology of Judgment and Decision-making” we may receive offering from other programs like anthropology, philosophy etc. (Mark Blair) There is a bit of a marketing mentality to my thoughts. I am even thinking there may be some interest in the political science dept. I believe the more appeal for this class we can produce, the more successful our efforts will be.
e. (Mark Blair) I am planning to either produce a news letter or briefings on a weekly basis to communicate the findings of the think tank meeting to the faculty of the intel department and allow them a redirect line of communication between the intel dept and the think tank. With the direct feedback and guidance of the intel faculty we may be able economize our efforts. (Prof Hoff) ~What about setting up a blog? A blog may make it easier for us all to communicate and reduce the amount of time individual members must dedicate to the production of briefing products.

Think Tank Meeting 1 Dec 2005

Minutes of Think Tank meeting 1 Dec 2005:


1. Meet and Greet


2. Members in Attendance:

a. Prof Robert Hoff, Director of Mercyhurst Psych Dept
b. Halen Allison, Student Director of Research, Intel Studies Dept
c. Lance Ramsey, Intel Studies Dept
e. Mark Blair, Intel Studies Dept


3. Discussion of each member’s understandings and desires of the out come of the think tank. Responses as follows:

a. Investigation of the utilities of psychological principals to intelligence analysis.
b. Advancement of individual knowledge and the enhancement of analytical abilities.
c. Advancement of the craft of intelligence itself.
d. Identification and/or formulation of Standard Operating Procedures (SOPs), to assess productions of intelligence, both retrospectively, and inclusive in the intelligence process.
e. Formulate a curriculum to enhance the Mercyhurst Intelligence Studies Program.
f. Enhance/Develop interoperability and academic cooperation between the Intelligence Studies Dept and the Psychology Dept.


4. Video: “Judgment and Decision Making”, available at Learner.org (http://learner.org/resources/series138.html program 11) The program provided an overview of several psychology principals of judgment and decision making. Discussion points are as follows:

a. Normative vs. Descriptive decision-making: How decisions ought to be made vs. how decisions are made. Intuitions may sometimes be correct yet they are often intrinsically wrong in their logic.
b. Available Heuristic: The principal that people make decisions based on readily available knowledge.
c. Reasoning by Representativeness or Similarity: When presented with a questions of probability people search for evidence in the question and try to match it with a similar answer. People are often more likely to give a more specific response that fits a stereotype ignoring a broader answer, which by nature is more probable.
d. Anchoring Effect: The principal that the question is posed with a parameter tends to “anchor” an estimative response. Responses, in general, will be pulled toward the parameter.
e. Risk Strategies: People are predictable when making choices which similar values. A person will favor a “sure thing” over a gamble if the outcome is a gain. A person will generally favor a gamble over a sure thing if the outcome is a loss.
f. Under certain knowable conditions intuitions should not be trusted; people are susceptible to predictable errors in decision-making. Knowing these predictable errors give us hope that we may be able to implement procedures to avoid these errors in decision making or at least identify situations in which a person is at risk of making these errors.
g. Groupthink: A condition in which like-minded people continually to reduce the lateral thought. This intellectual inbreeding produces very narrow and often unchallenged viewpoints.
h. Mind-guarding: Members of groupthink protect the status quo from any evidence to the contrary.
i. Devils Advocates and Dissenters: These two concepts are identified as effective tools to combat groupthink and mind-guarding.


5. The group then held a discussion on the points defined by the video.

a. The analysts were quick to point out that the craft of intelligence analysis is only concerned with descriptive assessments. And that any normative issues are only discussed is when giving descriptive analysis a query’s normative tendencies (i.e. cultural norms/ideology). Proper analysis absolves itself of the moral question and only describes what is evident.
b. The analysts also agreed that all of the predictable decision-making errors defined by the video could be found in the practice of analysis. The point was also made that rank and/or authority exacerbates these issues. Furthermore, that “experts” can often present such errors that go unopposed.
c. Reasoning by Representativeness or Similarity: Experts or high ranking members who are familiar with many similar requests for information (RFIs) or critical intelligence requirements (CIRs), may have developed a stereo-type of a culture of people, or a doctrine of a military force. Such expertise, while necessary and often provides extremely insightful analysis, may also run the risk of making decision-making errors. The analysts stated that these “experts” should play a supporting role for the analyst, and the analyst should be charged with making a sober analysis based on expert opinions, empirical evidence, and situational concerns.
d. Question from Professor Hoff: “Do any of you think there could have been instances of groupthink in the Bush administration in the question of WMDs in regards to the Iraq War?” Response: (Mark Blair): Of course. It is almost unavoidable. However, that doesn’t necessarily mean we can identify where and if the analytical process broke down. In fact the question, CIR, or RFI may be more at fault than the response from analysts. If the question, “Where are the WMDs?” is asked, a response of, “There are no WMDs” does not answer the RFI. Since I don’t know how the question was asked I cannot be sure if this is the case. The Bush administration does have divergence of opinion. The members who did not serve in Vietnam (Bush, Cheney, etc.) tend to be the “warhawks”; while the Vietnam vets (Powell, Armitage, etc.) tend to be more reluctant to go to war. So there is some dissention about policy in the Bush administration, which would combat groupthink; however there is an overall aire of American exceptionalism and a neo-manifest destiny feel to the Bush administration. Over all the think tank analysts agree that there was likely much pressure from above to come up with an analysis based on preconceived notions.
e. Review of Material for Next Meeting: The think tank then concluded they would review “Watchin’ the Analyst” by Ross Stapleton-Gray for next weeks meeting.

Purpose and Mission

PURPOSE: Assess and identify the applicability of psychological principals of decision-making to improve the craft of intelligence analysis for the purposes of developing a curriculum of The Psychology of Intelligence Analysis.

MISSION STATEMENT: It is our sincere belief that many well-established psychological principals of decision-making can combat the biases impeding sober and objective analyses of intelligent queries. We believe that if analysts are taught these principals they may enhance the analyst’s objective and subjective analytical abilities and elevate the profession of intelligence analysis as a whole. To this end we are resolved to develop such a curriculum and identify tools for the analyst to test the entire process of analysis from the intelligence requirement handed down to him/her to final conclusions.