ICANN Ombudsman Blog Creating Dialogue Affirming Fairness

October 11, 2015

EIU Own Motion Report

Filed under: Uncategorized — Chris LaHatte @ 7:45 pm

Office of the Ombudsman

Case 15-00110

In a matter of an Own Motion Investigation by the ICANN Ombudsman

Report dated 13th October 2015

Introduction

Part of the jurisdiction for the ICANN ombudsman is to identify systemic problems within the ICANN community. In 2014, I began to receive a number of complaints about the Community Priority Evaluation (CPE), and it became apparent that there was considerable criticism of the way this process was operating and results. By the end of the year I had considered this matter further and sought permission from the ICANN board to commence an own motion investigation. Normally when complaints are received in the office of the ombudsman, it is not necessary to obtain permission from anyone to open a file and commence investigation. Under the Ombudsman bylaws in ICANN Bylaw V, an exception is where the ombudsman wishes to investigate a matter without a complainant, and on his/her own motion.
This investigation therefore began on the 20th January 2015 when the Board Governance Committee of the ICANN board passed a motion authorizing the investigation in these terms:-
“Ombudsman’s “Own Motion” Investigation – The BGC discussed and considered the ICANN Ombudsman’s request for authorization to undertake his “own motion” investigation on the New gTLD Program’s Community Priority Evaluation (“CPE”) process. Consideration of, and authorization for, Ombudsman’s “own motion” investigations is within the BGC’s scope of responsibilities under its Charter. The Ombudsman has advised that he has received a number of different complaints about the CPE process. Because the filing of an Ombudsman complaint triggers a stay on the status of other applications in the same contention set, the Ombudsman has suggested that it would be appropriate for him to undertake his “own motion” investigation into the issues raised in these complaints as well as the overall CPE process, rather than continue investigations on individual complaints, so as to limit interference with individual applications. The BGC voted to authorize the Ombudsman to proceed with the “own motion” investigation he has proposed on the New gTLD Program’s CPE process.”

Facts

The history of the program needs to be explained. I rely on the ICANN New GTLD site for much of this.
Generic top level domains were created before ICANN even came into existence and were subsequently expanded by ICANN with several small programs. After considerable community discussion, at the ICANN Singapore meeting in July 2011, the ICANN community agreed to a major expansion of new generic top level domains. The current new gTLD program was therefore the first major release of new generic names for ICANN.
In 2005, ICANN’s Generic Names Supporting Organization (GNSO) began a policy development process to consider the introduction of new gTLDs, based on the results of trial rounds conducted in 2000 and 2003. The two-year policy development process included detailed and lengthy consultations with the many constituencies of ICANN’s global Internet community, including governments, civil society, business and intellectual property stakeholders, and technologists.
After approval of the policy, ICANN undertook an open, inclusive, and transparent implementation process to address stakeholder concerns, such as the protection of intellectual property and community interests, consumer protection, and DNS stability. This work included public consultations, review, and input on multiple draft versions of the Applicant Guidebook, a guide developed for applicants and others as a manual for the program.
In June 2011, ICANN’s Board of Directors approved the Guidebook and authorized the launch of the New gTLD Program. The Applicant Guidebook has been revised continuously throughout the process to take into account issues which arose. It is significant because it was developed by the ICANN community and therefore the process and procedure are in effect “owned” by the community. Because the program was completely new, it was difficult to anticipate exactly how the procedure would work, but it is accurate to say that it was a complex and sophisticated application process. The ultimate final New gTLD Applicant Guidebook dated June 4, 2012, was issued a few months after the application window closed.
Once the community agreed that there should be community-based applicants, it was necessary to set out a procedure within the process to evaluate the community applications. It was agreed that there should be a panel to evaluate the applications, and the panel which was chosen was the Economist Intelligence Unit (EIU). The EIU has been in existence for around 60 years, with experience in analysis of political, economic and business intelligence, and provision of reports for institutions and governments.
The EIU set up a specialised team to deal with this project, and I quote from their publication:-
“The Community Priority Evaluation panel comprises a core team, in addition to several independent evaluators. The core team comprises a Project Manager, who oversees the Community Priority Evaluation project, a Project Coordinator, who is in charge of the day-to-day management of the project and provides guidance to the independent evaluators, and other senior staff members, including The Economist Intelligence Unit’s Executive Editor and Global Director of Public Policy. Together, this team assesses the evaluation results. Each application is assessed by seven individuals: two independent evaluators, and the core team which comprises five people.”
After the panel was established, the process of assessing the community applications began. As the end of the process approaches, it appears a substantial number of community applicants were not successful, and some of the unsuccessful applicants, including some whose applications were displaced in favour of a community application, were deeply dissatisfied with the process and the results. One application in particular for the dot Gay community application, initially denied priority for the string .gay, was subsequently submitted to the Reconsideration process at ICANN and the ICANN board recommended the EIU evaluate the application again. This was because there was a bundle of letters and documents not considered, which the Board considered a technical error, but which required a new evaluation. This has now been released, but the community application did not gain sufficient points again. It has therefore gone to a contention set with other applications for this string.
It was apparent that there was considerable dissatisfaction, and eventually I determined that it was appropriate to conduct an own motion investigation because of the issues which had arisen.

Investigation

To undertake this investigation I have looked at the procedures as outlined on the relevant webpages for ICANN and the AGB (Applicant Guidebook ), as follows. The page on the ICANN website is http://newgtlds.icann.org/en/applicants/cpe and there is an explanation of the panel process at http://newgtlds.icann.org/en/applicants/cpe#understand together with a FAQ page.
I have also sought comment from community applicants, and those who have replied, have provided me with thoughtful and well reasoned comments. They have not all consistently accepted the fairness of the process of course.

Issues

The issue which I am required to investigate is the community application process for new generic top level name applications, and whether the process was fair and procedurally correct and the whether the processing of the applications by ICANN and the EIU was fair and procedurally correct.
Jurisdiction
The ICANN ombudsman has had jurisdiction to commence an own motion investigation since 2009. This is a power rarely exercised, and in this case there were a number of factors which influenced my decision to seek the permission required under Bylaw V from the ICANN board to commence the investigation. The first of these was that ICANN had decided that whenever an accountability mechanism of ICANN was invoked, accountability mechanisms being an ombudsman complaint, reconsideration request or IRP review, that the application for the new generic top-level name would generally be placed on hold, until the particular accountability mechanism had run its course. If therefore I conducted individual investigations for community applications, that would be treated as an accountability mechanism, and could potentially delay applications. So a generic own motion investigation would not have that effect.
In addition to this the number of complaints coming to me about the community evaluation process made it clear that there was a systemic issue which required investigation. I therefore sought permission from the ICANN board, which was granted. I should add that when I sought the permission, there was no adverse comment or argument against this, but rather a neutral processing of the application for permission.
Issues Raised in Submissions
There were a number of common themes in submissions made to me. These included:-
• cost estimates were not accurate and doubled by the time the applications for CPE were actually filed
• many of the submitters criticised the secret nature of the CPE application process and keeping the identity of the panellists confidential
• several of the submitters complained that there was gaming by those in opposition and activity which was regarded as fraudulent, during the CPE process
• the time taken to undertake CPE was seriously underestimated by ICANN
• there was a lack of transparency in the process, with some calling the whole process opaque
• a number commented on the low number of applications which actually passed (estimated to be 75% failed)
• the process was further complicated by competing applicants who gamed the process after CPE was completed by use of the reconsideration and IRP procedures
• the process was criticised as being overly legalistic, and changed after the applications were lodged, creating additional guidelines
• there was a lack of communication to educate and explain CPE to potential applicants and communities, with only one webinar and a failure to properly market the concept of CPE

Reasoning

Community Applications

All applicants were required to designate whether their application was a community-based gTLD, and there were requirements imposed by the AGB to show a number of features such as an ongoing relationship with a clearly delineated community, seeking a string which strongly and specifically related to the named community, had particular dedicated registration and use policies for registrants including security verification, and the application had to be endorsed by one or more communities representing the named community. All other applications were called standard applications. There were in the end 1930 applications but of these only 4% or 84 were community based. All applicants were told that a formal objection could be filed against any application on community grounds, even if it was not a community application. If there were a number of applicants for the same string, including community and standard applicants, then community-based applicants would be offered CPE. The effect of successfully achieving the minimum CPE score was that the community applicant would prevail. If the community applicant did not score at least 14 points to prevail in a community priority evaluation, then their application would remain in a contention set with the standard applicants. A number of the applicants commented that once the failure to gain sufficient points occurred, a community applicant had virtually no chance of success against a commercially funded standard applicant. This observation appears to be accurate looking at the auction prices (for the not well funded applicants in any event, although that is difficult to objectively assess). However the CPE process was designed to identify whether the community application met the criteria established by the AGB. The AGB notes that “community-based applications are intended to be a narrow category, for applications where there are unambiguous associations among the applicant, the community served and the applied for gTLD string”. There is nothing unfair about this form of evaluation in my view.
One submitter commented that some community applicants are commercial ventures formed by industry insiders, and the lack of funding could be attributed to varied levels of business development than a history of philanthropy or public works. I do not have direct evidence for this, but this is perhaps not surprising. This does however illustrate the point I make later about the limited consideration of scope for community applicants when the concept was first developed.
Fees for CPE
The AGB provided an estimate that the CPE fee could be $USD10,000. Successful applicants would have the deposit refunded. It was anticipated that there may be additional fees, although no indication was given of the amount. A number of the applicants observed that the fees charged in the end were substantially greater and complained that this was unfair especially for communities who did not have the same funding as commercial applicants. It is unfortunate that the fees charged did increase to such an extent, being $22,000 when actually set and charged. The AGB did not commit to a particular figure, but did give an indication. At least part of the problem would have been that no one knew precisely how long the process would take and the extent of engagement between applicants and the EIU. But my view is that there should have been more clarity about possible substantial increases.
Evaluation
At the start of the contention resolution stage, all community-based applicants were to be notified that they could seek community priority evaluation and pay the deposit. Before the evaluation process began, ICANN would have the ability to seek additional information relevant to the community priority evaluation. So if the ICANN staff thought further information would help an applicant, they could suggest the community applicant provide this. However the panel could also request clarification, using the Clarification Questions process.

Criteria for evaluation is explained by this quote from the AGB at Module 4, 4.2.3,
“The scoring process is conceived to identify qualified community-based applications, while preventing both “false positives” (awarding undue priority to an application that refers to a “community” construed merely to get a sought-after generic word as a gTLD string) and “false negatives” (not awarding priority to a qualified community application). This calls for a holistic approach, taking multiple criteria into account, as reflected in the process. The scoring will be performed by a panel and be based on information provided in the application plus other relevant information available (such as public information regarding the community represented). The panel may also perform independent research, if deemed necessary to reach informed scoring decisions.”
A number of the applicants criticised the application process as being “opaque”. I do not believe that criticism is justified because the paragraph cited does give the EIU panel the ability to undertake wide reaching research. When I discussed the research issue with the EIU, they did explain the extent of their research, which was clearly wide ranging and thorough. But the criteria for establishing a community-based application are also described quite clearly in my view. I can sympathise with any applicant who needed to read the AGB to study the process, but the information was there.
The AGB at module 4 sets out the way in which points were to be granted for each of the four criteria. An application would need to score at least 14 points to succeed in CPE, and a number of applicants did criticise or seek to review the allocation of points in particular criteria. There are detailed descriptions and definitions through paragraphs 4 .10 onwards which appear to make it very clear as to the standards which must be met.
So looking at the application process, from identification at the start as a community-based applicant through to achieving the appropriate number of points, in my opinion the AGB provided reasonably clear guidelines for any community-based applicant. I do not consider that the process as outlined in the AGB was unfair to applicants. This is distinct from the processing of applications which I consider next.

Was the Processing of Applications Unfair?

It is a more difficult question as to whether there was unfairness in the processing of the CPE evaluations. In undertaking this investigation I talked to the new gTLD team at ICANN, the EIU staff and also sought submissions as I refer to above. The purpose of the investigation was to see whether any of the evaluations were handled in a way which was unfair to the applicants or failed in the process by not giving them a proper opportunity to present the case for their CPE evaluation. My research was therefore to look at the way in which the evaluations proceeded, and whether the process for applicants met standards of fairness. It would be necessary to find issues such as bias, conflict of interest, failing to permit or read submitted material, unfairly excluding relevant material or failing to give appropriate weight to submissions and evidence.
The AGB and the material provided by the EIU certainly describe a process which should avoid any of the potential problems. The process that the EIU developed, which was approved by ICANN, was designed to evaluate each application in a consistent and objective manner based solely on the AGB criteria. Because this was a new process, then of course it is quite possible that there would be errors and mistakes in the handling of evaluations for CPE. During the course of the reconsideration request for .gay there was an example of an omission, and the result was a recommendation that the EIU perform a new CPE for the application, by doing two separate evaluations together with new members of the EIU core team conducting the assessment. The omission which caused the BGC to recommend a re-evaluation, was a failure to verify 54 letters of support for the applicant, although a substantial number of other procedural grounds were raised by the applicant and rejected by the BGC. The applicant sought to raise every conceivable problem and only succeeded on this specific issue. The BGC recommendation was itself controversial, and certainly not accepted by rival applicants who called this a “harmless error” and criticised the ability for the applicant to have another chance to be evaluated. I have read the analysis of the request, which is useful in the summary of the process and handling of information for any CPE evaluation. It is also a useful example of how the checks and balances in the AGB have actually worked in practice. It is clear to me that there has been a careful and thorough approach by both ICANN staff and the EIU panel in the processing of these applications.
The EIU also comments that they
“reviewed all application materials, including correspondence for the .gay application evaluation. This meets the AGB standards. The EIU took an extra step, as outlined in our evaluation guideline, to attempt to verify relevant letters of support and opposition under certain circumstances. For the .gay evaluation, a single piece of correspondence (i.e. one PDF document) that contained a number of letters of support was reviewed but not verified via email. Again, the verification is not required by the AGB. Moreover, the verification of these letters would have had absolutely zero impact on the score for community endorsement, given the dozens of other letters of support already reviewed and verified.”
I have included this comment for completeness.
The EIU submitted that this point should be made clear, either in the text or a footnote, in order for the reader to understand the scope of the error and the impact it had (i.e. no impact) on the evaluation outcome.

Tension between Standard and Community Applicants

There appears to have been a certain amount of tension in the competing processes between those who wanted names reserved for communities and those who felt that the names should be better marketed by commercial entities. Some of the community applicants had to face criticism and opposition from standard applicants, and sometimes led to some unusual results. I have had comments from community applicants, that the standard applicants were somewhat hostile and aggressive to the community applicants. One standard applicant commented to me that some community applicants were gaming the system by avoiding fair competition. But I do not agree with that proposition based on the information submitted to me. It does illustrate the intense competition however. However in terms of the processing by ICANN and the EIU, this does not appear to have caused any specific unfairness.
For example, .lgbt was successfully obtained by Afilias as a standard applicant over opposition from an international community, The International Lesbian Gay Bisexual Trans and Intersex Association (ILGA). The AGB provided that there could be a community objection, not necessarily of course from a competing applicant. ILGA was associated with the .gay application but had not made a separate community application for .lgbt. Their standing was accepted in the objection determination as an established institution associated with a clearly delineated community as defined in the AGB. But ultimately the panel determined that since there was already an application under way for the .gay string, then the interest of the gay community were protected and did not require further protection by the protection of .lgbt. Since Afilias was the sole applicant, they prevailed once the objection by ILGA was dismissed. But the application for .gay was made by a community applicant although there were also standard applicants for this string. In a reverse twist, the standard applicants have complained about the .gay application as not representing a community, and providing submissions to the EIU in opposition of the CPE evaluation. The re-evaluation has again not succeeded in gaining priority however, as the appropriate score has still not been achieved, in a decision very recently released by the EIU.
I cite the above as an example of the inconsistencies which have arisen between standard and community applications. A common theme of the submissions on community applicants was that they had limited resources, especially when faced with standard applicants with deep pockets. But the reality may be more complicated as it is difficult for me to assess the commercial strength of any applicant. The CPE process was designed to overcome such difficulties, and the procedure was set up whereby applicants could either apply or be invited to apply for CPE, although issues of resources and funding were not actually discussed in this context. But of course applicants and those in opposition were still free to use the community objection phase as well as make submissions in opposition or in support to the EIU. In terms of unfairness, it is difficult to criticise those involved from using procedures which were established in the AGB and available to anyone who wanted to participate.

Small Numbers of Community Applicants

An issue raised by a number of submitters was the low uptake of community applicants. There appears to have been 20 CPE results published (as at 10th September 2015) although there were 84 community applications. Of course if not in contention, then CPE would not be needed. To properly consider this issue, it would be necessary to extensively survey prospective applicants to see why they chose a community application or chose not to apply, and that data is not available. It is clear that an applicant, whether standard or community, would need very substantial resources in order to both apply, and properly operate a registry. It would be logical to assume that community based applicants would have more difficulty in raising funds than a commercial applicant. Even though community-based applications could seek a lower application fee if the applicant could show it satisfied the criteria for Applicant Support, but being a community based application was not a requirement to be eligible for the lower fee. However the backend for any registry and the infrastructure would have similar costing for a community or standard application, which may have deterred some community applicants. There is no doubt there was considerable support within the ICANN community for the concept of community applicants. But commercial reality would have set in when potential applicants considered the merits and costs of such an application. The question therefore arises as to whether it was fair to raise the expectations of those who may have wanted a community application, where the cost and various procedural hurdles could readily be seen to be almost insurmountable. I am not sure that there was sufficiently clear identification of the difference between a “community” and a “not-for-profit”, or “philanthropic institution”, when the original policy was developed. There was clearly a concept that some applicants would be supported by smaller fees and procedural advantages. The transformation of this policy into practice and procedure was somewhat more complex, as has been identified in this report.
My discussions with ICANN staff about assistance provided to community applicants informed me that they went to considerable trouble to assist. There was considerable awareness that some community applicants were struggling, and ICANN staff appear to have provided appropriate levels of support. From my discussions with the EIU panel, they appear to have had a similar level of support from ICANN where needed.
I should also note however, that some applicants did comment that they found the submission of material difficult, and that change requests were not always dealt with quickly or within the expectation of the applicant. I suspect this was largely a communication issue. Other applicants did observe that they had an excellent relationship with the ICANN staff handling any such change requests or provision of further information. On balance I believe that this was handled appropriately.
More recently it was noted that out of the 18 CPE results, 11 have then gone on to further ICANN Accountability Mechanism, which is either an ombudsman complaint, reconsideration or the IRP process. It has been observed that the process resulted in “winners” and “losers”. However out of those 11 reviews only the .gay result was reconsidered and sent for re-evaluation. The writers of this paper (cited below) conclude that the process is likely to have been fair with that level of review.

Anonymity

One common thread of complaint was that the EIU panel was anonymous. Other objection processes such as the string similarity or community objection procedures were different, because the procedure was open, and the names of the panellists appointed to process the dispute resolution were made known. There was an opportunity to challenge appointment, and some took this opportunity. I would observe that there is always a level of suspicion created when an anonymous panel is appointed. On one hand it does avoid the possibility of lobbying the panel members, but on the other hand it can create fears that the persons on the panel were not qualified or were biased or were inappropriate to be on the panel. In a document provided by the EIU panel, approved by ICANN and placed on the ICANN website, the EIU describes the process for evaluation and the process under which the evaluation took place, which reflects the criteria in the AGB. They state that all the evaluators checked conflict of interest, undertook regular training, were highly qualified and multilingual and had expertise in a wide range of areas. So there was a process for ensuring that the evaluators were appropriately qualified persons. I also discussed this specifically with the panel who gave me the same assurances. It should also be noted that CPE is not an objection process. Rather, CPE is an evaluation process, similar to the other evaluation processes where similarly, the individual evaluators were not named. Also, with all evaluations, whether CPE or any other evaluation process, the organization was made public, just not the individual evaluators. Further, the ultimate result of the evaluation is the result of the Organization that was engaged to perform the evaluation, and not of the individual evaluators. This is different from an Expert determination resulting from an objection proceeding, which is deemed the work product of the individual or three member panel.
In addition to the EIU part of the process for evaluation, they also worked with ICANN staff. Most of those who submitted comments said that the staff were very helpful, and it is worth noting that the EIU did specifically say to me that the ICANN staff did not try to influence their decisions at all, but acted as neutral providers of information. Draft evaluations results were sent to ICANN, and staff did ask for wording changes. They frequently asked if the EIU were comfortable with the score. They were neutral but anxious that the EIU was confident with the result. They were careful about the wording, and anxious to ensure that the community would be satisfied with the result.
The EIU note specifically in this context that:-
“We need to be very clear on the relationship between the EIU and ICANN. We advise on evaluations, but we are not responsible for the final outcome—ICANN is.
The SOW states (emphasis added):
(i) the Panel Firm acts as a service provider to ICANN, assessing applications and recommending an outcome, as well as to provide (as applicable) a written explanation setting out its rationale;
(ii) ICANN will be free in its complete discretion to decide whether to follow Panel Firm’s determination and to issue a decision on that basis or not;
(iii) ICANN will be solely responsible to applicants and other interested parties for the decisions it decides to issue and the Panel Firm shall have no responsibility nor liability to ICANN for any decision issued by ICANN except to the extent the Panel Firm’s evaluation and recommendation of a relevant application constitutes wilful misconduct or is fraudulent, negligent or in breach of any of the Panel Firm obligations under this SOW;
(iv) each decision and all associated materials must be issued by ICANN in its own name only, without any reference to the Panel Firm unless agreed in writing in advance;
It should be clear in this document that we recommend/advise, but are not responsible for the final outcomes.”
This is a point which has not been considered by any other submitter. So in the end it will be up to ICANN to make a decision on this recommendation. The final outcome is, as reminded by the EIU, a decision for the ICANN board.
On reflection, it may have been better to have a more transparent process for this evaluation, by not using an anonymous panel. This will be an issue which the community will need to consider in the next round of applications for community names, when the next edition of the AGB is produced for that round. It is not possible to recommend changes to the existing applications, for the practical reason that virtually all of them have been completed. But having an anonymous panel is of course something which cannot lead to a conclusion that this made the process unfair. It is simply a recommendation for openness and transparency.

Result

As a result of this investigation, I consider that while there were some problems which arose, such as the .gay letter issue, and I have made some criticisms of the process, such as the anonymity, I have not found issues sufficiently serious to recommend any action, other than recommendations about changes for the next round. The EIU supported by ICANN staff appear to have undertaken a very thorough and careful examination of the community applications, with sufficient feedback loops to ensure that the process was correct. There is some force in the actual result that only one CPE result has been sent for re-evaluation, although as a percentage it may be difficult to draw something out of a low number of applicants. But given the processes available to review, it is likely that the process has been conducted fairly.
There is no doubt that the concept of community-based applications was supported by the ICANN community when the new gTLD program was initiated and developed. There was however a need for a better scope of understanding of what those community-based applications were for, and what sort of persons or organisations would benefit from the use of a community based top level domain. Some consideration should have been given to the types of community which could use their own top level domain, whether these were to be charitable, community organisations or perhaps even NGOs or others. At least one submitter made the point that there was an element of public interest which they considered should have been the basis for community applications. But perhaps there should have been more discussion about how communities would actually benefit from having their own top level domain, given that any applicant would have substantial expenses but also the need to develop the technical infrastructure required for a registry, or partner with an organisation with that ability. The concept gave rise to expectations, which have to a considerable extent, not been met. I do not make this comment as a criticism of any of the parties, but it is something which can be considered if there are subsequent rounds of applications.

Chris LaHatte
Ombudsman

© 2016 Internet Corporation for Assigned Names and Numbers