ECHO Controlled Documents

The ECHO Team assigns each public document a unique identifier. This identifier is used to reference the document in emails and on the ECHO website. As each document is updated, the most recent version are made available on the website and, when appropriate, are emailed out to the impacted ECHO partner community and reviewed at an ECHO Technical Committee (ETC) meeting. The public documents managed in this way include the following items. Click the "more..." link to view the current listing of available documents.

Partner Guides & Documentation

Partner Guides & Documentation

Overview

These guides are intended to assist data, client and service providers develop interfaces to the ECHO system.

Documents

ECHO REST Search Guide

ECHO REST Ingest Guide

Data Partner User Guide

  • Download (PDF)
  • Version History
  • Jan 15, 2010, Version 10.6: Added Getting Started Section & Finalized comments
  • Nov 01, 2009, Version 10.5: Refactored document structure to allow for easier information access. Updated document according to recent schema and functionality changes. Updated images used for spatial
  • Jan 16, 2008, Version 10.3: Incorporated ECHO Operations' comments. Removed "Authenticators" as they are no longer used
  • Dec 03, 2007, Version 10.2: Added specific information on how to associate browse metadata to a collection/granule already loaded into the database.
  • Nov 01, 2007, Version 10.1: Initial version
  • Back to Top

Client Partner User Guide

  • Download (PDF)
  • Version History
  • Dec 18, 2008, Version 10.8: Updated for catalog service api changes.
  • Nov 18, 2008, Version 10.7: Added patternList definition and usage. Added links to DTDs. Removed references to global granule searching.
  • Jul 01, 2008, Version 10.6.2: Added additional information on instrument short name condition.
  • Jun 17, 2008, Version 10.6.1: Added bounding box searching to query sections
  • Jun 02, 2008, Version 10.5: Added option definition changes to order option sections
  • May 22, 2008, Version 10.4: Update query sections for Cartesian searching changes
  • Apr 22, 2008, Version 10.3: Added instrumentShortName to query sections
  • Jan 16, 2008, Version 10.2: Incorporate ECHO Operations comments
  • Nov 01, 2007, Version 10.1: Initial version
  • Back to Top

ECHO Forms Specification

  • Download (PDF)
  • Version History
  • Dec 02, 2008, Version 0.7: Added new type for Output control to create a link.
  • Nov 27, 2007, Version 0.6: Updated select control documentation.
  • Mar 29, 2007, Version 0.5: Added information about the limitations related to relevancy and pruning the instance.
  • Mar 23, 2007, Version 0.4: Added the validation reference table in the appendix.
  • Oct 04, 2006, Version 0.3: Updated the forms and the autopopulate extension to the final namespaces for the v9 release.
  • Aug 29, 2006, Version 0.2: Making updates from review and implementation.
  • Aug 18, 2006, Version 0.1: Initial Version
  • Back to Top

TestTrack Pro User Guide

ECHO Approved Country List

Operations Concepts Documents

Operations Concepts Documents

Overview

The ECHO Team develops Operational Concepts to propose changes to ECHO, PUMP, or WIST functionality as a means of reviewing impacts to current functionality and all ECHO Partners. Each Operations Concept document is reviewed during an ECHO Technical Committee Meeting. ECHO Partners are encouraged to attend this meeting or to provide feedback via email. After the review, and after all outstanding issues have been resolved to the approval of the ECHO Partners, the Operations Concept will then be scheduled for implementation in an ECHO development sprint.

The Operational Concept Documents covered on this page include the following items:

Note: Not all Ops Concepts have been externally reviewed at an ETC, and therefore have not been scheduled for implementation.

Documents

DOI Field Addition (ECHO_OpsCon_030)

This paper proposes the addition of a new, non-required metadata element in the ECHO 10 Collection schema (e.g. <DOI >) to represent DOIs for Datasets.

  • Status: In Review
  • Version History
  • July 10, 2013, Draft

CollectionType Field Addition (ECHO_OpsCon_029)

This paper proposes the addition of a new, non-required metadata element in the ECHO 10 Collection schema (e.g. <CollectionType>) to identify non-science-quality products.

  • Status: In Review
  • Version History
  • May 23, 2013, Draft (as NRT)
  • July 15, 2013, Draft

ISO 19115 Strategy (ECHO_OpsCon_028)

This document outlines a strategy for accommodating incoming ISO 19115 variations and providing a unified ISO 19115 format as well as access to the original metadata.

  • Status: In Review
  • Version History
  • February 22, 2013, Draft

Reverb Order Workflow Re-work (ECHO_OpsCon_027)

This document describes a proposed redesign of the Reverb ordering workflow and various improvements enabling easier metadata access for individual collections and granules.

  • Status: Gradual Implementation
  • Version History
  • January 21, 2013, Final

PUMP User Import (ECHO_OpsCon_026)

This document proposes a change to ECHO Provider User Management Program (PUMP) that allows a PUMP provider to find and add to a provider group any user within the EOSDIS User Registration System (URS) without requiring that the user had previously logged into ECHO or Reverb.

  • Status: Implemented
  • Version History
  • January 13, 2013, Final

ECHO User Address Updates (ECHO_OpsCon_025)

This document describes work to be performed as part of EED Task 2 Revision 4. With the Introduction of the EOSDIS User Registration System (URS) disparate EOSDIS applications now have access to a common set of authentication and core user information fields. As part of URS Phase I, ECHO moved to be compatible with URS and now all authentication, user creation, and user updates are proxied to the URS. In general the ECHO User concept and the URS User concept fit well together, however, the location of the URS mandatory Country field in the optional ECHO User Address fields has become problematic. Specific enhancements have been proposed to make several fields in the legacy Address information optional and require exactly one Address for a user.

  • Status: Ready for Review
  • Version History
  • May 30, 2012, Version 1: Initial Draft

Visualization Tools (ECHO_OpsCon_024)

This document describes work to be performed as part of EED Task 2 Revision 4. Specific enhancements have been requested to add capabilities to Reverb so that it will support image browsing for granules with associated browse imagery. Users will be able to select granules through their associated imagery as an additional mechanism for moving items into and out of the Reverb shopping cart. These enhancements are categorized under the heading “Image View.”

Additionally, users will be able to select groups of granules temporally by supplying a set date/time range. Users can adjust the range through UI controls and view granule imagery across datasets. These enhancements are categorized under the heading “Timeline View.”

  • Status: Ready for Review
  • Version History
  • April 2, 2012, Version 060: Internal Review

Browse Reclassification (ECHO_OpsCon_023)

This document describes proposed changes to ECHO data model to reclassify the browse record. It is proposed that the ECHO data model will no longer have a formal browse record. If approved, all changes proposed in this document will be implemented as a part of the ECHO Task 2 Rev 2 Multi-Format Ingest work, currently scheduled for Operational deployment in late-Summer 2011.

  • Status: Ready for Review
  • Version History
  • April 1, 2011, Version 1: ESDIS Review
  • April 27, 2011, Version 1: External Review

Browse MimeType Updates (ECHO_OpsCon_022)

This document describes the proposed changes to the existing FTP Ingest service and Reverb. There will be no visible changes to the ECHO kernel and no changes to WIST. If approved, all changes proposed in this document will be implemented as a part of the ECHO Task 2 Rev 2 Multi-Format Ingest work, currently scheduled for Operational deployment in late-Summer 2011.

  • Status: Ready for Review
  • Version History
  • April 19, 2011, Version 1: Initial External Version

Task 2 Rev 2 - Multi-Format Metadata Support (ECHO_OpsCon_020)

This document describes the work to be performed as a part of EED Task 2 Revision 2. Specific enhancements have been requested to add capabilities to ECHO so that it will support Ingest and metadata presentation in multiple formats, specifically ISO 19115. Updates will also be implemented in Reverb to leverage the new capabilities and to display the ISO 19115 metadata fields.

  • Status: Ready for Review
  • Version History
  • Dec 1, 2010, Version 1: Initial Internal Draft

ECHO Services (ECHO_OpsCon_019)

This document describes proposed changes to the ECHO ExtendedServices service. The proposed changes are in response to the EED Task 3 statement of work. Specifically, the following items have been requested:

  • Would like to have services better supported in ECHO including the ability to easily discover services for granules and collections.
  • Would like to have services support exposed via REST
  • Would like better support for service providers to register services in ECHO via PUMP
  • Would like to have a mechanism for users to browse services via a well defined keyword structure.
  • Status: Ready for Review
  • Version History
  • Sept 15, 2010, Version 2: Updated REST API methods & addressed minor issues with proposed design.
  • Sept 10, 2010, Version 1: Initial Internal Version

ACL Reporting (ECHO_OpsCon_018)

This document describes a proposed Operational solution for providing enhanced status notifications to ECHO Client Partner applications. Implementation of the proposed capability will be performed by the ECHO Operations team and will not be a part of the core ECHO baseline.

  • Status: Ready for Review
  • Version History
  • Sept 9, 2010, Version 2: Initial External Version
  • Aug 23, 2010, Version 1: Initial Internal Version

ACL Reporting (ECHO_OpsCon_017)

This document describes the proposed changes to PUMP and the ECHO API to provide reporting on Catalog Item and Provider Object ACLs.

  • Status: Reviewed
  • Version History
  • Aug 03, 2010, Version 3: Added Appendix A - ACL Report Page Item Limits
  • Apr 19, 2010, Version 2: Initial External Version
  • Apr 13, 2010, Version 1: Initial Internal Version

Bulk Option Assigment Deletion (ECHO_OpsCon_016)

As documented in NCR 1100991, PUMP does not allow users to delete option assignments in bulk. To delete option assignments in PUMP, users have to perform an option assignment search for the target collections and then delete the option assignments one-by-one from the option assignment search results page. Depending on the number of collections and options assignments this delete process can be cumbersome.

  • Status: Implemented
  • Version History
  • Apr 05, 2010, Version 1: Initial Internal Version

Ingest File Error Handling (ECHO_OpsCon_015)

This document describes proposed changes to Ingest reporting to add functionality which will limit the number of reported file errors for each metadata file processed in each job.

  • Status: Implemented
  • Version History
  • Mar 30, 2010, Version 2: Initial External Version
  • Mar 15, 2010, Version 1: Initial Internal Version

ECHO Group Management Ops Concept (ECHO_OpsCon_014)

This document describes proposed changes to the group management API to add functionality needed to support the proposed access control level (ACL) changes presented in a separate operations concept.

  • Status: Implemented
  • Version History
  • Oct 16, 2009, Version 1: Initial External Version

ECHO ACLs and Roles Ops Concept (ECHO_OpsCon_013)

This document describes proposed changes to the access control level (ACL) system to reduce complexity and implement requested functionality which includes expanding the system beyond the current metadata catalog scope. The current ACL system feature-set is a carryover from earlier, versions of ECHO. Now that that API is used regularly by providers and client applications, a number of limitations, complexities, and unnecessary features have been identified. In order to addresses these issues, the ACL API needs to be modified based on past experience and future needs. At the same time, the need for a more encompassing ACL system has been identified. To support more complex use cases proposed by the DAACs, ECHO needs to support ACLs for more domain objects including but not limited to provider orders and provider policies.

  • Status: Implemented
  • Version History
  • Oct 06, 2009, Version 4: Updated to incorporate feedback from ETC
  • Sep 25, 2009, Version 3: ETC Review
  • Sep 10, 2009, Version 2: Updated Internal Version
  • Sep 01, 2009, Version 1: Initial Internal Version

Additional Attribute Order (ECHO_OpsCon_012)

ECHO allows for multiple additional attributes per collection or granule and does not associate any order to those additional attributes. However, providers and users of ECHO metadata may consider additional attributes to be ordered by the order in which they appear the metadata. Ingest does not preserve the original order of additional attributes sent by providers. As a result, additional attributes will be presented in any order when collection or granule metadata is retrieved from ECHO. As part of ECHO 10.12, Ingest was modified to preserve the order of additional attribute values for a single granule additional attribute.

  • Status: Implemented
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

Order Dispatch Fault Tolerance (ECHO_OpsCon_011)

ECHO dispatches provider orders to multiple providers using up to 5 concurrent threads. Each thread delivers one order using a SOAP/HTTP transfer to the provider's endpoint. Once the transfer is complete, the thread becomes available again to dispatch the next queued order. In most cases the preparation and dispatch of an order takes only a few minutes and the order queue remains relatively short or empty. In the event that a provider's endpoint is unreachable or there is an error processing the order for the provider (for example, a SOAP fault), Echo delays the order transmission and will automatically requeue the order at a later date based on the provider's policies.

  • Status: Implemented
  • Version History
  • Sep 12, 2009, Version 2: Updated to incorporate feedback from ETC
  • Aug 31, 2009, Version 1: Initial Version

ECHO Reconciliation (ECHO_OpsCon_010)

All reconciliation activities performed by ECHO data providers are currently facilitated through the GetDatasetInformation method within the ECHO API's DataManagementService. This method generates an XML file containing the following metadata items and transmits that file via ftp to a configured destination:

  • Dataset ID
  • ECHO Collection GUID
  • ECHO Granule GUID
  • Granule UR
  • Granule Visibility flag
  • Granule Insert Date
  • Granule Acquisition Begin & End Date
  • Granule Production Date
  • Granule Provider Last Update Date
  • Granule ECHO Last Update Date
  • Granule Online Access URLs
  • Granule Online Resource URLs & Types
  • Browse URL
  • Browse Size
  • Browse Provider Last Update
  • Browse ECHO Last Update Date
  • Browse Insert Date

Providers can specify one collection at a time for reconciliation and have ECHO filter on the following data characteristics to limit the data which is generated by the method.

  • Temporal Range - Filters based on production, Acquisition, Insert, and LastUpdate fields.
  • AvailableOnline - Filters based on whether granules have an OnlineAccess URL.
  • BrowseAvailable - Filters based on whether the granule has an associated browse record.
  • VisibleOnly - Filters based on whether the granule's visibility flag is set to true.
  • Status: Implemented
  • Version History
  • Oct 01, 2009, Version 5: Final Version

Spatial Keywords Restructuring (ECHO_OpsCon_009)

The ECHO data model was historically based upon existing metadata models at the time of its creation. Since that time there have been modifications to certain externally managed metadata fields which result in an inconsistency when compared with the ECHO data model's representation. Previously, ECHO updated its 'ScienceKeyword' structure to correctly match the updated GCMD structure for its science keywords. Similarly, the GCMD manages a list of 'location' keywords which have a historical association with ECHO's 'SpatialKeyword' structure. The new GCMD structure is as follows: Category > Type > Sub-Region1 > Sub-Region2 > Sub-Region3 > Detailed Location. ECHO currently has a single tiered structure only allowing for a single value in its SpatialKeywords.

  • Status: Draft
  • Version History
  • Sept 7, 2010, Version 7: Removed reference to ECHO 9.0 DTD and BMGT Adapters.
  • Apr 13, 2009, Version 6: Initial Version

Data Quality Summary Management (ECHO_OpsCon_008)

The WIST wwwvalids.conf contains data quality summaries which are assigned to datasets based upon a small number of characteristics (e.g. short name, instrument, DAAC, etc). These summaries are maintained by the ECHO Operations group and any changes must be coordinated with ECHO data partners to ensure the proper information is displayed. Each data quality summary which is associated with a granule that is added to a WIST user's shopping cart will be displayed once during the process of adding granules to the user's cart. Users must choose to accept or reject the data quality summary which is displayed to them. This functionality is a carryover from EDG functionality, as are many of the summaries which are displayed to end users.

This document outlines changes to ECHO and PUMP which will allow ECHO Data Partners to create, update, delete, and assign Data Quality Summaries to their data without requiring ECHO Operations coordination. The proposed functionality will also allow all ECHO clients to display the confnigured Data Quality Summaries, providing consistent user experiences within ECHO clients.

  • Status: Pending External Review
  • Version History
  • July 21, 2010, Version 4: Included ETC Feedback. Updated Proposed Solution to indicate Reverb DQS workflow.
  • July 12, 2010, Version 3: Initial Public Release
  • July 2, 2010, Version 2: Revised Internal Draft
  • Aug 28, 2009, Version 1: Initial Internal Draft

Suspend and resume provider (ECHO_OpsCon_007)

This document describes the proposed implementation of a mechanism to suspend and resume search and order capabilities on a per provider basis. This operations concept was prepared in response to the trouble ticket 11004665 'Allow disabling a provider for search and order by Admin via PUMP'. The trouble ticket states the following, In an effort to reduce scheduled downtime of ECHO it is desirable to disable a single DAAC from the query and ordering for a short period of time. This time can be used to rebuild/modify indexes, update metadata, migrate table structures, etc. without needing to completely shutdown ECHO. We can approximate this functionality by disabling the provider at the DB level for specific situations. However, this scenario should be fully planned and made accessible via the ECHO API and surfaced as Admin functionality in pump.

  • Status: Reviewed (Internal Only)
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

ECHO Online Access & Resource URL Uniqueness (ECHO_OpsCon_006)

The ECHO data model currently includes OnlineAccess and OnlineResource URLs associated with collections and/or granules. The intention of an OnlineAccess URL is to provider end users with a link that will directly take them to an http or ftp site where the collection or granule can be accessed (e.g. downloaded). The intention of an OnlineResource URL is to provider end users with links providing additional information regarding the collection or granule (e.g. instrument info, dataset descriptions). The metadata for OnlineAccess URLs also has URLDescription and MimeType fields, while OnlineResource URLs have Description, Type, and MimeType fields. Both OnlineAccess and OnlineResource URLs are currently uniquely identified by the actual URL itself, not using either the Type or MimeType fields in identifying URLs. Currently within ECHO all OnlineResource URLs can be uniquely identified by a combination of the URL and Type (not MimeType).

  • Status: Reviewed (Internal Only)
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

ECHO Resource Reconcilliation (ECHO_OpsCon_005)

This document describes the proposed implementation of a URL resource reconciliation service that will validate persisted URLs in ECHO. The need to validate URLs persisted in ECHO has been identified in the BMGT Reconciliation White Paper as follows, 'ECHO will also perform periodic checks of datapool URLs that have been ingested into ECHO. These checks will make sure that the URLs are well formed and that they point to an existing endpoint. Any errors found will be reported to the DAAC in a manner that is TBD. While ECHO doesn't modify online access or online resource URLs as part of Ingest, there is a desire to ensure that the URLs stored have not expired or that there was not a data entry problem or mis-configuration of the exported data at the DAAC.

  • Status: Reviewed (Internal Only)
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

Collection Level TwoDCoordinates (ECHO_OpsCon_004)

Granule metadata includes an element named TwoDCoordinate which allows for a granule to be spatially located via a two-dimensional coordinate system. This information is only available in the granule metadata. ECHO currently receives metadata from USGS_EROS for the WRS-1 and WRS-2 projections within these elements. ECHO Ingest also adapts specific PSAs from ECS metadata to create 2D coordinate system information for MISR and MODIS Tile grids. In instances such as Platforms/Instruments/Sensors and Additional Attributes, the collection level metadata contains defining information for these metadata constructs. The granule metadata then references a collection-level defined item and supplies values for the necessary fields. There is no mechanism to do this with TwoDCoordinates.

  • Status: Implemented
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

ECHO Calendar & Provider Information (ECHO_OpsCon_003)

This document describes the proposed replacement of the Uptime Calendar, extensions to the provider profile's additional information, and associated enhancements to ECHO Services and PUMP. The Uptime Calendar is external to ECHO and is mainly used to pass information from providers to WIST users. Providers use the Uptime Calendar web client to publish alert and events that are then persisted by the Uptime Calendar service. WIST uses the Uptime Calendar service to retrieve any alerts or events that have been published by providers for display in WIST. Because the Uptime Calendar is separate from ECHO it requires separate user accounts and account management. Although WIST is currently the only client using the Uptime Calendar service, the information published by providers could be useful to any ECHO client as it commonly pertains to data-access outages or the status of data products. To address these issues with the Uptime Calendar, a new ECHO Calendar Service will be developed to replace the current Uptime Calendar service and PUMP will be augmented to allow for managing ECHO Calendar events using the ECHO Calendar Service. Also, to support an enhancement request from providers to allow configurable provider information such as an overview description and general information links, extensions will be made to the provider profile's additional information field.

  • Status:Ready for Review
  • Version History
  • Sept 22, 2019, Version 3: Externally Reviewed Version
  • Sept 16, 2010, Version 2: Revised Internal Draft
  • Aug 28, 2009, Version 1: Initial Version

Additional Attribute Type Handling (ECHO_OpsCon_002)

Additional attributes are simply key/value pairs associated with collections and granules. A collection defines the name, type, and description of all supported additional attributes. A collection may also optionally define a value for the attribute. A granule additional attribute must reference an attribute defined in the parent collection and specify a specific value. A granule specified value for an additional attribute will override a value specified for the same additional attribute at the collection level. Additional attributes contain different types of information such as numbers (integer and floating point), dates, times, and text strings. Currently all of these values are stored as text strings in the ECHO database. Because of this storage design, clients are now seeing errors when searching for numeric values because the database cannot convert the string and date types to numbers for comparison.

  • Status: Implemented
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

ECHO & WIST Science Keywords Modifications (ECHO_OpsCon_001)

This document describes proposed changes to the ECHO Ingest Schema, AQL DTD , Collection Results DTD, and Taxonomy Service and WIST User Interface in order to align the usage of GCMD managed science keywords. The current Science Keywords data model does not match its analogous structure managed by the GCMD. Data centers are choosing science keywords based upon this externally managed list, however they are unable at this time to submit the full and correct keywords to ECHO in their collection metadata. There is provider interest in adhering to this list, and thus ECHO too should make it possible for data centers to have their data properly represented within ECHO's holdings.

  • Status: Implemented
  • Version History
  • Aug 28, 2009, Version 1: Initial Version

ECHO Technical Committee (ETC) Meeting Minutes

ECHO Technical Committee (ETC) Meeting Minutes

Tip: To download a document, right-click and select "Save Link As..."

Overview

The biweekly ECHO Technical Committee (ETC) meeting is held between ECHO and all data and client partners. During this meeting, the current Operational status is discussed along with any new or ongoing issues that have occured in the previous 2 weeks. In addition, any relevant technical issues brought forward for discussion and decision making. The decisions from this group will directly affect the changes made to ECHO and WIST. The agenda and minutes for this meeting are sent to the echo-all@echo.nasa.gov mailing list. Historical minutes are available on this page for download.

ETC Minutes are available for the following years.

2013 ETC Minutes

2013 ETC Minutes

January 22, 2013

January 22, 2013

Attendance

Name Organization
Katie Baynes ECHO
Doug Newman ECHO
Jon Velapoldi ECHO
Matt Cechini ESDIS
Jessica Garron ASF
Ben McMurray ORNL
Mary Nair GHRC
John Scialdone SEDAC
Jody Rundell LPDAAC
Travis Kroh LPDAAC
Matt Martens LPDAAC
Julie Luebke LPDAAC
Chris Finch JPL
Cathy Fowler NSIDC
Chris Bond NSIDC
Frank Schaffer NSIDC
Lisa Booker NSIDC

ECHO System Status & PM Planning

Unplanned maintenance

  • 01/08/13 9:15pm EST - 01/09/13 8:40am EST - A cascading failure of Elastic Search in operations caused all search and indexing capabilities within operations to fail, elements not dependent on elastic continued to function. Degraded Service Event Report: 01.08.2013

Ingest Issues:

  • 12/14/12 9:00am EST Operations: LPDAAC ingest paused repeatedly. Transient errors causing pause. Fixed catalog-rest and redeployed (rolling) to fix issue
  • 12/17/12 9:00am EST Operations: LPDAAC ingest volume prompted us to reconfigure the pipe distribution of legacy ingest. LPDAAC is up to 10. LAADS and LARC were reduced from 3 to 1.
  • 01/02/13 4:00pm EST Operations: Throttled LPDAAC ingest to original levels (3 pipes). LARC and MODAPS back to 3 pipes.
  • 01/17/13: Operations LPDAAC_ECS ingest issues with sequencing number caused pause of provider. Resolved by LPDAAC with ops assistance.

System Issues

  • LPDAAC Reprocessing Load (see section below)
  • 01/20/13 7:06pm - 10:06pm EST: Operations outage of opsfs1 causing outage of EIAT. Browse did not appear to be affected.
  • Query time outliers have been identified as queries with high paging numbers (in the 1000s). Ncr 11011126 resolves this issue and is scheduled for 10.58

LPDAAC Reprocessing

  • Ingest rate of 12-18 million per week (1.7m – 2.6m per day, peaking at 3m per day)
  • Indexing was lagging behind Ingest (NSIDC notified us on December 27th of an anomaly)
  • January 3rd, spun up 4 additional kernels to increase indexing capability (from 4 to 8 kernels)
  • 01/03/13 Adding additional nodes caused Orders on those nodes to stall. Firewall rules updates resolved this.
  • January 4th, spun up 5 additional kernels to increase indexing capability (from 8 to 13 kernels)
  • Some providers required re-indexing to ensure deletes were properly propagated
  • Excellent exercise for future growth and load planning

Recent Metrics

Recent Reverb Metrics

reverbqueries 01.22.2013 0

Recent Performance Metrics

avg query perf 01.22.2013

Query Performance Breakdown

query perf hist 01.22.2013

In Operations 10.56.8

  • Pushed to operations 12/12
  • 4 week sprint, ended 11/16
  • 8.2 BMGT Reconciliation Report Work
  • Metrics collection updates
  • Preparation for Torquebox deployment
  • Global Lock redesign

In Test 10.57

  • 6 week sprint, ended 12/28
  • URS field synchronization work
  • NCR 11011303 Reverb 10.36.1. Add "map all" feature to granule results (Verify Failed - See Discussion Below)
  • NCR 11013291 Several URS accounts not being created locally in ECHO (Operations Verified - please check your APIs for validity)
  • NCR 11013538 ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset (Verify Failed - Jon)
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary' (Moved to 10.58)
  • NCR 11013371 Errors for ECHO ESIP OpenSearch for collections from OBPG (Operations Verified)
  • NCR 11013387 Service form names in different providers should not collide (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted   (Operations Verified)
  • NCR 11013480 Virtual tags for the ESI service implementation wiped out (Operations Verified)
  • NCR 11013506 Update Java applications to use Java 7 (Operations Verified)
  • NCR 1101228 Mandatory fields on Reverb account creation page do not match URS mandatory fields (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted (Operations Verified)

NCR 11013538:
There was some discussion about how this issue can be reproduced. Any search contraints (or none) can result in this behavior. If you experience this issue, please notify ECHO Operations immediately.

NCR 11013506:
Frank asked if we had any issues related to the Java 7 update. I told him there were no known issues, but it turns out that one issue, NCR 11013572 has been discovered and is being backported to support the 10.57 release.

In Development 10.58

  • 3 week sprint, ending 2/1
  • NCR 11005035 Total number of possible results should be returned (OpenSearch)
  • NCR 11011126 Improve Catalog REST query paging performance
  • NCR 11013284 Reverb Granule Detail Browse jpeg not displayed with I.E. 9
  • NCR 11013386 Rename our 'echo-esip' endpoint 'echo-open-search' Please check any clients you may have for impacts
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary'
  • NCR 11013403 Reverb does not allow users with a "." in the username, but URS does.
  • NCR 11013485 Allow users to order JPEG browse image in Reverb Question out to SDPS about ordering capabilities
  • NCR 11013507 Dataset search results available, but not visible from Reverb.
  • NCR 11013521 OPS:Apparent disconnect between URS and ECHO PUMP Operations Concept has been distributed for internal review. Once this has been ok'd we can send it on to Dawn and company at LP
  • NCR 11013537 Calendar event caching is not working

NCR 11011303 "Map All" Discussion

  • Good for small result sets, but for over 10 results, this is probably not practical
  • Scrolling through the list would prove very problemtic from a performance perspective

After discussing during the meeting we agreed that I would send out a poll offering alternatives and allow voting among several options.

Link to Poll

ISO 19115 Status Update

  • New translation of ISO 19115 is upcoming as part of 10.59
  • Updates to ensure ingest of ISO 19115 via FTP also in 10.59

ECHO Workshop 2013

  • This is in the planning stages, will be in around GSFC
  • What dates would not work for your team
  • Who might be interested from your DAAC?
  • Preliminary Agenda has a developer slant, but if there is a call for a different, non-technical track, please give me feedback!

Upcoming Events

  • 1/23 10.57 update in PT
  • 1/30 10.57 to Operations

Operations Concept: Order Workflow Re-work

  • Emailed to this group yesterday afternoon.
  • Feedback welcome and encouraged

All feedback should be sent to echo@echo.nasa.gov or kathleen.baynes@nasa.gov by next meeting of the ETC (2/5/2013). After all feedback has been collected, I will create an area on the Earth data wiki to address the most controversial issues in a public forum.

After two more weeks of public discussion via the wiki and ETC, we will incorporate feedback, vote on issues and decide on the proper course for implementation and prioritization.

Open Floor

February 5, 2013

February 5, 2013

Attendance

Name Organization
Katie Baynes ECHO
Doug Newman ECHO
Luther Lighty ESDIS
Andy Mitchell ESDIS
Chris Finch JPL
Cathy Fowler NSIDC
Greg Cates LARC
Ed Seiler GES DISC
Merlie Hansen SEDAC
Jody Rundell LPDAAC
Kelly Lemig LPDAAC
Matt Martens LPDAAC
Mary Nair GHRC
Ben McMurray ORNL
Renea Ericson NSIDC
Chris Bond NSIDC
Lisa Booker NSIDC
Frank Schaffer NSIDC
Jessica Garron ASF

ECHO System Status & PM Planning

System Issues

  • Query time outliers have been identified as queries with high paging numbers (in the 1000s). Ncr 11011126 resolves this issue and is scheduled for 10.57.5
  • 01/24 Operations outage from 2:15pm - 3pm EST due to elastic memory issues. Additional elastic nodes added to provide continuity of service in the event of single node failure
  • 01/25 – 01/27 Operations degradation of performance due to java virtual machine problems with one of the additional elastic nodes. Removal of node restored performance. Investigating elastic/jvm issues.
  • 01/30/13 Operations: PM extended by 2 hours due to infiniband switch failure and elastic re-balancing post elastic backup.

Recent Metrics

Recent Reverb Metrics

reverb queries 02 05 2013

Recent Performance Metrics

avg query perf 02 05 2013

Query Performance Breakdown

query perf hist 02 05 2013

In Operations 10.56.9

  • Pushed to operations 12/12
  • 4 week sprint, ended 11/16
  • 8.2 BMGT Reconciliation Report Work
  • Metrics collection updates
  • Preparation for Torquebox deployment
  • Global Lock redesign
  • 10.56.9: Rails Security Patch
  • 10.56.9: High Page Request fix
  • 10.56.9: Temporary disabling a rarely used and poorly performing route: search_facets

In Test 10.57

  • 6 week sprint, ended 12/28
  • URS field synchronization work
  • NCR 11011303 Reverb 10.36.1. Add "map all" feature to granule results Closing as per poll results. Opening a performance NCR
  • NCR 11013291 Several URS accounts not being created locally in ECHO (Operations Verified - please check your APIs for validity)
  • NCR 11013538 ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset (Verify Failed - Jon)(Moving to 10.59)
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary' (Moved to 10.58)
  • NCR 11013371 Errors for ECHO ESIP OpenSearch for collections from OBPG (Operations Verified)
  • NCR 11013387 Service form names in different providers should not collide (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted   (Operations Verified)
  • NCR 11013480 Virtual tags for the ESI service implementation wiped out (Operations Verified)
  • NCR 11013506 Update Java applications to use Java 7 (Operations Verified, SSL issues resolved)
  • NCR 1101228 Mandatory fields on Reverb account creation page do not match URS mandatory fields (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted (Operations Verified)

In Test 10.58

  • 3 week sprint, ended 2/1
  • NCR 11005035 Total number of possible results should be returned (OpenSearch)
  • NCR 11011126 Improve Catalog REST query paging performance
  • NCR 11013284 Reverb Granule Detail Browse jpeg not displayed with I.E. 9
  • NCR 11013386 Rename our 'echo-esip' endpoint 'echo-open-search' Please check any clients you may have for impacts
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary'
  • NCR 11013403 Reverb does not allow users with a "." in the username, but URS does.
  • NCR 11013484 Allow users to order JPEG browse image in Reverb
  • NCR 11013507 Dataset search results available, but not visible from Reverb.
  • NCR 11013521 OPS:Apparent disconnect between URS and ECHO PUMP Operations Concept has been distributed for internal review. Once this has been ok'd we can send it on to Dawn and company at LP
  • NCR 11013537 Calendar event caching is not working

In Development 10.59

  • 3 week sprint, ending 2/22
  • NCR 11005035: Total number of possible results should be returned
  • NCR 11010755: Move ECHO-ESIP to utilize the T2R2 REST API
  • NCR 11013198: Open Search polygon searching issues
  • NCR 11013457: Searching catalog-rest for collections with 'version' parameter should work
  • NCR 11013466: Granule search results "disappearing" when scrolling through a specific search.
  • NCR 11013514: Reverb is rendering metadata overtop of the "Show Metadata" link
  • NCR 11013520: Request a method be created to inform Echo Ops Support when dataset option assignments are updated in PUMP
  • NCR 11013522: Blink Usability: Remove billing contact information from ordering contact information (assuming OK from ETC)
  • NCR 11013525: Blink Usability: Help Icons Should Be Visible at all Times (assuming OK from ETC)
  • NCR 11013534: Update Elastic index design and reindex data to support SDPS 8.2 and ECHO Performance improvement
  • NCR 11013538: ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset (Moved from 10.57)
  • NCR 11013568: Reverb issuing 'EOSDIS Service Implementation: Request Timeout' error
  • NCR 11013574: 10.57.2 Reverb Partner-Test - Data Access requests failing to submit
  • NCR 11013575: Reverb 10.56.8. Error when clicking on "order" button from cart for certain orders
  • NCR 11013576: Investigate ISO ingest via FTP for ASF SMAP products

Doug Wants to Know: Do you use the ECHO Open Search html interface?

GES DISC is using this interface for testing purposes. Matt Martens will check with LPDAAC development.

ISO 19115 Status Update

  • New translation of ISO 19115 is upcoming as part of 10.59
  • Updates to investigate ingest of ISO 19115 via FTP also in 10.59

ECHO Workshop 2013

  • Tentative Dates: April 30 and May 1, 2013
  • Draft Agenda Below
				Overview/History/Agenda
				Year in Review
				     New providers
				     New Ingest methods
				     Recorded holdings/recorded ingests
				     Uptime and Response improvements
				ECHO and ISO Planning
				     SMAP
				     SAGE III

				Development Workshop
				    REST Overview
				    RESTful Ingest Hackathon
				        Sample ECHO Developer Data Providers?
				    RESTful Discovery Hackathon
				        Sample ECHO Developer Clients?
				        Bulk Data Retrieval
				    Getting the Most Out of ECHO Metrics APIs
				    Using ECHO's OpenSearch API

				Future Planning
				    Metadata Harmony
				    BMGT Rewrite Impacts
				    Reverb Changes
				     Workflow and Blink Usability Changes
				     ISO Export

				Open Discussion
				

Upcoming Events

  • 2/6 10.57.5 to Operations, 10.58 to Testbead, IPv6 prep for all development networks (TB and PT)

Operations Concept: Order Workflow Re-work

  • Compiling results now

Open Floor

February 19, 2013

February 19, 2013

Attendance

Name Organization
Katie Baynes ECHO
Doug Newman ECHO
Peter Plofchan ECHO
Luther Lighty ESDIS
Chris Finch JPL
Ben McMurray ORNL
Ed Seiler GES DISC
Mary Nair GHRC
Matt Martens LPDAAC
Travis Kroh LPDAAC
Cathy Fowler NSIDC
Frank Schaffer NSIDC
Lisa Booker NSIDC

ECHO System Status & PM Planning

System Issues

  • 02/07/13 Partner Test: Ingest restart required due to stuck NSIDC job
  • 02/12 - 13/13 All: vendor security patches applied
  • 02/15 Partner Test: 10.58.4 pushed containing ‘show stopper’ ncrs for 10.58 in Operations
  • 02/15 Partner Test: elastic node ptelastic2 removed from environment due to memory consumption issues

Recent Metrics

Recent Reverb Metrics

reverb queries 02192013

Recent Performance Metrics

avr query perf 02192013

Katie ETC Question:   Has Reverb performance been noticably different? Lisa Booker stated that there had been no noticable difference.

Query Performance Breakdown

quer hist 02192013

In Operations 10.57.8

  • 6 week sprint, ended 12/28
  • URS field synchronization work
  • NCR 11013291 Several URS accounts not being created locally in ECHO (Operations Verified - please check your APIs for validity)
  • NCR 11013371 Errors for ECHO ESIP OpenSearch for collections from OBPG (Operations Verified)
  • NCR 11013387 Service form names in different providers should not collide (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted   (Operations Verified)
  • NCR 11013480 Virtual tags for the ESI service implementation wiped out (Operations Verified)
  • NCR 11013506 Update Java applications to use Java 7 (Operations Verified, SSL issues resolved)
  • NCR 11012288 Mandatory fields on Reverb account creation page do not match URS mandatory fields (Operations Verified)
  • NCR 11013385 Reverb Partner-test 10.54.1, Data Access Services - clicking 'Reload Current Page' causes request to be resubmitted (Operations Verified)

In Test 10.58

  • 3 week sprint, ended 2/1
  • NCR 11011126 Improve Catalog REST query paging performance (Operations Verified)
  • NCR 11013386 Rename our 'echo-esip' endpoint 'echo-open-search' (Operations Verified)
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary' (Not easily reproduced, additional logging has been added to investigate future occurances)
  • NCR 11013403 Reverb does not allow users with a "." in the username, but URS does. (Awaiting Operations Verification)
  • NCR 11013507 Dataset search results available, but not visible from Reverb. (Awaiting Operations Verification)
  • NCR 11013521 OPS:Apparent disconnect between URS and ECHO PUMP Ops Con approved, Moving to Sprint 10.60 (hopefully)
  • NCR 11013537 Calendar event caching is not working (Awaiting Customer Verification)

In Development 10.59

  • 3 week sprint, ending 2/22
  • NCR 11005035: Total number of possible results should be returned
  • NCR 11010755: Move ECHO-ESIP to utilize the T2R2 REST API
  • NCR 11011745: REVERB:Limited "Search for Granules by ID (added unplanned)
  • NCR 11013198: Open Search polygon searching issues
  • NCR 11013457: Searching catalog-rest for collections with 'version' parameter should work
  • NCR 11013466: Granule search results "disappearing" when scrolling through a specific search. (pushed out)
  • NCR 11013514: Reverb is rendering metadata overtop of the "Show Metadata" link
  • NCR 11013520: Request a method be created to inform Echo Ops Support when dataset option assignments are updated in PUMP (pushed out)
  • NCR 11013522: Blink Usability: Remove billing contact information from ordering contact information
  • NCR 11013534: Update Elastic index design and reindex data to support SDPS 8.2 and ECHO Performance improvement
  • NCR 11013538: ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset
  • NCR 11013568: Reverb issuing 'EOSDIS Service Implementation: Request Timeout' error
  • NCR 11013574: 10.57.2 Reverb Partner-Test - Data Access requests failing to submit
  • NCR 11013576: Investigate ISO ingest via FTP for ASF SMAP products

ECHO Workshop 2013

  • Tentative Dates: April 30 and May 1, 2013
I know this meeting is usually attended by user services groups, I want to make sure that the agenda caters to you. What do you want to hear about at the Workshop?

Upcoming Events

  • 2/20 10.58 to Operations

Operations Concept: Order Workflow Re-work

Open Floor

March 5, 2013

March 5, 2013

Attendance

Name Organization
Katie Baynes ECHO

ECHO System Status & PM Planning

System Issues

  • 02/19 Java security patches applied to Partner Test and Test Bed (Operations patched during regular PM the following day)
  • 02/26/13 Lost ECHO Operations from 3:00pm - 3:10pm EST approximately
  • Work on Reconciliation continues, two NCRs in 10.60 to address issues

Recent Metrics

Recent Reverb Metrics

Screen Shot 2013 03 05 at 2.56.09 PM

Recent Performance Metrics

Screen Shot 2013 03 05 at 2.56.19 PM

Query Performance Breakdown

Screen Shot 2013 03 05 at 2.56.28 PM

In Operations 10.58.4

  • 3 week sprint, ended 2/1
  • NCR 11011126 Improve Catalog REST query paging performance (Operations Verified)
  • NCR 11013386 Rename our 'echo-esip' endpoint 'echo-open-search' (Operations Verified)
  • NCR 11013399 8.1+ OPS. BMGT ingest error for element 'boundary' (Not easily reproduced, additional logging has been added to investigate future occurances)
  • NCR 11013403 Reverb does not allow users with a "." in the username, but URS does. (Operations Verified)
  • NCR 11013507 Dataset search results available, but not visible from Reverb. (Awaiting Customer Verification)
  • NCR 11013537 Calendar event caching is not working (Awaiting Customer Verification)

In Test 10.59

  • 3 week sprint, ending 2/22
  • In testbed now, moving to partner test tomorrow 3/6
  • OpenSearch Re-write
  • NCR 11005035: Total number of possible results should be returned
  • NCR 11010755: Move ECHO-ESIP to utilize the T2R2 REST API
  • NCR 11011745: REVERB:Limited "Search for Granules by ID (added unplanned)
  • NCR 11013198: Open Search polygon searching issues
  • NCR 11013457: Searching catalog-rest for collections with 'version' parameter should work
  • NCR 11013466: Granule search results "disappearing" when scrolling through a specific search. (pushed out)
  • NCR 11013514: Reverb is rendering metadata overtop of the "Show Metadata" link
  • NCR 11013520: Request a method be created to inform Echo Ops Support when dataset option assignments are updated in PUMP (pushed out)
  • NCR 11013522: Blink Usability: Remove billing contact information from ordering contact information
  • NCR 11013534: Update Elastic index design and reindex data to support SDPS 8.2 and ECHO Performance improvement
  • NCR 11013538: ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset (moved to 10.60)
  • NCR 11013568: Reverb issuing 'EOSDIS Service Implementation: Request Timeout' error
  • NCR 11013574: 10.57.2 Reverb Partner-Test - Data Access requests failing to submit
  • NCR 11013576: Investigate ISO ingest via FTP for ASF SMAP products (Need to coordinate with ASF SMAP team)

In Development 10.60

  • 4 week sprint, ending 3/22
  • NCR 11013484 Allow users to access JPEG browse image in Reverb (8.2 showstopper)
  • NCR 11013520 Request a method be created to inform Echo Ops Support when dataset option assignments are updated in PUMP
  • NCR 11013521 OPS:Apparent disconnect between URS and ECHO PUMP
  • NCR 11013538: ECHO 10.56.6. Reverb is reporting incorrect number of granules for a dataset
  • NCR 11013590 Upgrade rest-rails project to Rails 3.2
  • NCR 11013605 Test orders and service invocations against IPv6 endpoints
  • NCR 1103620 Document possible ingest error cases on Catalog REST API (BMGT Re-write)
  • OpenSearch Re-write Cleanup as needed

ECHO Workshop 2013

OpenSearch Re-write in Beta on Testbed

  • Link
  • Pointing to Operations Data for search

Operations Concept: Order Workflow Re-work

  • Discussion to be closed this evening. Decisions and NCRs will be made and pla
  • Looking to schedule this as part of 10.63

Upcoming Events

  • 3/6 10.59 to Partner Test

Open Floor

2012 ETC Minutes

2012 ETC Minutes

March 20, 2012

March 20, 2012

Attendance List

Name Organization
Katie Baynes ECHO
Ayanna Dennis ECHO
Matt Cechini ECHO
Luther Lighty ECHO
Jessica Garron ASF
Jon Pals SDPS
Chris Finch PODAAC/JPL
Lisa Booker NSIDC
Doug Fowler NSIDC
Mary Nair GHRC
Ben McMurry ORNL
Lindsey Parker ASDC/LARC
Merlie Hansen SEDAC
Rosy Cordova GCMD

ECHO System Status & PM Planning (Ayanna Dennis)

Operations, partner test and testbed Environments have been stable. There have been some ingest pauses for specific providers. In Ops, NSIDC and GSFC experienced ingest pauses on 3/18 and 3/19 respectively. USGS EROS experienced a pause in ingest in the partner test environment

There is planned maintenance scheduled for tomorrow (3/21). During this PM, various OS patching will be done and the Ops environment will be upgraded from 10.41.8 to 10.44.9.

Katie's Introduction

I am a former developer on both Reverb and URS systems with 10 years software experience and have been with Raytheon for over five years. Look forward to working with everyone!
My (virtual) door is always open. Email Me

WIST Retirement (03/28/2012)

Show-stoppers

Remaining to Verified in Operations (10.41.7)

  • 11012229 - HDF Browse Issue, fix involved using a new library to execute the conversion, increased logging and notification triggers to help identify and diagnose further issues

Available for Verification n Partner Test and reverb-beta

  • 11012176 - DAR ID Search not working
  • 11012181 - Place name searching
  • 11012111 - Not able to select ranges with latest version of firefox and chrome
  • 11012112/2132 - Additional attribute string searches not working
  • 11012201 - New spatial entry dialogs not showing in Safari
  • 11011418 - Update Reverb look and feel to use Earthdata.nasa.gov styling

WIST Retirement

  • Redirection from WIST to Reverb, with an option to return to WIST, is in place.
  • Need to make sure that individual DAAC sites no longer refer to WIST
  • GLAS Orbit reindexing should be done (related to NCR 11012277)
  • ECHO website will be redirected to the new Earthdata Pages (earthdata.nasa.gov/echo)

WIST retirement date is scheduled for 3/28/12.

  • The final Reverb release before WIST transition (10.44.9) will be available for testing in Operations 3/21/12.

ECHO & EOSDIS User Registration System Integration (Date TBD)

URS Technical Committee has formed

  • Meeting alternate Tuesdays 3:00 PM EDT when ETC does not meet

Mailing Lists

Testbed

  • Remains transitioned and connected to URS.

Partner Test

  • Transition occurred on 3/14
  • No major issues reported
  • Individual problems resolved on a case-by-case basis

Operations

  • Date is TBD
  • Initial PR email to all ECHO stakeholders prior to sending them to the ECHO user base
  • Targeting to send out PR email to users 2 weeks prior to the transition
  • Account conflict emails will be sent 1 week prior to transition

URS Transition Working Group

  • Led by Matt Cechini
  • Meeting times TBD

New Topic: REST-only Ingest Option

  • Initially, subscriptions and Notification and GetDataSetInformation (related to reconciliation) will not be supported via REST-only ingest. To be implemented later.
  • Work has been identified and is moving forward to support this for any interested providers, please contact echo@echo.nasa.gov if interested in this new ingest option

Two Week Sprint 10.45

Newly arising Reverb WIST items

NCR 11012327 GLAS subsetting issue for items in the order

REST-only items

URS items

Upcoming Events

  • Operations 10.44.9 Update - 3/21/12
  • Operations -> URS Migration - TBD
  • WIST Retirement - 3/28/12

Open Floor

Doug Fowler brought up a concern related to NCR 11012277 and the newly reindexed data. He believes that this fix may have revealed other problems related to bounding box searches. This could become a high priority fix for NSIDC. Follow-up: NCR 11012015 seems to be related to this issue.

April 3, 2012

April 3, 2012
  • ECHO System Status & PM Planning (Ayanna Dennis)
  • Partner Test Issues WIST Retirement (occurred 03/28/2012)
  • ECHO & EOSDIS User Registration System Integration (04/25/2012)
  • Upcoming Release 10.45
  • Currently In Work 10.46
  • New ETC Topic: Beta Visualization Features
  • Upcoming Events
  • New ETC Topic: ECHO Informational Topics

April 17, 2012

April 17, 2012
  • ECHO System Status & PM Planning (Ayanna Dennis)
  • Upcoming Planned Outages/Status
  • ECHO & EOSDIS User Registration System Integration (04/25/2012)
  • Upcoming Release 10.45
  • In Work 10.46
  • Reverb Beta Visualization Features
  • Upcoming Events
  • Open Floor
  • ECHO Environments Overview

May 1, 2012

May 1, 2012
  • ECHO System Status & PM Planning (Ayanna Dennis)
  • ECHO & EOSDIS User Registration System Integration
  • In Work/Upcoming Release 10.47 (being pushed to testbed 5/9)
  • New ETC Discussion: Total Query Time
  • Reverb Beta Visualization Features
  • Upcoming Events
  • Open Floor
  • Ted Habermann Talk - MENDS, ISO 19115 

May 15, 2012

May 15, 2012
  • Attendance
  • ECHO System Status and PM Planning (Doug Newman)
  • Partner Test News
  • Upcoming Operations Release 10.47
  • In Work 10.48
  • Reverb Visualization
  • Upcoming Events
  • Open Floor

2011 ETC Minutes

2011 ETC Minutes

January 11, 2011 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Reverb 10.32 Planning & Status
  • 10.33 Planning

March 8, 2011 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 10.35 ECHO/WIST NCR Overview
  • Reverb Status & Technical Discussions

March 22, 2011 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 10.34.2 Fast Track to Ops
  • Reverb Status & Technical Discussions
  • ECHO vs Client Order Creation Emails

April 5, 2011 ETC Minutes

2010 ETC Minutes

2010 ETC Minutes

December 07, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Task 2 Rev 2 - Multi-Format Metadata Support - Ops Concept Review

November 23, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • NCR 11010392 - Reverb field normalization
  • Reverb Status Update
  • Edit Option Definitions - Ops Concept Review

November 09, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 10.30 & 10.31 Review
  • Reverb Status Update

October 12, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Reverb Status Update
  • 10.29 & 10.30 Review

September 14, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Reverb Status Update
  • 10.28 Release Review
  • ECHO Atom Feed Ops Concept
  • TestTrack Pro Workflow Updates
  • Discovering granules w/o spatial information

August 17, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 8/10/2010 Degraded Service Event
  • Reverb Feedback (Open Forum)
  • ACL Management Ops Concept Review

July 20, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Reverb Feedback & Development Plan Review
  • ECHO 10.28 Sprint 1 Candidate NCRs
  • ECHO Status Atom Feed
  • Data Quality Summary Ops Concept Review

July 06, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 6/22/2010 Degraded Service Event
  • NCR 11005287 - Persistent Collections in ACLs
  • ECHO 10.25, 26, & 27 Release Planning
  • ECHO Status Atom Feed
  • Reverb Release Planning

June 08, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 5/26/2010 Degraded Service Event
  • ACL & Group Management - Follow Up
  • ECHO 10.24 & 10.25 Release Planning
  • Test Track Pro Visibility

May 25, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ACL & Group Management - Final Planning
  • ECHO Ingest Accounting Tool URLs
  • Controlled Documents Website Review
  • ECHO-ESIP OpenSearch API Review
  • ECHO v9 API Decommissioning

Mar 30, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Degraded Service Event (3/16/2010)
  • Ingest File Error Handling Ops Concept
  • PUMP & ACL Reports
  • Metadata Element

Mar 16, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ACL & Group Management (Open discussion)
  • ECHO 10.22 & 10.23 Release Planning
  • ECHO v10 Interface

Mar 02, 2010 ETC Minutes

  • Main Topics
  • ACL & Group Management (Open discussion)
  • ECHO 10.22 & 10.23 NCRs
  • Returned Granule Ordering

Feb 16, 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Group Management & ACLs How To Guide Review

Feb 02 2010 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ACLs - Visibility & Orderability
  • NCR 11004831

2009 ETC Minutes

2009 ETC Minutes

Nov 24, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO High Availability (Dan Pilone)
  • ECHO Next Generation Client (Codename "Reverb")

Oct 20, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Candidate Collections for Internal Testing
  • ECS BMGT Reconciliation Patch
  • Group Management ACL

Sep 01, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Degraded Service Events
  • Order Dispatching Fault Tolerance
  • Additional Attribute Order Ops Concept Review
  • ECHO/WIST "Not Orderable" Error Message
  • NCR 11004697 -WIST Product Email
  • Communications Policies

Aug 18, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO Reconciliation Ops Concept Review
  • OrbitCalculatedSpatialDomain Mandatory Fields

Jul 21, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.14 Plans
  • NCR 11004648 -PUMP -ACL reference to Granule ID
  • ECHO Email Transition

Jul 07, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.13 & 10.14 Plans
  • Additional Attribute Order Preservation
  • Internal vs. External ECHO Status List

Jun 23, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.12
  • ECHO & WIST 10.13
  • WIST Bulk Ordering
  • Ingest Event Notifications

Jun 09, 2009 ETC Minutes

May 26, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO Browse Status
  • Results Sorting Status
  • SOAPScope Usage
  • WIST Order Email & the Provider "Additional Information"

May 12, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Browse Recovery Plans
  • Results Sorting Status
  • ECHO 10.11 Plans
  • Data Partner Ingest Notifications
  • TwoDCoordinateSystem Ops Concept Review
  • ECHO Database Partitioning

Apr 28, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Browse Recovery Plans
  • Results Sorting Status
  • Order Form Standards

Apr 14, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • Browse Recovery Plans
  • ECHO & WIST 10.10 Plans
  • Results Sorting Status
  • ACL Creation Review
  • NASA Earth Science Imagery (NESI)

Mar 03, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.9
  • ECHO & WIST 10.10
  • ECHO & WIST 10.11
  • Additional Attribute Type Handling Ticket Review (Revisited)
  • NCR 11004171 -PUMP ACL UI Simplification

Feb 17, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.8
  • NCR 11004393 -Password Reset User Experience
  • Additional Attribute Type Handling Ticket Review

Feb 03, 2009 ETC Minutes

Jan 27, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 2009 ECHO Workshop
  • MISR & MODIS Tile Searching

Jan 13, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • 2009 ECHO Workshop Agenda Items
  • ECHO-Goddard Firewall Configuration
  • EIAT Access
  • GetDataSetInfo Reconciliation

Jan 06, 2009 ETC Minutes

  • Download (PDF)
  • Main Topics
  • ECHO & WIST 10.9 Plans
  • PUMP Row Highlighting

Degraded Service Events

Degraded Service Events

Tip: To download a document, right-click and select "Save Link As..."

Overview

A Degraded Service Event is generated when a significant outage occurs in the ECHO Operational, Partner Test or Test Bed system. The purpose of this document is to provide a high level and detailed summary regarding the factors that contributed to the outage. Also included in this document is a detailed timeline of events, NCRs or TTs relating the issues, and future mitigation steps to prevent further outages of the same type.

Degraded Service Events are available for the following years.

2013

August 16th, 2013 Degraded Service Event (Operations)

While resolving a legacy ingest concurrency issue, one of the catalog-rest nodes was unable to obtain a lock on the infinispan cache, effectively making that node unresponsive.

August 13th, 2013 Degraded Service Event (Operations)

A crash of one of the elastic nodes causes search performance to degrade substantially, until that node was removed from the elastic cluster.

July 31st, 2013 Degraded Service Event (Operations)

Inadvertent deletion of an elastic search alias (our means of providing search functionality within ECHO) in operations caused all search and indexing capabilities within operations to fail, elements not dependent on elastic continued to function.

June 26th, 2013 Degraded Service Event (Partner Test)

Partner test ingest and search services experienced an outage on Wednesday, 06/26/13 at 1:43PM ET. Elastic search suffered a failure due to available memory shortages on the ptelastic1 machine. After extensive triage the memory allocation for elastic search was reduced. This allowed elastic search to operate with slightly degraded performance without exhausting the memory available on ptelastic1. Partner Test was fully operational at 4:04PM ET.

April 5th, 2013 Degraded Service Event

  • Executive Summary

All of the operations services provided by ECHO systems experienced an outage Friday, 04/-5/13 at 17:20. The outage was noted by an external monitoring system. All Operations servers were up and online for the entire period.  The systems automatically recovered four (4) minutes after the initial outage alert.

March 8th, 2013 Degraded Service Event

Intermittent outages of Operations were observed from 7:00pm EST onwards. This was due to an issue with NetOps’ primary DNS. The outages were not sustained enough for the switchover to the secondary DNS to kick in. This was performed manually at around 8:00pm and full connectivity was restored at 9:07pm.

March 6th, 2013 Degraded Service Event

Concurrent hits on the catalog-rest query metrics endpoint caused a performance hit severe enough to lock up all our catalog-rest nodes in Operations preventing access to new queries.

February 20th, 2013 Degraded Service Event

The Operational change of enforcing the redirecting of HTTP to HTTPS for catalog-rest endpoints caused legacy ingests to fail to propagate to catalog-rest. This failure did not result in any provider suspensions and remained undetected until 03/05 due to a misdiagnosis of the problem.

January 24th, 2013 Degraded Service Event

The operations elastic search node on dbrac2node3 failed due to lack of available memory on 01/25/13 at 1:56pm EST. The memory issues were due to problems with our facet endpoint implementation. At this point nodes 1 and 2 attempted to replicate the to compensate for the lost node. At 2:46pm node 1 ran into memory issues. We believe that the size of the elastic search index is now large enough that it cannot be housed on only 2 nodes. Consequently a decision was made to restart the entire cluster at 2:55pm and Operations came back on line around 3:00pm. The resultant re-balancing of the elastic search cluster completed at around 3:30pm.

January 8th, 2013 Degraded Service Event

A cascading failure of elastic search (our means of providing search functionality within ECHO) in operations caused all search and indexing capabilities within operations to fail, elements not dependent on elastic continued to function. For example, Legacy ingest, Reverb dataset search and Reverb login were unaffected.

2012

August 5th, 2012 Degraded Service Event

ECHO experienced an unexpected outage due to a concurrency issue in Ruby on Rails.

July 17th, 2012 Degraded Service Event

ECHO experienced an unexpected outage due to a concurrency issue in Ruby on Rails.

June 8th, 2012 Degraded Service Event

ECHO experienced a problem with its search cluster in the Operational Environment. Our search platform, ElasticSearch, had become unresponsive on all three nodes.  After some initial time spent troubleshooting, the decision was made to restart ElasticSearch.  Attempts to restart ElasticSearch failed due to a Linux system problem forcing the java process to fail with a segmentation fault and resulting in intranode communication problems.  We rebooted the Linux hosts serving ElasticSearch and our Oracle Database, thus allowing ElasticSearch to start. 

2011

2010

August 10, 2010 Degraded Service Event

During a planned GSFC-managed maintenance activity, a redundant Power Distribution Unit (PDU) was taken offline. After taking the redundant PDU offline, the remaining PDU was unable to handle the load and the resulting power oversubscription caused critical GSFC managed network components to lose power. These network components were upstream from the ECHO network, causing a loss of external connectivity to all ECHO resources. Once power was restored, the necessary network components resumed operation and ECHO became available to external users.

June 22, 2010 Degraded Service Event

Access to all externally visible ECHO systems was lost resulting from a configuration change made to the network load balancers. A new host entry was added to the ECHO load balancers to support ECHO development activities. An initially unperceived error existed in one of the critical configuration files on one of the redundant load balancers. This caused a cascading set of issues which forced a loss of external accessibility. The ECHO system administrators initially resolved the issue to restore system access. However, the initial resolution ultimately resulted in an increasingly degraded level of availability. When this was noted by the ECHO team, action was taken to apply a permanent fix to restore access to the ECHO systems.

May 26, 2010 Degraded Service Event

The ECHO 10.23 release promoted the new ACL and Group Management capabilities into an active state whereby access to provider objects and catalog items are affected by the new provider configured ACLs. After the deployment of 10.23 into the Operational system it was immediately noticed that query performance was degraded for some providers, specifically LPDAAC_ECS, NSIDC_ECS, and LARC. During this initial period of poor performance, the average query time for a granule query jumped from 12s to 212s.

The ECHO development and database team correlated the poor query performance to long running database searches. After investigating the root cause, the ECHO development, database, and operations team worked to introduce 3 minor patches to the Operational system returning the average query time to its original performance level. During this work, there was no Operational downtime required due to the recent high availability efforts.

Mar 16, 2010 Degraded Service Event

All instances of ECHO and WIST experienced an outage on the evening of Tuesday 3/16/10 starting at 4:45pm EST. Analysis has identified that a significant network event occurred which caused a loss of internal and external network connectivity. ECHO Operations was notified of the system issues by its automated monitoring tool and immediately contacted ECHO System Administrators. ECHO System Administrators were able to restore connectivity to all systems by 6:00pm EST. At that point, the Partner Test and Testbed systems were fully available. The Operational system remained unavailable due to an issue with connectivity between the Oracle RAC nodes. This problem was identified at 7:00pm EST and the ECHO System Administration and Database teams worked to identify the root cause of the problem. By 9:00pm EST the decision was made to restart each RAC node and restore them to proper working order. This took approximately 1 hour, completing at 10:00pm EST. Subsequent to this activity, the Operational kernels were restarted, restoring ECHO and WIST search and order capabilities. Operational Ingest was restarted at 10:45pm EST. No data loss or corruption occurred as a result of this outage.

2009

Oct 24, 2009 Degraded Service Event

All instances of WIST experienced a 90 minute outage on the morning of Saturday 10/24/09 from 9:00am EST to 10:30 EST. There were no other system impacts. The outage was promptly detected and reported to the ECHO Operations, DBA, and SA teams. The cause of the outage was self-correcting and has not been determined after investigation. There does not appear to have been any unexpected activities on the WIST host during that time, but it is noted that routine processes took longer to process than normal. ECHO has decided that these long running processes were a symptom of a system issue instead of the actual cause.

Sep 17, 2009 Degraded Service Event

On Thursday September 17th, at 9:25am (EDT), ECHO experienced a power outage that affected operational ECHO services. This outage was the result of two separate combined events.

  • 1. GSFC Facilities Management Department (FMD) had scheduled to perform a repair to a GSFC power system component. The ECHO team did not get notification of this planned outage and was not prepared for it.
  • 2. In conjunction to this power outage, a component of the ECHO's redundant power system had failure on 9/11/09. Although a service appointment from the vendor had been scheduled for Wednesday 9/16, other administrative issues prevented the technician from accessing the site. The technician returned on Thursday 9/17 and was repairing the failed component at the time of the outage. Had this repair been performed on Wednesday, there would have been no ECHO outage. The effect of the combined factors resulted in a power loss to a critical storage component for the Operational database.

Aug 26, 2009 Degraded Service Event

On August 26, at approximately 2 pm, ECHO experienced a slowness in response while submitting orders to the LPDAAC order fulfillment service. After the configured internal timeout, ECHO was correctly moving orders to the retry queue and processing subsequent orders. Due to the large number of orders which were being serviced for LPDAAC, ECHO was in a state where non-LPDAAC orders were not being serviced in a timely manner. A similar situation occurred around June 21st - 24th with LPDAAC order fulfillment connectivity. An NCR was written at that time, procedures put in place to detect the situation, and NCR 11004606 was written to track an ECHO enhancement to increase order dispatching fault tolerance.

 

Page Last Updated: May 2, 2019 at 10:33 AM EDT