Others - please chime in with thoughts.

-mdg


On Oct 19, 2018, at 1:24 PM, Tod Olson <tod@uchicago.edu> wrote:

To lay out some of the context explicitly:

The Reporting SIG raised the need for implementing libraries to be write custom reports that gather data from across the disparate data domains or storage modules. This is an operational need for most if not all libraries to go live. These reporting needs range from communicating out to external entities like government and national organizations to local operational needs, including the need to do research on the state of one's data. Having the data split into different silos is a large obstacle, and having a solution that will work in a hosted environment is also large obstacle.

The Reporting SIG has presented these needs to the Product Council and asked for them to be met by what has been called Data Warehouse Reference Implementation. The PC seems to be in agreement with the Reporting SIG that these are critical needs to be met at the project level. Put another way, how many libraries will adopt a system that cannot demonstrate solid reporting? That is a valid project-level concern.

Some of the pushback within the TC is about what is and is not properly part of FOLIO, arguing that a data warehouse is outside of FOLIO proper and would be a considerable drain on resources. To me this seems like misalignment around scope between the TC on one hand and the Reporting SIG and PC on the other.

There is an additional question about the technology needed to support reporting. The Reporting SIG has tried to state the outcomes, but not proscribe technologies so as not to preclude some possible solutions. That said, many in the Reporting SIG are accustomed to direct SQL access to the data and have a bias in that direction for the usual reasons: comfortable with the capabilities, the there is already a sunk cost with training that could be leveraged. The TC (or some delegated party) should feel free to propose a non-RDBMS solution, but acknowledge the need to sell a different type of solution.

It is also the case that the Reporting SIG, in anticipation of a data warehouse reference, is currently dividing up the reports in the master spreadsheet and looking for people within the institutions (often the SIG member) who can take responsibility for writing those reports. These are all going into JIRA, assigned to SIG members. I don't know if that's useful context for the TC. It does speak to the Reporting SIG expecting to do a lot of the work themselves. And the intent is to be able to share reports, once written, among the libraries.

I rather suspect that there will need to be some direct dialogue between the Reporting SIG and the TC.

-Tod

On Oct 17, 2018, at 12:55 PM, Mike Gorrell <mdg@indexdata.com> wrote:

Here are my notes/thoughts after our meeting today. Please comment as necessary.

  1. The Tech Council (and FOLIO at large) doesn’t have a technical blueprint on meeting reporting needs for FOLIO.
  2. We have 2 potentially related efforts, A reference implementation of the Library Data Platform (UXPROD-1128  and what turned into the AES (UXPROD-330) - both of which earlier were thought to represent the ‘official’ approach to reporting. These two aren’t aligned.
  3. FOLIO needs an official approach to reporting, including a technical blueprint and a clear delineation between what the FOLIO platform provides and whatever an implementation/tenant needs to provide themselves.

Specific thoughts related to the LDP document and the proof of concept that is being built:
  1. It is concerning that the data warehouse approach may be too closely tied to schemas inside FOLIO in that changes to FOLIO would force upkeep/maintenance of the ETL
  2. The original thought of FOLIO providing a Data Lake (from which a tenant would create potentially its own data warehouse or other reporting capabilities) may alleviate some of the maintenance burden associated with changing FOLIO Modules/schemas.
  3. The proof of concept is scheduled to finish in a week or two. We agreed to provide the core team support needed to finish this.
  4. It is assumed that the LDP effort aligns with what the Reporting SIG desires. In as much as the LDP isn’t totally aligned with the TC, that means that the TC is not aligned with the Reporting SIG, which is a problem.

The Tech Council has architectural responsibility for FOLIO and its approach to reporting. As such, the following actions are recommended:
  1. Creation of a reporting architectural blueprint. This would include where data is streamed vs batch exported, etc, where data lakes may be involved and generally how tenants would report against their FOLIO content and activity
  2. We need to resolve open questions related to the LDP approach/prototype, starting with documented and prioritized concerns. These should be discussed with Nassib. It’s possible those discussions impact the Blueprint. 
  3. We need to align with the Reporting SIG. Once we have performed the actions above we should meet to present the blueprint and discuss their needs and concerns.

Looking forward to your comments. I’d like to have some email dialog (or would you prefer a Google Doc?) so that we have as many issues ironed out prior to next Wednesday’s meeting. If anyone wants to volunteer to draft any part of the reporting blueprint or documented concerns with LDP, please do volunteer.

Thanks!

-mdg


To unsubscribe from this list please go to http://archives.simplelists.com


To unsubscribe from this list please go to http://www.simplelists.com/confirm.php?u=SeK0ArgpLZB2ijVHc1q8eivZ4CXy8J2w