Ulf,
I recommend reading the document from Jason and us here at TAMU for our rancher/kubernetes deployment.  While specific to rancher, it should work on any kubernetes deployment.

https://github.com/folio-org/folio-install/tree/kube-rancher/alternative-install/kubernetes-rancher/TAMU

See this section for okapi clustering:

https://github.com/folio-org/folio-install/tree/kube-rancher/alternative-install/kubernetes-rancher/TAMU#okapi-notes

We had the okapi developers add a plugin to okapi for hazelcast to use kubernetes/etcd for cluster discovery.

Brandon Tharp | Systems Administrator III
Digital Initiatives | Texas A&M University Libraries
Tel. 979.458.5760 | Fax 979.845.6238

On 3/13/19, 11:31 AM, "sysops-sig@ole-lists.openlibraryfoundation.org on behalf of Jason Root" <sysops-sig@ole-lists.openlibraryfoundation.org on behalf of jroot@library.tamu.edu> wrote:

    Ulf, I will be out for the rest of the week, however I would love to talk with you more about this.

    I believe you have essentially ran into the same issues we have with such a system. Issues we hope to have addressed by some further Okapi development and integration with Kubernetes and/or other orchestration toolset.

    -Jason R.

    Sent from my iPhone

    > On Mar 13, 2019, at 11:25 AM, Ulf Seltmann <seltmann@ub.uni-leipzig.de> wrote:
    >
    > Hi all,
    >
    > While creating a workflow to deploy folio in a kubernetes-cluster i
    > stumbled upon some principals of folio which i find rather complicated.
    > Since it might also be a philosophical manner I want to discuss it here
    > instead of the slack-chat (also it is a lot of text).
    >
    > # Okapi-Registry
    > One thing is the "global registry"[1] which holds a description of all
    > backend- and frontend-modules in all versions.
    > As I understand it its purpose is to provide every okapi with a list of
    > all installable modules (frontend and backen) in all versions, so okapi
    > can automatically deploy the modules needed by a tenant into its own
    > docker-daemon.
    > In a single-server environment this might be a comfortable solution to
    > a manual setup where an admin has to create each backend-module in
    > particular.
    > In a cluster-setup on the other hand i find it difficult to give okapi
    > access to an underlying cluster or let okapi having its "own cluster"
    > within the cluster.
    > In my setup I create all backend-modules myself, and here i find it
    > redundant to let okapi know of all installable the modules. Since i
    > take care of all installed modules okapi does not need to know which
    > modules exists afterall.
    > My deployment-process for backend-modules right now looks like this:
    > [2]. The problem is that i have to create a new docker-image for each
    > backend-module in order to wrap the helper-script in it. As you can
    > imagine i would like to omit it and just use the folio-images instead.
    >
    > # Okapi-Cluster
    > The docs I found so far[3] explain running okapi in cluster-mode by
    > using a cluster-toolkit called hazelcast. It describes clustering on a
    > single machine and on separate machines.
    > There are some issues i have with this cluster-solution:
    > - I as admin have to decide on which okapi-node a backend-module is
    > deployed
    > - One Okapi-node is acting as proxy and is proxying all traffic to the
    > responsible okapi-node (the one on which the backend-module is
    > deployed), so there is no load-balancing possible between the okapi-
    > nodes
    > - Okapi itself seems to be stateful with its registered modules, so the
    > application and cannot be load-balanced at startup. However i could
    > loadbalance okapi by increasing its replicas after all modules were
    > registered. Perhaps it is an easy task to make module-data storage-
    > agnostic as well.
    >
    > In ideal an application should be infrastructure-agnostic and process
    > what it gets with the information it has. If the state is part of that
    > process it should be available to the application and each of its
    > replicas.
    >
    > I would like to hear your opinion about it and please correct me if I
    > am wrong. Also I intend to participate in the conference-call on friday
    > .
    >
    > Regards
    > Ulf
    >
    > [1]: http://folio-registry.aws.indexdata.com
    > [2]: https://tinyurl.com/y6l3mxvu
    > [3]:
    > https://github.com/folio-org/okapi/blob/5fa2880fccd1b3d9013e89fed401bd7233b8b994/doc/guide.md#running-in-cluster-mode
    >
    > --
    > Ulf Seltmann
    > Bereich Digitale Dienste
    > Webmaster
    >
    > Universitätsbibliothek Leipzig
    > Beethovenstraße 6, 04107 Leipzig
    >
    > T +49 341 97 30624
    >
    > seltmann@ub.uni-leipzig.de
    > www.ub.uni-leipzig.de
    >
    > ------------------------------------------------------
    > You received this message because you are subscribed to OLE Mailing List
    > "sysops-sig".
    > To unsubscribe from this list and stop receiving emails from it, follow
    > this link: http://archives.simplelists.com.
    > To post to this group, send email to
    > sysops-sig@ole-lists.openlibraryfoundation.org
    > <mailto:sysops-sig@ole-lists.openlibraryfoundation.org>.
    > Visit this group at
    > https://ole-lists.openlibraryfoundation.org<https://ole-lists.openlibraryfoundation.org>
    > .
    ------------------------------------------------------
    You received this message because you are subscribed to OLE Mailing List
    "sysops-sig".
    To unsubscribe from this list and stop receiving emails from it, follow
    this link: http://www.simplelists.com/confirm.php?u=YqPVWK1HxxWxsYYSLtSEYuWrm7FNkoOL.
    To post to this group, send email to
    sysops-sig@ole-lists.openlibraryfoundation.org
    <mailto:sysops-sig@ole-lists.openlibraryfoundation.org>.
    Visit this group at
    https://ole-lists.openlibraryfoundation.org<https://ole-lists.openlibraryfoundation.org>
    .