[kwlug-disc] Callout to CS majors/Other technically savvy klwug listers.

[Doug Moen]

> Why isn't github that server? Why isn't git the backend?

These are fine choices. To use github, I'd be using a combination of git protocol, OAuth and github's proprietary API.

I don't want to tie the project to a proprietary protocol and server owned by one company. I could support both github and gitlab, and allow people to run their own gitlab servers for hosting Curv content. Now it's more complicated, I need an abstraction layer that has a module for talking to github, and a different module for talking to gitlab, plus a system for indexing and searching for distributed content.

> - The "database" of information is a git repository. Maybe it just
>   points to locations where you can find interesting Curv projects.
>   Maybe it aggregates the code for those Curv projects into the
>   repository itself.

I want Curv to have a package system. A package is a collection of Curv source files and data files, which is versioned, and distributed as a single unit over the internet. Curv programs can refer to external packages using URLs. It's a distributed system.

Git repos are a good representation for packages, because they provide the abstraction of a file system hierarchy, and they provide versioning, and you can reference them using URLs.

There is a database of public Curv packages, but I'm fuzzy on the details of how it is stored, accessed, searched and updated. As you say, it could be a git repo.

Let's consider the user experience. You visit a web site, "curv3d.org". There is an array of images of top rated Curv projects: you can click on one and interact with it. You can search for an existing project. You can clone a project and edit it, or you can create a new project from scratch. This all runs in your web browser. If you try to save your changes, you are prompted to log in. Since this is a distributed system, you'll be logging in to github.com, or gitlab.com, or your own gitlab server. Then when you save your changes, they might be saved to a new repository, which must be given a name and created on the git server. If the repository is public, then the metadata for your project is added to the public package database.

> - You start this git repo and host it on Github. This can be thought
>   of as the canonical repository, where most people go to find Curv
>   libraries.
> 
> - Say somebody wants to add an entry to the repo. That is a pull
>   request followed by a merge. (I wish standard git supported pull
>   requests.) You could also do the Linux kernel thing where you have
>   subordinates who do much of the project filtering and standards, and
>   then you periodically pull those changes into your repo.

This sounds workable for an initial prototype, but I would like something a lot more automatic than this. I want to provide a very simple and easy user experience, see above.

If I'm only using github.com for Curv projects, I could probably use the github API to search for repos that match certain criteria, which automatically flag the repo as being a Curv project. Then nobody has to do a pull request; they just need to make their repo public, and the Curv web application will automatically add metadata to any new github repos it creates to make the repo findable.

If the system is fully distributed, and you can host Curv projects on your own gitlab server, then I need a more sophisticated solution. Maybe, if there is an open source project that supports distributed libraries of digital content, I could use that project's code to solve the problem, rather than reinventing the wheel.

> - Oh no! You have given up on Curv and take down the repo. That is
>   okay. If somebody else has forked the project then there is the
>   chance that the Curv repo lives on. 
> 
> - Oh no! There has been a big fight, and now there is a schism as to
>   what the repo should contain. So now there is a hard fork and
>   multiple repositories, similar to the way you can have multiple APT
>   repos in Debian. 
> 
> - The frontends for CurvNet would want the ability to point at
>   different repos, and have some mechanism for resolving conflicting
>   packages. Again, I think APT has solved this problem. 
> 
> - Oh no! Github does not offer free hosting for CurvNet any more.

Yes, this is a good reason to aim for a distributed design, rather than put all eggs in the github basket.

> The
>   backend is just a git repo, so it can be hosted elsewhere. Maybe
>   that elsewhere is a server people pay for (at which time you have to
>   figure out how to incentivize this payment). Or maybe you find some
>   Github competitor which will host the repo for free. The end users
>   just (or "just") need to point their repositories to the new
>   location.
> 
> - Maybe you could also allow repos to be mirrored in the same way APT
>   repos are mirrored. Git makes this possible, although maybe it is
>   not trivial to do this automatically?

Yes, this points at a weakness in my plan to use URLs to reference Curv packages. What if the package moves to another server? Then a bunch of other projects might break due to dangling references. I understand that the Go language project is maybe working on a solution to this problem in the next major Go release.

A hypothetical system that supports distributed libraries of digital content might solve this problem by providing abstract, location independent URLs that are magically mapped to real URLs at concrete addresses. I don't know how it would work, though.

I've seen lots of references to projects that enable a decentralized internet. Seems like there are a lot of different projects out there, not all of them requiring a blockchain. I have not had time to delve into this stuff yet.

> The nice thing about such a setup is that it would be federated in
> multiple ways. There could be many different repositories with
> different package listings. There could be ways for some repos to pull
> information from others. And there would be ways for repos to mirror
> each other. There is no "CurvCoin" that you mine to become
> unbelievably rich and/or attract VC funding, but that does not sound
> like one of your goals.
> 
> This may be wholly untenable, or it may be already implemented and
> available for you to use. But it sounds to me as if there are some
> solutions to this problem that do not require a lot of operational
> costs and do not need a blockchain.

Thanks for brainstorming this with me.

Doug.