#46 Remote Checkers

otevřený
otevřeno před 9 měsíci uživatelem hz · 1 komentářů
hz okomentoval před 9 měsíci

I like the filesystem, but Ike brought up a good point that catsoop would be easier to set up (paticularly with modern tools) if content lived in a database (or could be cached to a database) instead.

This would allow multiple small VPS’s to serve the web pages by talking to a DB server to grab cached page information as well as log information.

If the checker’s database were also stored on that machine, it would be relatively easy to connect multiple workers to it that could handle the checking (and to scale up/down in response to load).

I like the filesystem, but Ike brought up a good point that catsoop would be easier to set up (paticularly with modern tools) if content lived in a database (or could be cached to a database) instead. This would allow multiple small VPS's to serve the web pages by talking to a DB server to grab cached page information as well as log information. If the checker's database were also stored on that machine, it would be relatively easy to connect multiple workers to it that could handle the checking (and to scale up/down in response to load).
hz added the
enhancement
label před 9 měsíci
hz added the
devops
label před 9 měsíci
hz added the
in progress
label před 6 měsíci
hz přiřadil(a) sobě toto před 6 měsíci
hz okomentoval před 5 měsíci
Vlastník

6.145 has been running with a PostgreSQL backend for a while now (using the code from this branch), and it is mostly going smoothly. But there are a couple of pain points that make me want to investigate other solutions before going all-in on that one; specifically:

  1. allowing Postgres or filesystem storage complicates the codebase quite a lot (a lot of functionality has to be implemented both for PSQL and for the filesystem, not to mention that the existing codebase assumes filesystem access just about everywhere).

  2. there is a significant performance difference between the two, particularly when loading pages that read from a lot of logs (PSQL is way slower, even on localhost).

  3. full functionality would require rewrites of course material for anyone using cs_local_python_import or opening files, and opening files would be limited to files in the course tree.

These concerns are big enough that I think it’s worth exploring other options. In particular, I’m going to spend some time working on an alternative implementation that uses the filesystem, in the hopes that the above two issues go away while still preserving the key features of this. In particular, I think the important thing is the ability to have checkers run on separate machines from the web server (and the ability to dynamically add/remove checkers in response to load).

6.145 has been running with a PostgreSQL backend for a while now (using the code from [this branch](https://catsoop.org/git/hz/catsoop/src/branch/postgres)), and it is mostly going smoothly. But there are a couple of pain points that make me want to investigate other solutions before going all-in on that one; specifically: 1. allowing Postgres _or_ filesystem storage complicates the codebase quite a lot (a lot of functionality has to be implemented both for PSQL and for the filesystem, not to mention that the existing codebase assumes filesystem access just about everywhere). 1. there is a significant performance difference between the two, particularly when loading pages that read from a lot of logs (PSQL is way slower, even on `localhost`). 1. full functionality would require rewrites of course material for anyone using `cs_local_python_import` or opening files, and opening files would be limited to files in the course tree. These concerns are big enough that I think it's worth exploring other options. In particular, I'm going to spend some time working on an alternative implementation that uses the filesystem, in the hopes that the above two issues go away while still preserving the key features of this. In particular, I think the important thing is the ability to have checkers run on separate machines from the web server (and the ability to dynamically add/remove checkers in response to load).
hz změnil(a) název z Cache things in a traditional database na Remote Checkers před 5 měsíci
Přihlaste se pro zapojení do konverzace.
Bez milníku
Bez zpracovatelů
1 účastníků
Oznámení
Termín dokončení

Žádný termín dokončení.

Závislosti

Tento úkol momentálně nemá žádné závislosti.

Načítá se…
Není zde žádný obsah.