#97 Alternate Storage Backends for Course Material

Open
opened 1 week ago by hz · 0 comments
hz commented 1 week ago

After #76, we have the ability to store course content in a PostgreSQL backend (and the ability to write other such backends). Another step in the process of enabling remote checkers will be to allow all the course content to be stored in a different backend as well. Given how heavily catsoop relies on the filesystem, this probably involves setting up a kind of virtual filesystem in whatever DB engine we are using, and then caching relevant things that we want to be able to look up without a complicated traversal of a directory structure (publicly-accessible static files, pages, user information, etc).

Then we also need to provide abstractions for opening files, cs_local_python_import, etc.

Since the default method will be storing things on the filesystem, most subjects won’t actually have to change anything. But for folks who want to store course content in a database, they’ll need to rewrite anything that opens a file, and limit such access to files within the course hierarchy.

As with the log storage backend, I’ll start with PostgreSQL as the first alternative backend, and we can think about adding other backends later. I’ll likely borrow some pieces from #58, but probably with a lot of refactoring to try to properly abstract these things out, and avoid the situation I had there, where there were lots of checks for the storage backend throughout the source.

After #76, we have the ability to store course content in a PostgreSQL backend (and the ability to write other such backends). Another step in the process of enabling remote checkers will be to allow all the course content to be stored in a different backend as well. Given how heavily catsoop relies on the filesystem, this probably involves setting up a kind of virtual filesystem in whatever DB engine we are using, and then caching relevant things that we want to be able to look up without a complicated traversal of a directory structure (publicly-accessible static files, pages, user information, etc). Then we also need to provide abstractions for opening files, `cs_local_python_import`, etc. Since the default method will be storing things on the filesystem, most subjects won't actually have to change anything. But for folks who want to store course content in a database, they'll need to rewrite anything that opens a file, and limit such access to files within the course hierarchy. As with the log storage backend, I'll start with PostgreSQL as the first alternative backend, and we can think about adding other backends later. I'll likely borrow some pieces from #58, but probably with a _lot_ of refactoring to try to properly abstract these things out, and avoid the situation I had there, where there were lots of checks for the storage backend throughout the source.
hz added this to the 2020.9.0 milestone 1 week ago
hz added the
enhancement
label 1 week ago
hz added the
in progress
label 1 week ago
hz added the
devops
label 1 week ago
hz self-assigned this 1 week ago
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.