mirror of
https://github.com/Dictionarry-Hub/database.git
synced 2025-12-11 16:26:58 +00:00
add(devlog): update creation date in 'Vision Almost Realised' and add new logs for 'Modular Choices', 'Architecture Overhaul', and 'Profile Tweaks' (all old logs)
This commit is contained in:
19
dev_logs/Architecture Overhaul.md
Normal file
19
dev_logs/Architecture Overhaul.md
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
---
|
||||||
|
title: Architecture Overhaul
|
||||||
|
slug: architecture_overhaul
|
||||||
|
author: santiagosayshey
|
||||||
|
created: 2024-8-13
|
||||||
|
tags: [devlog, architecture]
|
||||||
|
---
|
||||||
|
|
||||||
|
Hey @everyone, here's a small update on what I've been working on lately:
|
||||||
|
|
||||||
|
# Architecture Overhaul
|
||||||
|
|
||||||
|
As the project has grown bigger, it's gotten quite difficult to keep track of and manage a billion different custom formats, quality profiles, etc. To help improve development productivity, I've planned a complete overhaul of Dictionarry's architecture. This starts with separating things into modules - namely a separate database which powers the website and the profilarr tool.
|
||||||
|
|
||||||
|
Next up is standardizing the actual entries inside the database. The biggest issue in development right now is making / editing / updating the same thing multiple times. If you have the same regex pattern for multiple CFs, it needs to be updated for each one of them. Quality profiles across different apps have miniscule differences in syntax (eg. web-dl in radarr vs web in sonarr), which means we need multiple files with tiny differences.
|
||||||
|
|
||||||
|
Working in this system is extremely error prone and time consuming. To fix this, I'm creating a standard unique to dictionarry based on a **single definition format**, i.e. Regex patterns, Custom Formats and Quality Profiles are defined once, and repeated in other places using foreign keys. I don't know exactly _how_ this will look, but the plan is simplicity above all. Outside of improving productivity, I hope this standard helps encourage people who feel less confident with custom formats / quality profiles make more intuitive changes to their own setups.
|
||||||
|
|
||||||
|
Now, the problem with this new and improved standard is - the arrs won't be able to read the files anymore. Solution: A compiler! This is where the fun begins; we take our simple, easy-to-develop-for files and push them through the compiler. Out pops the required syntax, with those weird naming rules (web-dl for radarr, web for sonarr), without the developer needing to ever worry about it!
|
||||||
28
dev_logs/Modular Choices.md
Normal file
28
dev_logs/Modular Choices.md
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
---
|
||||||
|
title: Profile Tweaks
|
||||||
|
slug: profile_tweaks
|
||||||
|
author: santiagosayshey
|
||||||
|
created: 2024-12-3
|
||||||
|
tags: [devlog, architecture, user_choice]
|
||||||
|
---
|
||||||
|
|
||||||
|
Hey @everyone, here's a small (but very important) post on the new update system!
|
||||||
|
|
||||||
|
## Current Profilarr
|
||||||
|
|
||||||
|
Currently, there is 0 support for updates in Profilarr. This is obviously not ideal; it's a nightmare to keep up to date with changes and almost certainly breaks any custom changes you make.
|
||||||
|
|
||||||
|
## Profilarr v1
|
||||||
|
|
||||||
|
Users will be able to view incoming and outgoing changes, as well as resolve any conflicts between the two. To achieve this, a user friendly GUI has been built on top of Git's merge functionality and allows fine control over what should be merged / ignored. More specifically, this functionality allows us to make custom changes and choose to retain them once a new update comes around.
|
||||||
|
|
||||||
|
- As an example, let's say you've made the Dolby Vision custom formats negative because your TV doesn't support it. A new update has come out which shuffles around HDR scores, and this leads to a merge conflict between the two custom format scores.
|
||||||
|
- In the settings page, you can choose to accept the incoming change or retain your local changes. Profilarr will 'remember' your choice and stop prompting you to update this custom format until a new update comes out, in which case, the situation repeats. Keep local or accept incoming.
|
||||||
|
|
||||||
|
### Settings Page
|
||||||
|
|
||||||
|
Profilarr now includes a dedicated page for 'Sync Settings'. It allows you to link / unlink a database repository, view and change branches as well as deal with incoming / outgoing changes and their conflicts. This page has been planned for developers too; you can add an authenticated github dev token to your environment and you have the ability to make changes directly to Profilarr's database (not to stable, obviously).
|
||||||
|
|
||||||
|
# Beta Release
|
||||||
|
|
||||||
|
- Still not quite ready yet, but I'm working hard to get it out! Stay tuned :hearts:
|
||||||
32
dev_logs/Profile Tweaks.md
Normal file
32
dev_logs/Profile Tweaks.md
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
---
|
||||||
|
title: Profile Tweaks
|
||||||
|
slug: profile_tweaks
|
||||||
|
author: santiagosayshey
|
||||||
|
created: 2024-12-3
|
||||||
|
tags: [devlog, architecture, user_choice]
|
||||||
|
---
|
||||||
|
|
||||||
|
Hey @everyone, I've been hard at work on the next Profilarr version over the past few weeks and have new stuff to show off!
|
||||||
|
|
||||||
|
The profiles we make are meant to be (really good) starting points, not a strict standard on what you _should_ be grabbing. Up until now, profiles existed as singular entities that don't respect custom changes. Merge conflict resolution was a big step in the right direction for this (read more in the last dev log), but it's a bit more hands on, and not something I expect most people to engage with.
|
||||||
|
|
||||||
|
Enter 'Profile Tweaks'. These are simple check boxes you can enable / disable and are unique to YOUR profiles. They will ALWAYS be respected, regardless of what updates we make to the base profile. For now, these tweaks include:
|
||||||
|
|
||||||
|
- Prefer Freeleech
|
||||||
|
- Allow Prereleases (CAMS, Screeners, etc)
|
||||||
|
- Language Strictness
|
||||||
|
- Allow Lossless audio
|
||||||
|
- Allow Dolby Vision without Fallback
|
||||||
|
- Allow bleeding edge codecs (AV-1, H266)
|
||||||
|
|
||||||
|
(Some are only available for specific profiles, eg lossless audio for 1080p Encode profiles).
|
||||||
|
|
||||||
|
If anyone has any tweak ideas (even super specific ones), please let me know and I'll work on getting it integrated! Here's an image of the Tweaks Tab:
|
||||||
|
|
||||||
|
## Profilarr Progress
|
||||||
|
|
||||||
|
- Progress is steady, I've been working on it every day since my semester ended. It's taken way, way longer than I've expected (sorry!) but I'm happy with how it's starting to look.
|
||||||
|
- Git integration is complete and working, but needs lots of testing.
|
||||||
|
- Data modules (custom formats, regex patterns, quality profiles) are complete and fully implement the existing logic from Radarr / Sonarr.
|
||||||
|
- I am currently in the progress of porting existing data to the new database (https://github.com/Dictionarry-Hub/database/tree/stable) in the new profilarr standard format. This is going to take a while, as I have to write descriptions, add tags, test cases, etc.
|
||||||
|
- Finally, I am starting to work on the compilation engine (https://discord.com/channels/1202375791556431892/1246504849265266738/1272756617041154049) and the import module. Once these things are complete, and I'm confident we won't run into massive bugs, I'll release a beta docker image. ETA? I really don't know, but I'm working as hard as I can.
|
||||||
56
dev_logs/Shiny New Stuff.md
Normal file
56
dev_logs/Shiny New Stuff.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
---
|
||||||
|
title: Shiny New Stuff
|
||||||
|
slug: shiny_new_stuff
|
||||||
|
author: santiagosayshey
|
||||||
|
created: 2024-8-19
|
||||||
|
tags: [devlog, architecture]
|
||||||
|
---
|
||||||
|
|
||||||
|
hey @everyone, hope you guys are well. Here's another update!
|
||||||
|
|
||||||
|
# Motivation
|
||||||
|
|
||||||
|
I've been really struggling to work on this project for a few months now - I'll finally get some time at the end of the week but feel completely unmotivated to work on it for more than an hour. Well... after cracking the architecture problem last week and seeing all the support from you guys, I've felt especially motivated to dive back in.
|
||||||
|
|
||||||
|
# Profilarr v2 (not really v2 but it sounded cool)
|
||||||
|
|
||||||
|
Profilarr is getting some really nice upgrades. Here's an outline of the most important ones:
|
||||||
|
|
||||||
|
## It's now a full stack application.
|
||||||
|
|
||||||
|
This means we have a frontend: a site that users can visit to adjust, import, and export regexes, custom formats, and quality profiles. It's built in a way that aims to 'remaster' how it's implemented in Radarr/Sonarr. All the existing functionality is there, but with some really nice quality of life features:
|
||||||
|
|
||||||
|
- **Single definition format**: As outlined in the previous dev log, Profilarr's version of this system will use a single definition format. Notably, this allows you to set regex patterns ONCE, then add that regex as a condition inside a custom format.
|
||||||
|
- **Sorting and Filtering**: You can now sort and filter items by title, date modified, etc.
|
||||||
|
- **Exporting/Importing**: The standard format now allows _everyone_ to import/export regexes, custom formats, and quality profiles freely - no need to query APIs to do this anymore.
|
||||||
|
- **Syncing**: Instead of clogging up everyone's arrs with unused custom formats, the sync functionality now only imports _used_ items.
|
||||||
|
- **Mass selection**: You can mass select items to import/export/sync/delete.
|
||||||
|
- **Tags**: Instead of manual selection, you can set tags on specific custom formats/quality profiles that should be synced. This works similar to how Prowlarr uses tags to selectively sync indexers. Since we are also using the same database for the website, tags can also be used for little tidbits of information too. Like where a release group is an internal at!
|
||||||
|
- **Testing**: Developers can now permalink regexes to regex101. This makes it really easy to develop and test simultaneously.
|
||||||
|
- **Descriptions**: You can now explain what specific items are for. No need to look it up on the website to see what it does.
|
||||||
|
|
||||||
|
## Backend Improvements
|
||||||
|
|
||||||
|
The backend is essentially what Profilarr is right now - a tool to sync some JSON files to your arrs. However, this also has some major improvements:
|
||||||
|
|
||||||
|
- **Git integration**: You can select a remote repository to connect to and:
|
||||||
|
- Add, commit, and push files; branch off; merge into. This isn't that useful for end users, but I cannot stress enough how much time and suffering this has saved me. Being able to revert regex/custom format/quality profiles to the last commit is my favorite thing I've ever coded.
|
||||||
|
- **Branching**: You can have different branches for different things. Of course, this is useful for development, but it also allows you to do things like: separate setups for Radarr/Sonarr/Lidarr. Most importantly, it allows us developers to set stable, dev, and feature branches.
|
||||||
|
- **Pulling**: You can now pull in changes from specific branches from a remote repository. You can view differences and decide if you want to pull these changes in. You can set it to be automatic and only alert on merge conflicts (you change something, but an incoming change for that item exists as well). You can choose to get the most stable branch or the latest features merged into develop.
|
||||||
|
- **External sources**: You can set your own repo of regexes, custom formats, and quality profiles and share it with whoever you want. As I mentioned in my last dev log, I'll be working on a compiler to convert our standard Profilarr format with the existing arr format. The really cool thing about this is it works both ways. This means the git integration + compiler will allow you to use Profilarr with the trash guides. It'll probably take some tweaking, but I know it's definitely possible now.
|
||||||
|
|
||||||
|
## Containerisation
|
||||||
|
|
||||||
|
Profilarr will FINALLY be dockerised.
|
||||||
|
|
||||||
|
# Development
|
||||||
|
|
||||||
|
With these changes in place, it has massively improved and sped up development. Working in a proprietary tool now allows me the freedom to just implement a feature whenever I want to. Want to filter custom formats with the release tier tag? Boom, implemented. Want to auto-apply scores to custom formats in quality profiles based on tags? Boom, implemented.
|
||||||
|
|
||||||
|
## Machine Learning
|
||||||
|
|
||||||
|
This part is mostly speculation and rambling - nothing concrete yet. I really want to incorporate some kind of AI help into Profilarr. A button you can press to auto-generate regex or a custom format. I've read countless Reddit posts of someone unfamiliar with regex/custom formats/profiles asking for help in trying to learn. "How do I write a custom format that matches x265 releases under size x?" It's so easily solved using AI.
|
||||||
|
|
||||||
|
I want to implement this one day, I just don't have enough knowledge or experience to do it yet. The best I've come up with is something that sends a request to OpenAI's API with a prompt. The results are less than ideal. But just imagine the future where some kind of machine learning tool has access to an entire database of regexes, custom formats, and quality profiles curated by hundreds of people, and can use that knowledge to predict patterns and truly tailor stuff to suit people's needs. Who knows if it ever gets to that point, but that's my vision for Dictionarry.
|
||||||
|
|
||||||
|
Ramble over, as you can tell I've been feeling pretty motivated lately!
|
||||||
@@ -2,13 +2,11 @@
|
|||||||
title: Vision (Almost) Realised
|
title: Vision (Almost) Realised
|
||||||
slug: vision_almost_realised
|
slug: vision_almost_realised
|
||||||
author: santiagosayshey
|
author: santiagosayshey
|
||||||
created: 2025-12-24
|
created: 2024-12-24
|
||||||
tags: [devlog, architecture, gppi]
|
tags: [devlog, architecture, gppi]
|
||||||
---
|
---
|
||||||
|
|
||||||
hey @everyone, small log for today!
|
Hey @everyone, small log for today!
|
||||||
|
|
||||||
## Vision (almost) Realized
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ python profile_compile.py 'profiles/1080p Encode.yml' '1080p Encode (sonarr - master).json' -s
|
$ python profile_compile.py 'profiles/1080p Encode.yml' '1080p Encode (sonarr - master).json' -s
|
||||||
|
|||||||
Reference in New Issue
Block a user