database/bundles/dev_logs.json

[
  {
    "_id": "Architecture Overhaul",
    "content": "Hey @everyone, here's a small update on what I've been working on lately:\n\nAs the project has grown bigger, it's gotten quite difficult to keep track of and manage a billion different custom formats, quality profiles, etc. To help improve development productivity, I've planned a complete overhaul of Dictionarry's architecture. This starts with separating things into modules - namely a separate database which powers the website and the profilarr tool.\n\nNext up is standardizing the actual entries inside the database. The biggest issue in development right now is making / editing / updating the same thing multiple times. If you have the same regex pattern for multiple CFs, it needs to be updated for each one of them. Quality profiles across different apps have miniscule differences in syntax (eg. web-dl in radarr vs web in sonarr), which means we need multiple files with tiny differences.\n\nWorking in this system is extremely error prone and time consuming. To fix this, I'm creating a standard unique to dictionarry based on a **single definition format**, i.e. Regex patterns, Custom Formats and Quality Profiles are defined once, and repeated in other places using foreign keys. I don't know exactly _how_ this will look, but the plan is simplicity above all. Outside of improving productivity, I hope this standard helps encourage people who feel less confident with custom formats / quality profiles make more intuitive changes to their own setups.\n\nNow, the problem with this new and improved standard is - the arrs won't be able to read the files anymore. Solution: A compiler! This is where the fun begins; we take our simple, easy-to-develop-for files and push them through the compiler. Out pops the required syntax, with those weird naming rules (web-dl for radarr, web for sonarr), without the developer needing to ever worry about it!\n\nHere's a canvas page I made in Obsidian which visualizes this architecture:\n\n![Archiecture Diagram](https://i.imgur.com/HcXFNHU.png)",
    "last_modified": "2025-01-22T04:42:17.206980+00:00",
    "title": "Architecture Overhaul",
    "slug": "architecture_overhaul",
    "author": "santiagosayshey",
    "created": "2024-8-13",
    "tags": [
      "devlog",
      "architecture"
    ]
  },
  {
    "_id": "Modular Choices",
    "content": "Hey @everyone, here's a small (but very important) post on the new update system!\n\n## Current Profilarr\n\nCurrently, there is 0 support for updates in Profilarr. This is obviously not ideal; it's a nightmare to keep up to date with changes and almost certainly breaks any custom changes you make.\n\n## Profilarr v1\n\nUsers will be able to view incoming and outgoing changes, as well as resolve any conflicts between the two. To achieve this, a user friendly GUI has been built on top of Git's merge functionality and allows fine control over what should be merged / ignored. More specifically, this functionality allows us to make custom changes and choose to retain them once a new update comes around.\n\n- As an example, let's say you've made the Dolby Vision custom formats negative because your TV doesn't support it. A new update has come out which shuffles around HDR scores, and this leads to a merge conflict between the two custom format scores.\n- In the settings page, you can choose to accept the incoming change or retain your local changes. Profilarr will 'remember' your choice and stop prompting you to update this custom format until a new update comes out, in which case, the situation repeats. Keep local or accept incoming.\n\n### Settings Page\n\nProfilarr now includes a dedicated page for 'Sync Settings'. It allows you to link / unlink a database repository, view and change branches as well as deal with incoming / outgoing changes and their conflicts. This page has been planned for developers too; you can add an authenticated github dev token to your environment and you have the ability to make changes directly to Profilarr's database (not to stable, obviously).\n\n# Beta Release\n\n- Still not quite ready yet, but I'm working hard to get it out! Stay tuned :hearts:",
    "last_modified": "2025-01-22T04:42:17.206980+00:00",
    "title": "Modular Choices",
    "slug": "modular_choices",
    "author": "santiagosayshey",
    "created": "2024-12-3",
    "tags": [
      "devlog",
      "architecture",
      "user_choice"
    ]
  },
  {
    "_id": "Profile Tweaks",
    "content": "Hey @everyone, I've been hard at work on the next Profilarr version over the past few weeks and have new stuff to show off!\n\nThe profiles we make are meant to be (really good) starting points, not a strict standard on what you _should_ be grabbing. Up until now, profiles existed as singular entities that don't respect custom changes. Merge conflict resolution was a big step in the right direction for this (read more in the last dev log), but it's a bit more hands on, and not something I expect most people to engage with.\n\nEnter 'Profile Tweaks'. These are simple check boxes you can enable / disable and are unique to YOUR profiles. They will ALWAYS be respected, regardless of what updates we make to the base profile. For now, these tweaks include:\n\n- Prefer Freeleech\n- Allow Prereleases (CAMS, Screeners, etc)\n- Language Strictness\n- Allow Lossless audio\n- Allow Dolby Vision without Fallback\n- Allow bleeding edge codecs (AV-1, H266)\n\n(Some are only available for specific profiles, eg lossless audio for 1080p Encode profiles).\n\nIf anyone has any tweak ideas (even super specific ones), please let me know and I'll work on getting it integrated! Here's an image of the Tweaks Tab:\n\n## Profilarr Progress\n\n- Progress is steady, I've been working on it every day since my semester ended. It's taken way, way longer than I've expected (sorry!) but I'm happy with how it's starting to look.\n- Git integration is complete and working, but needs lots of testing.\n- Data modules (custom formats, regex patterns, quality profiles) are complete and fully implement the existing logic from Radarr / Sonarr.\n- I am currently in the progress of porting existing data to the new database (https://github.com/Dictionarry-Hub/database/tree/stable) in the new profilarr standard format. This is going to take a while, as I have to write descriptions, add tags, test cases, etc.\n- Finally, I am starting to work on the compilation engine (https://discord.com/channels/1202375791556431892/1246504849265266738/1272756617041154049) and the import module. Once these things are complete, and I'm confident we won't run into massive bugs, I'll release a beta docker image. ETA? I really don't know, but I'm working as hard as I can.",
    "last_modified": "2025-01-22T04:42:17.206980+00:00",
    "title": "Profile Tweaks",
    "slug": "profile_tweaks",
    "author": "santiagosayshey",
    "created": "2024-12-3",
    "tags": [
      "devlog",
      "architecture",
      "user_choice"
    ]
  },
  {
    "_id": "Shiny New Stuff",
    "content": "hey @everyone, hope you guys are well. Here's another update!\n\n# Motivation\n\nI've been really struggling to work on this project for a few months now - I'll finally get some time at the end of the week but feel completely unmotivated to work on it for more than an hour. Well... after cracking the architecture problem last week and seeing all the support from you guys, I've felt especially motivated to dive back in.\n\n# Profilarr v2 (not really v2 but it sounded cool)\n\nProfilarr is getting some really nice upgrades. Here's an outline of the most important ones:\n\n## It's now a full stack application.\n\nThis means we have a frontend: a site that users can visit to adjust, import, and export regexes, custom formats, and quality profiles. It's built in a way that aims to 'remaster' how it's implemented in Radarr/Sonarr. All the existing functionality is there, but with some really nice quality of life features:\n\n- **Single definition format**: As outlined in the previous dev log, Profilarr's version of this system will use a single definition format. Notably, this allows you to set regex patterns ONCE, then add that regex as a condition inside a custom format.\n- **Sorting and Filtering**: You can now sort and filter items by title, date modified, etc.\n- **Exporting/Importing**: The standard format now allows _everyone_ to import/export regexes, custom formats, and quality profiles freely - no need to query APIs to do this anymore.\n- **Syncing**: Instead of clogging up everyone's arrs with unused custom formats, the sync functionality now only imports _used_ items.\n- **Mass selection**: You can mass select items to import/export/sync/delete.\n- **Tags**: Instead of manual selection, you can set tags on specific custom formats/quality profiles that should be synced. This works similar to how Prowlarr uses tags to selectively sync indexers. Since we are also using the same database for the website, tags can also be used for little tidbits of information too. Like where a release group is an internal at!\n- **Testing**: Developers can now permalink regexes to regex101. This makes it really easy to develop and test simultaneously.\n- **Descriptions**: You can now explain what specific items are for. No need to look it up on the website to see what it does.\n\n## Backend Improvements\n\nThe backend is essentially what Profilarr is right now - a tool to sync some JSON files to your arrs. However, this also has some major improvements:\n\n- **Git integration**: You can select a remote repository to connect to and:\n  - Add, commit, and push files; branch off; merge into. This isn't that useful for end users, but I cannot stress enough how much time and suffering this has saved me. Being able to revert regex/custom format/quality profiles to the last commit is my favorite thing I've ever coded.\n  - **Branching**: You can have different branches for different things. Of course, this is useful for development, but it also allows you to do things like: separate setups for Radarr/Sonarr/Lidarr. Most importantly, it allows us developers to set stable, dev, and feature branches.\n  - **Pulling**: You can now pull in changes from specific branches from a remote repository. You can view differences and decide if you want to pull these changes in. You can set it to be automatic and only alert on merge conflicts (you change something, but an incoming change for that item exists as well). You can choose to get the most stable branch or the latest features merged into develop.\n  - **External sources**: You can set your own repo of regexes, custom formats, and quality profiles and share it with whoever you want. As I mentioned in my last dev log, I'll be working on a compiler to convert our standard Profilarr format with the existing arr format. The really cool thing about this is it works both ways. This means the git integration + compiler will allow you to use Profilarr with the trash guides. It'll probably take some tweaking, but I know it's definitely possible now.\n\n## Containerisation\n\nProfilarr will FINALLY be dockerised.\n\n# Development\n\nWith these changes in place, it has massively improved and sped up development. Working in a proprietary tool now allows me the freedom to just implement a feature whenever I want to. Want to filter custom formats with the release tier tag? Boom, implemented. Want to auto-apply scores to custom formats in quality profiles based on tags? Boom, implemented.\n\n## Machine Learning\n\nThis part is mostly speculation and rambling - nothing concrete yet. I really want to incorporate some kind of AI help into Profilarr. A button you can press to auto-generate regex or a custom format. I've read countless Reddit posts of someone unfamiliar with regex/custom formats/profiles asking for help in trying to learn. \"How do I write a custom format that matches x265 releases under size x?\" It's so easily solved using AI.\n\nI want to implement this one day, I just don't have enough knowledge or experience to do it yet. The best I've come up with is something that sends a request to OpenAI's API with a prompt. The results are less than ideal. But just imagine the future where some kind of machine learning tool has access to an entire database of regexes, custom formats, and quality profiles curated by hundreds of people, and can use that knowledge to predict patterns and truly tailor stuff to suit people's needs. Who knows if it ever gets to that point, but that's my vision for Dictionarry.\n\nRamble over, as you can tell I've been feeling pretty motivated lately!",
    "last_modified": "2025-01-22T04:42:17.206980+00:00",
    "title": "Shiny New Stuff",
    "slug": "shiny_new_stuff",
    "author": "santiagosayshey",
    "created": "2024-8-19",
    "tags": [
      "devlog",
      "architecture"
    ]
  },
  {
    "_id": "Vision Almost Realised",
    "content": "Hey @everyone, small log for today!\n\n```bash\n$ python profile_compile.py 'profiles/1080p Encode.yml' '1080p Encode (sonarr - master).json' -s\nConverted profile saved to: 1080p Encode (sonarr - master).json\n\n$ python importarr.py\nImporting Quality Profiles to sonarr : Master\nUpdating '1080p Encode' quality profile : SUCCESS\n```\n\nThese two commands are the culmination of the architecture overhaul I talked about in August: https://discord.com/channels/1202375791556431892/1246504849265266738/1272756617041154049. The Profilarr standard format _**works**_. A typical profile is now about 300 lines (down from 1000 each for radarr / sonarr), is able to be compiled from PSF to Radarr OR Sonarr (and back!). Regex patterns allow format resolution, so no more editing the same thing 5, 10... 20 times.\n\nI'm currently in the process of hooking up the database to the new website, and that's looking pretty cool too. I cannot even explain how good it feels to be able to edit a profile once inside Profilarr, push those changes directly from Profilarr, have those changes reflected as incoming changes for end users, and as updated information on the website all in one fell swoop.\n\nIt's taken a huge effort the past 4 months, and I still have to actually connect it to the backend, but I'm fairly happy with how it's turned out. The changes won't be all that evident right away for you guys, but it's going to save me (and anyone who wants to contribute) hours upon hours of development time for everything that I have planned.\n\n## Golden Popcorn Performance Index Changes\n\nThe current GPPi algorithm is strong, but fundamentally flawed. It does not take into consideration release groups who have no data. There are terrific new groups (ZoroSenpai for example) who should be tier ~2 at least, but aren't simply because they have no data. How do we fix this?\n\n### Popularity\n\nFor every encode at a specific resolution for a movie / tv show that is currently _popular_, a release group receives +1 score to their GPPi. At the end of every month, the score is reset, and the previous score is normalized (tbd on how) and added to their permanent GPPi score (up to a certain point and probably never past tier ~3)\n\nThis process will be completely automatic and will hopefully solve the problem of new good release groups.\n\n### Grouping\n\nThe previous 'tiers' for release groups was just natural intuitive grouping. Humans are surprisingly very, very good at pattern recognition so it was never really a problem. However, it was manual, and we dont like manual around here. Enter 'K Means Clustering'. Essentially it's just a fancy algorithm that finds natural break points between groups of numbers. Using K means, I've dropped the number of 1080p Tiers from 7 down to 5 which in turn has increased immutability. Small changes, but will be important in the long run.\n\n## Thank You!\n\nThat's all for today, I hope everyone's doing alright and enjoying the holidays :grinning:",
    "last_modified": "2025-01-22T04:42:17.206980+00:00",
    "title": "Vision (Almost) Realised",
    "slug": "vision_almost_realised",
    "author": "santiagosayshey",
    "created": "2024-12-24",
    "tags": [
      "devlog",
      "architecture",
      "gppi"
    ]
  }
]