size of repo and generated files/translated_images from Co-op Translator

[bmerkle]

I recently had to get a clean fork of the project again and was a bit puzzled about the grown size I did a bit analysis and want to share the following obervations:

I am curious to understand why there is such a heavy generation of translated_images (see the folder)
IMO, these are generated files which should not necessarily be put under version control but rather into the output (maybe we should think about a publishing pipeline which then included the whole translations)

The same question is about translations (lot of files, it does not cause size problems (as IMO translated_images) but do we have to store generated files in the repo ? I always tell my students not to store them, because they can be reproduced via the tool chain and they are mainly output (like obj files or dll or lib files or clang generated tables).

So here are some numbers:

• the main repo has 1GB data (without translated images)
• the "translated_images" folder has 3 GB ! it can not be even displayed in github webpage anymore (you get an error message because > 6000 files, it can not display any git information anymore)
• the "translated" folder has about 4000 files, while the whole rest of the project has 1800 files for the content.

I really like Co-op Translator as it save a lot of work, however i think we should not store its output in git.
Rather push it into a container or a some binary which we store in artifactory or the github registry..

what do you think about this ?

Below are some screenshots from WizTree which shows that "translated_images" and "translation" contribute significantly to the size. There are also my downloaded LLMs for chapter 19-slnm, the python venv and the git repo store itself above those numbers, but then these two output folder immediatly follow.

Here is a picture of the github UI which has problems.
https://github.com/microsoft/generative-ai-for-beginners/tree/main/translated_images

[github-actions]

👋 Thanks for contributing @bmerkle! We will review the issue and get back to you soon.

[bmerkle]

Hey @skytin1004, as i am not a expert in co-op-translator maybe you can help (as you are the creator 👍 )

my questions are:

• could we store the output somewhere else better (e.g. in a package manager as a generated resource file/jar file/zip file) ?
• could we store the output for images in different folders instead of just one folder (the current approach will bring git or filesystems in general to its limits)

I am happy to create feature requests with descriptions and maybe a PR if you think this would be useful :-)

[skytin1004]

Hi, @bmerkle thanks for raising this.

I considered the artifact-based approach you suggested.
Technically it works well for static-site pipelines, but for this repository it’s very difficult to apply. The main reason is that users currently browse and search the translated content directly inside the GitHub file tree, and Co-op Translator is intentionally designed to place the translated files there.

If the output moved into artifacts, the translated documents would no longer be visible on GitHub, which breaks the way users consume this repo today.

Given that limitation, the only feasible alternative I’ve been thinking about is creating a separate translations repository:
Repo A: source (current repo)
Repo B: translated + translated images (similar to how this repo currently works)

Co-op Translator syncs Repo B whenever the source updates This would keep the main repo small and prevent issues, while still allowing users to view translations on GitHub. But I admit it would make the maintenance workflow a bit more complex for the maintainers.

I’ll keep exploring if there’s a cleaner approach on the Co-op Translator side as well.

Happy to continue discussing if there are other ideas or constraints we should consider.

[skytin1004]

@bmerkle Regarding the second point, splitting the translated images into multiple subfolders is definitely something Co-op Translator can support.

You are right that keeping thousands of files in a single directory can cause issues in both the GitHub UI and local filesystems. I will explore adding an option to organize the translated images by language or by chapter so the structure scales better :)