Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?

[Jonathan Dieter]

Right now, we're not using zdicts for the F34 zchunk metadata because
they were only added in fedora-repo-zdicts-2103.1-2 (which should now
be in the updates repo in all current Fedora releases).

If we could update fedora-repo-zdicts to 2103.1-2 on whichever servers
generate the metadata (preferably before the 34 GA metadata is
generated), that should significantly reduce the size of the metadata.

Thanks,

Jonathan
_______________________________________________
infrastructure mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Kevin Fenzi]

Post by Jonathan Dieter
Right now, we're not using zdicts for the F34 zchunk metadata because
they were only added in fedora-repo-zdicts-2103.1-2 (which should now
be in the updates repo in all current Fedora releases).
If we could update fedora-repo-zdicts to 2103.1-2 on whichever servers
generate the metadata (preferably before the 34 GA metadata is
generated), that should significantly reduce the size of the metadata.

Which metadata are you looking at?

bodhi-backend01 has fedora-repo-zdicts-2103.1-2.fc32.noarch
compose-branched does not have that package installed.

But it should be using the one in the chroot right?

kevin

[Jonathan Dieter]

Post by Kevin Fenzi

Post by Jonathan Dieter
Right now, we're not using zdicts for the F34 zchunk metadata because
they were only added in fedora-repo-zdicts-2103.1-2 (which should now
be in the updates repo in all current Fedora releases).
If we could update fedora-repo-zdicts to 2103.1-2 on whichever servers
generate the metadata (preferably before the 34 GA metadata is
generated), that should significantly reduce the size of the
metadata.

Which metadata are you looking at?
bodhi-backend01 has fedora-repo-zdicts-2103.1-2.fc32.noarch
compose-branched does not have that package installed.
But it should be using the one in the chroot right?

I'm not sure. Looking at the zchunk metadata from the latest compose,
there's no dictionary (Chunk 0 is 0 length):

$ zck_read_header -c 8ec4565f29e273b132baee435e7e3ef6cdae866ba9c9811f87caa5bc9a1f112a-primary.xml.zck | head -n 11
Overall checksum type: SHA-256
Header size: 512869
Header checksum: bcc8e5e5d152d6e7e0036a1b9e2661633ebd5e33cab55a976ca7baab3f1f4e1d
Data size: 29656818
Data checksum: eea9776727a3b5a9b2232f78d91cc8ef2104622ff5fc5a9785ce2f218b5b2dcb
Chunk count: 25580
Chunk checksum type: SHA-512/128

Chunk Checksum Start Comp size Size
0 00000000000000000000000000000000 512869 0 0
1 cfb57636818534e317930134427b8e4b 512869 122 160

Looking at the updates-testing metadata, there is a dictionary:

$ zck_read_header -c df4f7320d337a2d90594e1c8a91bcd2448d18854a57753ee2f83edbeb9880ace-primary.xml.zck | head -n 11
Overall checksum type: SHA-256
Header size: 57227
Header checksum: 00e2c3bc5b9413e39d109f9d121a38e41076113df9113f02fbea43bfc950b99b
Data size: 3969888
Data checksum: 94ad299653a3b8f6d5428cbd2115579d38e435de86e9d280dc08dccb4ef46ae3
Chunk count: 2841
Chunk checksum type: SHA-512/128

Chunk Checksum Start Comp size Size
0 a130547b7e9263e4722be934af8d1552 57227 29904 112640
1 450a0b82e6b3499ec7996832bc091af6 87131 102 160

The package needs to be available in whatever environment createrepo_c
is run from, so if we're running createrepo_c from a chroot, that's
where the package should be.

I don't know if it helps, but looking at
https://kojipkgs.fedoraproject.org/compose/branched/Fedora-34-20210401.n.0/logs/x86_64/createrepo-Everything.rpm.x86_64.log
it looks like the dictionaries aren't available during the compose.

If there's anything else I can do to help troubleshoot, please let me
know.

Thanks,

Jonathan

[Kevin Fenzi]

ok. I've installed fedora-repo-zdicts on both branched and rawhide
composers.

Lets see if that works in tomorrow's compose.

kevin

[Jonathan Dieter]

Post by Kevin Fenzi
ok. I've installed fedora-repo-zdicts on both branched and rawhide
composers.
Lets see if that works in tomorrow's compose.

Thanks so much! Fingers crossed. :)

Jonathan

[Jonathan Dieter]

Post by Kevin Fenzi
ok. I've installed fedora-repo-zdicts on both branched and rawhide
composers.
Lets see if that works in tomorrow's compose.

Thanks so much!  Fingers crossed. :)

I've just checked the latest compose and the repodata now has zdicts.
Thanks again Kevin for getting that package into the composers.

Jonathan

[Kevin Fenzi]

Post by Jonathan Dieter

Post by Kevin Fenzi
ok. I've installed fedora-repo-zdicts on both branched and rawhide
composers.
Lets see if that works in tomorrow's compose.

Thanks so much!  Fingers crossed. :)

I've just checked the latest compose and the repodata now has zdicts.
Thanks again Kevin for getting that package into the composers.

Excellent. Easy fix. :)

kevin