[Vote ended on 2024-12-02] Saving space on PyPI

Project Discussions
21

Yep, that sounds good to me…

22

After talking about this at the community meeting, let’s proceed as follows: If nobody objects, start two votes this weekend:

Vote 1: Should we delete wheels first, before deleting actual releases?

  1. Yes. First delete wheels (starting with the oldest, until there’s enough space) before deleting anything else.
  2. No. Always delete a full release (both source distribution and wheel).

Vote 2: Should we delete first all apha1 releases, then all alpha2 releases, etc. (and for each pre-rerelease type, start with the oldest)? Or should we first delete the oldest pre-release, then the second oldest, etc.?

  1. First delete the oldest to newest alpha1, then the oldest to newest alpha2, then alpha3, beta1, beta2, etc.
  2. First delete the oldest pre-relrease, then the second oldest, etc.

Also we think that the policy should be documented on the ansible-build-data docsite (Ansible Package Release Management, built from ansible-build-data/docs at main · ansible-community/ansible-build-data · GitHub), so that if we happen to be in this situation again in some years we can find the decision next to all other stuff relevant to doing Ansible releases.

Any opinions / objections / …, in particular from @SteeringCommittee?

2 Likes
23

I think the two votes make sense, but I am more hesitant about having a permanent policy. I guess it makes sense, but I really don’t want to be in this situation again. We should try to be more proactive about checking our data usage and requesting increases well in advanced of the point we’d run out of space. Perhaps we can have a policy to check data usage before each a1 release and request an increase if we have, say, around 3 GB or less available. It also might make sense to start updating the sdists to ansible-build-data’s Releases section so we have the release artifacts preserved in two places. This could be integrated into the release workflow pretty easily.

2 Likes
24

I agree. On the other hand, if we run into this situation again it would be great to not have to discuss again what we should delete first.

@felixfontein AFAIR you’ve created the release workflow. Do you think it would be possible to add “X from Y GB used on PyPI” before or after the release to the PR?

25

I didn’t create it, but worked on it. While we can show how much is currently in use, we have no idea how large the limit is. Currently only folks with admin rights to the ansible PyPI repo can find out the actual limit.

26

As @mariolenz wrote it would be great if we never need this again. But if we do, we should be able to do emergency measures (with a process already defined) immediately instead of having to discuss for 1-2 weeks (or more) first before we can do another release.

27

Here are two votes. @SteeringCommittee members should vote for the votes marked with [Steering Committee], and other community members should vote for the votes marked with [Community]. The votes end on December 1st!

[Steering Committee] When having to delete releases due to lack of space, should we delete wheels first, before deleting actual releases?
  • Yes. First delete wheels (starting with the oldest, until there’s enough space) before deleting anything else.
  • No. Always delete a full release (both source distribution and wheel).
0 voters
[Community] When having to delete releases due to lack of space, should we delete wheels first, before deleting actual releases?
  • Yes. First delete wheels (starting with the oldest, until there’s enough space) before deleting anything else.
  • No. Always delete a full release (both source distribution and wheel).
0 voters
[Steering Committee] Should we delete first all apha1 releases, then all alpha2 releases, etc. (and for each pre-rerelease type, start with the oldest)? Or should we first delete the oldest pre-release, then the second oldest, etc.?
  • First delete the oldest to newest alpha1, then the oldest to newest alpha2, then alpha3, beta1, beta2, etc.
  • First delete the oldest pre-release, then the second oldest, etc.
0 voters
[Community] Should we delete first all apha1 releases, then all alpha2 releases, etc. (and for each pre-rerelease type, start with the oldest)? Or should we first delete the oldest pre-release, then the second oldest, etc.?
  • First delete the oldest to newest alpha1, then the oldest to newest alpha2, then alpha3, beta1, beta2, etc.
  • First delete the oldest pre-release, then the second oldest, etc.
0 voters
28

@SteeringCommittee Please vote if you haven’t done yet!

29

Please note that we extended the vote until 2024-12-02. When Mario and I talked about this earlier today the vote hadn’t reached quorum yet (it has reached it by now). Also it looks like the Ansible releases planned for Tuesday might get delayed since currently AZP doesn’t work for collections (this affects community.general, community.crypto, community.docker, community.rabbitmq, and hetzner.hcloud), which prevents merging PRs and creating releases, in particular the three planned community.general releases that should go into the Ansible releases.

30

I’ve now closed the vote. We reached quorum in both votes. The results are:

  1. We first delete wheels of existing pre-releases, and only start deleting pre-releases once all wheels of pre-releases have been deleted. We start by deleting the wheels for the oldest pre-releases that have wheels, and continue until enough space is left (unless all wheels of pre-releases are gone and we have to delete releases).

  2. When we have to delete releases, we start with the oldest pre-release, then the second-oldest pre-release, and so on. We also don’t unnecessary delete more pre-releases than needed to have enough space to make releases.

Thanks everyone! @gundalow can you delete some wheels of pre-releases so we have ~300 MB free for the three releases?

31

Thanks for all the discussion folks

I’ve just deleted:

  • ansible-6.0.0a1-py3-none-any.whl (41.0 MB)
  • ansible-6.0.0a2-py3-none-any.whl (39.2 MB)

Project size is now 9.7 GiB

32

We also had to delete ansible-6.0.0a3-py3-none-any.whl to have enough space to do all three releases (11.1.0, 10.7.0, 9.13.0). Fortunately we likely won’t have to do releases until the end of January, so until then we don’t have to delete anything anymore. Hopefully by then Project Limit Request: ansible - 20 GB · Issue #5106 · pypi/support · GitHub has been processed. (The current PyPI repo size is 9.962 GiB.)