If you want to only have the collections you prefer in your image, then you should either use a proper base image (fedora/centos/ubi, I think ansible-builder recently added support for other distros as well, you just have to define the package manager commands); or you could use the community-minimal-ee which contains only 3 ansible.* collections.
You should also consider using multiple EE’s. The AWX-EE has everything it needs for AWX to function. This should be the control plane’s EE (i.e. control_plane_ee_image). Your custom EE’s can be added as extra execution environments and you can set them as the default EE for your Org, while still allowing the AWX itself to use its own EE.