I have been using AWS MWAA for a while, but recently had some issues with conflicting packages. So I decided to investigate this a little further.

For this purpose:

1. I created an MWAA v2.0.2 instance without additional Python packages. The idea here was to grab the packages they install by default and try to replicate it in my local.

2. Here are the packages they install by default:

alembic==1.5.8
amqp==2.6.1
apache-airflow==2.0.2
apache-airflow-providers-amazon==1.4.0
apache-airflow-providers-celery==1.0.1
apache-airflow-providers-cncf-kubernetes==1.0.0
apache-airflow-providers-databricks==1.0.1
apache-airflow-providers-docker==1.2.0
apache-airflow-providers-ftp==1.0.1
apache-airflow-providers-http==1.1.1
apache-airflow-providers-imap==1.0.1
apache-airflow-providers-oracle==1.1.0
apache-airflow-providers-postgres==1.0.2
apache-airflow-providers-presto==1.0.2
apache-airflow-providers-sftp==1.2.0
apache-airflow-providers-slack==3.0.0
apache-airflow-providers-sqlite==1.0.2
apache-airflow-providers-ssh==1.3.0
apache-airflow-providers-tableau==1.0.0
apispec==3.3.2
argcomplete==1.12.3
attrs==20.3.0
Babel==2.9.1
bcrypt==3.2.0
billiard==3.6.4.0
bleach==4.1.0
blinker==1.4
boto3==1.9.199
botocore==1.12.253
cached-property==1.5.2
cachetools==4.2.4
cattrs==1.5.0
cchardet==2.1.7
celery==4.4.7
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
clickclick==20.10.2
colorama==0.4.4
colorlog==5.0.1
commonmark==0.9.1
connexion==2.7.0
croniter==0.3.37
cryptography==3.4.7
cx-Oracle==8.3.0
defusedxml==0.7.1
dill==0.3.2
Django==2.2.24
dnspython==1.16.0
docker==3.7.3
docker-pycreds==0.4.0
docutils==0.15.2
email-validator==1.1.2
Flask==1.1.2
Flask-AppBuilder==3.3.2
Flask-Babel==1.0.0
Flask-Caching==1.10.1
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.5.1
Flask-WTF==0.14.3
flower==0.9.7
google-auth==2.3.3
graphviz==0.16
gunicorn==19.10.0
humanize==3.4.1
idna==2.10
importlib-metadata==1.7.0
importlib-resources==1.5.0
inflection==0.3.1
iso8601==0.1.14
isodate==0.6.0
itsdangerous==1.1.0
Jinja2==2.11.3
jinjasql==0.1.8
jmespath==0.10.0
jsonschema==3.2.0
kombu==4.6.11
kubernetes==11.0.0
lazy-object-proxy==1.4.3
lockfile==0.12.2
Mako==1.1.4
Markdown==3.3.4
MarkupSafe==1.1.1
marshmallow==3.11.1
marshmallow-enum==1.5.1
marshmallow-oneofschema==2.1.0
marshmallow-sqlalchemy==0.23.1
natsort==7.1.1
numpy==1.21.0
oauthlib==3.1.1
openapi-schema-validator==0.1.5
openapi-spec-validator==0.3.0
packaging==21.3
pandas==1.3.4
paramiko==2.8.1
pendulum==2.1.2
presto-python-client==0.7.0
prison==0.1.3
prometheus-client==0.8.0
psutil==5.8.0
psycopg2==2.9.2
psycopg2-binary==2.7.7
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycurl==7.43.0.5
Pygments==2.8.1
PyJWT==1.7.1
PyNaCl==1.4.0
pyparsing==3.0.6
pyrsistent==0.17.3
pysftp==0.2.9
python-daemon==2.3.0
python-dateutil==2.8.1
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2021.1
pytzdata==2020.1
PyYAML==6.0
requests==2.25.1
requests-oauthlib==1.3.0
rich==9.2.0
rsa==4.7.2
s3transfer==0.2.1
sentry-sdk==0.19.4
setproctitle==1.2.2
six==1.15.0
slack-sdk==3.12.0
SQLAlchemy==1.3.24
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.37.0
sqlparse==0.4.2
sshtunnel==0.1.5
statsd==3.3.0
swagger-ui-bundle==0.0.8
tableauserverclient==0.17.0
tabulate==0.8.9
tenacity==6.2.0
termcolor==1.1.0
text-unidecode==1.3
tornado==6.1
tracer-utils==0.0.1.dev32+g26636a5.d20220107
typing-extensions==3.7.4.3
unicodecsv==0.14.1
urllib3==1.25.11
vine==1.3.0
watchtower==1.0.6
webencodings==0.5.1
websocket-client==1.2.1
Werkzeug==1.0.1
WTForms==2.3.3
zipp==3.4.1

3. I tried to install these on my local, on the same Python version they are using - Python 3.7.10
4. This resulted in a conflict itself:

ERROR: Cannot install -r requirements-direct.txt (line 4) and watchtower==1.0.6 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested watchtower==1.0.6
    apache-airflow-providers-amazon 1.4.0 depends on watchtower~=0.7.3

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

AWS should have used the constraints-3.7.txt file they mention in their documentation here https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-faqs.html#python-version but it doesn't look like it looking at installed packages.

How is it possible to install these conflicting packages and does the apache-airflow-providers-amazon package even work since it depends on watchtower~=0.7.3