On the BI team I work in, we have about 20 ETL pipelines and other jobs. These are containerized and on AWS. We’re using one state file to manage the deployment of these. We’re using AWS Dynamo to store the state. We’re operationalizing a couple of forecasting models, containerizing them and getting them up to AWS. With a couple of more over the next year.
Having multiple simultaneous deployment attempts is one characteristic to consider. Not having our ML/DS deployments possibly mess up our production deployments is another.
Question: What is the best practice, or other key consideration, for when to decide to introduce another state file?