Deployment Mode
Flink can be deployed in the following three modes:
- Session Mode
- Per-Job Mode
- Application Mode
The main difference between the above methods is
- Cluster life cycle and resource isolation guarantee
- Is the main method of the application executed on the client or on the cluster?
SessionMode (session mode)
Session mode assumes that there is an already running cluster and uses the resources of that cluster to execute any submitted applications. Applications executing in the same (session) cluster use and therefore compete for the same resources. The advantage of this is that you don't have to create a cluster for every submitted job. However, if one of the jobs behaves improperly or the taskmanager is closed, all jobs running on that taskmanager are affected by the failure. In addition to negatively affecting the job that causes the failure, this also means a potential large-scale recovery process. In addition, running multiple jobs in one cluster means that the load on the JobManager is increased, which is responsible for bookkeeping all jobs in the cluster.
Per-Job Mode
To provide better resource isolation guarantees, the Per-Job model uses available cluster manager frameworks (e.g. YARN,Kubernetes) Start the cluster for each submitted job. This cluster is only applicable to this job. After the job is completed, the cluster will be torn down and all attached resources (files, etc.) will be cleared. This provides better resource isolation, as improper work can only drag down its own taskmanager. Additionally, since there is one per job, it can spread bookkeeping work across multiple JobManagers. == For these reasons, the preferred work resource allocation model is the preferred mode for many production reasons. ==
Application Mode
In all the above modes, the main method of the application is executed on the client side.This process involves downloading dependencies for the application locally, performing main extracts the application representation (i.e., JobGraph) that the Flink runtime can understand, and shipping the dependencies and JobGraph to the cluster. This makes the client a large resource consumer, as it may require a lot of network bandwidth to download dependencies and ship binary files to the cluster, and requires CPU cycles to execute main. This problem can be more obvious when sharing clients across users.
Based on this, "Application Mode" will create a cluster for each submitted application, but this time, the main method of the application is executed on the JobManager. Each application creation cluster can be seen as creating a session cluster that is only shared between jobs for a particular application and being tear down when the application completes. With this architecture, the application pattern can provide the same resource isolation and load balancing as the job-by-job mode, but ensure the granularity of the entire application. Executing main on the JobManager saves the required CPU cycles and also saves the bandwidth required to download dependencies locally. Furthermore, since each application has only one JobManager, it can more evenly distribute the network load of the dependencies of the application in the cluster.
Notice:In application mode, main is executed on the cluster JobManager rather than on the client. This may have an impact on your code, for example, you must use the application's JobManager to access any paths registered in the environment using registerCachedFile.
Compared to job-by-job mode, application mode allows submission of applications containing multiple jobs. The order in which jobs are executed is not affected by the deployment mode, but by the calls to start the job. Using blocked execute can create a command, which will cause the execution of the "next" job to be delayed until the "this" job is completed. Using non-blocking executeAsync() causes the "next" job to start before the "this" job completes
Summarize
In session mode, clusterlife cycleNothing to do with the life cycle of any job running on the cluster, and resources are shared among all jobs. The "job-by-job" mode allocates a cluster for each submitted job, which requires higher costs, but this has better isolation guarantees because resources are not shared among jobs. In this case, the life cycle of the cluster will be bound to the life cycle of the job. Finally, application mode creates a session cluster for each application and executes the main method of the application on that cluster.
Finally, attach the corresponding cli of these three modes
- Session mode
./flink run -c .DailyTimeSliceActionPersonCountStatisticV2 -m yarn-cluster /home/wxcui/test/RealTimeProject-1.1.0-jar-with-dependencies.jar --brand dyc --flag dev --job_name dyc_DailyActionPersonCountSliceStatistic_dyc_test --kafka_topic mall_dyc_person_count --group_id wxcui001 --start_offset timestamp --offset_timestamp 1595174400000 -- PLAINTEXT --parallelism 8 --enable_checkpoint true
- Per-Job Mode (Just add -yd(detach mode) before -m yarn-cluster)
./flink run -c .DailyTimeSliceActionPersonCountStatisticV2 -yd -m yarn-cluster /home/wxcui/test/RealTimeProject-1.1. --brand dyc --flag dev --job_name dyc_DailyActionPersonCountSliceStatistic_dyc_test --kafka_topic mall_dyc_person_count --group_id wxcui001 --start_offset timestamp --offset_timestamp 1595174400000 -- PLAINTEXT --parallelism 8 --enable_checkpoint true
- Application Mode
./bin/flink run-application -t yarn-application ./examples/batch/