助力工业物联网：工业大数据服务域上的Shell调度系统测试（3）
发布时间
阅读量：
阅读量
本课程体系不仅包含适合初学者入门的零基础学习资源，同时也为具备三年以上经验的技术人员提供了深入学习和提升的进阶内容，几乎覆盖了大数据领域95%以上的核心知识点，形成了完整的知识架构。
鉴于资料总量较大，此处仅展示部分内容的目录截图，完整资料包括大型企业面试经验、学习笔记、源代码讲解文档、实战项目案例、课程大纲与学习路径图、配套教学视频，并将根据实际情况持续进行更新。
```
# 导入模块
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
from datetime import timedelta
# 设置默认参数
default_args = {
'owner': 'airflow',
'email': ['airflow@example.com'],
'email\_on\_failure': True,
'email\_on\_retry': True,
'retries': 1,
'retry\_delay': timedelta(minutes=1),
}
# 定义DAG对象
dag = DAG(
'first\_airflow\_dag',
default_args=default_args,
description='first airflow task DAG',
schedule_interval=timedelta(days=1),
start_date=days_ago(1),
tags=['itcast\_bash'],
)
# 创建任务实例
run_bash_task = BashOperator(
task_id='first\_bashoperator\_task',
bash_command='echo "hello airflow"',
dag=dag,
)
# 执行任务流程
run_bash_task - 工作中使用bashOperator
```bash_command='sh xxxx.sh'
- xxxx.sh：根据需求
* Linux命令
* hive -f
* spark-sql -f
* spark-submit python | jar
+ **提交**
```
执行名为first_bash_operator.py的Python脚本文件+ **查看**
+ 执行
*
**总结** * 完成对Shell命令执行流程的测试安排
## 知识点08：依赖调度测试
**目标** ：达成AirFlow的依赖调度测试
**实施** *
**需求** ：借助BashOperator来安排多个Task的执行流程，并建立相应的依赖关联
**代码** * 创建cd /root/airflow/dags
vim second_bash_operator.py* 研制# import
from datetime import timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
# define args
default_args = {
'owner': 'airflow',
'email': ['airflow@example.com'],
'email\_on\_failure': True,
'email\_on\_retry': True,
'retries': 1,
'retry\_delay': timedelta(minutes=1),
}
# define dag
dag = DAG(
'second\_airflow\_dag',
default_args=default_args,
description='first airflow task DAG',
schedule_interval=timedelta(days=1),
start_date=days_ago(1),
tags=['itcast\_bash'],
)
# define task1
say_hello_task = BashOperator(
task_id='say\_hello\_task',
bash_command='echo "start task"',
dag=dag,
)
# define task2
print_date_format_task2 = BashOperator(
task_id='print\_date\_format\_task2',
bash_command='date +"%F %T"',
dag=dag,
)
# define task3
print_date_format_task3 = BashOperator(
task_id='print\_date\_format\_task3',
bash_command='date +"%F %T"',
dag=dag,
)
# define task4
end_task4 = BashOperator(
task_id='end\_task',
bash_command='echo "end task"',
dag=dag,
)
say_hello_task >> [print_date_format_task2,print_date_format_task3] >> end_task4*
**提交**python second_bash_operator.py* **查阅**
**小结** * 完成对AirFlow依赖调度功能的测试验证
## 知识点09：Python调度测试
**目标** ：**达成Python代码的调度测试**
**实施** *
**需求** ：安排Python代码Task的执行
**代码** * 生成cd /root/airflow/dags
vim python_etl_airflow.py* 研制# import package
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
import json
# define args
default_args = {
'owner': 'airflow',
}
# define the dag
with DAG(
'python\_etl\_dag',
default_args=default_args,
description='DATA ETL DAG',
schedule_interval=None,
start_date=days_ago(2),
tags=['itcast'],
) as dag:
# function1
def extract(\*\*kwargs):
ti = kwargs['ti']
data_string = '{"1001": 301.27, "1002": 433.21, "1003": 502.22, "1004": 606.65, "1005": 777.03}'
ti.xcom_push('order\_data', data_string)
# function2
def transform(\*\*kwargs):
ti = kwargs['ti']
extract_data_string = ti.xcom_pull(task_ids='extract', key='order\_data')
order_data = json.loads(extract_data_string)
total_order_value = 0
for value in order_data.values():
total_order_value += value
total_value = {"total\_order\_value": total_order_value}
total_value_json_string = json.dumps(total_value)
ti.xcom_push('total\_order\_value', total_value_json_string)
# function3
def load(\*\*kwargs):
ti = kwargs['ti']
total_value_string = ti.xcom_pull(task_ids='transform', key='total\_order\_value')
total_order_value = json.loads(total_value_string)
print(total_order_value)
# task1
extract_task = PythonOperator(
task_id='extract',
python_callable=extract,
)
extract_task.doc_md = """\
#### Extract task
A simple Extract task to get data ready for the rest of the data pipeline.
In this case, getting data is simulated by reading from a hardcoded JSON string.
This data is then put into xcom, so that it can be processed by the next task.
"""
# task2
transform_task = PythonOperator(
task_id='transform',
python_callable=transform,
)
transform_task.doc_md = """\
#### Transform task
A simple Transform task which takes in the collection of order data from xcom
and computes the total order value.
This computed value is then put into xcom, so that it can be processed by the next task.
"""
# task3
load_task = PythonOperator(
task_id='load',
python_callable=load,
)
load_task.doc_md = """\
#### Load task
A simple Load task which takes in the result of the Transform task, by reading it
from xcom and instead of saving it to end user review, just prints it out.
"""
# run
extract_task >> transform_task >> load_taskpython python_etl_airflow.py* **查阅**
**小结** * 完成对Python代码执行调度的测试工作
## 知识点10：Oracle与MySQL调度方法
* **目标** ：掌握Oracle与MySQL的调度机制
* **实施**
**既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！**
**由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新**
**[需要这份系统化资料的朋友，可以戳这里获取]()**
UEUd-1715732435139)]
[外链图片转存中…(img-hN6oRedZ-1715732435139)]
[外链图片转存中…(img-bcyPGgRl-1715732435139)]
**既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上大数据知识点，真正体系化！**
**由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新**
**[需要这份系统化资料的朋友，可以戳这里获取]()**全部评论 (0)
还没有任何评论哟~