DATA SCIENCE
Snowflake Data Science Training Summary
I have enrolled in a private Snowflake Data Science Training. Let me list what I learned from it.
The notebook / 65 articles
Ideas into practice
Notes on LLMs, machine learning, data engineering, and the systems that make them work.
Showing 51–60 of 65 posts
AI ENGINEERING
AutoGen HttpClient
Now we can add our proxy config in our client, which will resolve connection issue.
SOFTWARE ENGINEERING
How to execute python modules
We can use internal runpy to execute different moduls in our project.
MACHINE LEARNING
Model Registry
Problem: How to introduce ml-based production/features to cross-functional teams.
DEVOPS
Setup Minikube
Set up Minikube on macOS, deploy a sample application, and submit a Spark job to a local Kubernetes cluster.
DATA ENGINEERING
Azure Data Factory (Data Flow)
Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my work.
DATA ENGINEERING
Spark Dataframe window function
A Scala reference for working with window functions in Spark SQL.
DATA ENGINEERING
Spark Optimization
Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to farthest:
DATA ENGINEERING
Spark SQL
Spark submit options and tuning notes covering executors, memory, parallelism, and resource allocation on YARN.
DATA ENGINEERING
Airflow
Airflow DAG examples for scheduling Python tasks, configuring retries and queues, and separating task implementations.