Data Engineering Podcast
OCTOBER 22, 2023
Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the development, delivery, and evolution of data products.
The Pragmatic Engineer
OCTOBER 24, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover two out of seven topics from today’s full issue on The Man Behind the Big Tech Comics. To get full issues twice a week, subscribe here.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
OCTOBER 25, 2023
1. Introduction 2. Six Steps for a Clean Data Warehouse 2.1. Understand the business 2.2. Make data easy to use with the appropriate data model 2.3. Good input data is necessary for a good data warehouse 2.4. Define Source of Truth (SOT) and trace its usage 2.5. Keep stakeholders in the loop for a more significant impact 2.6. Watch out for org-level red flags ?
Waitingforcode
OCTOBER 24, 2023
It's time to start the series covering Apache Spark 3.5.0 features. As the first topic I'm going to cover Structured Streaming which has got a lot of RocksDB improvements and some major API changes.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
KDnuggets
OCTOBER 26, 2023
No-code or low-code functionalities in data science have gained significant traction in recent years. These solutions are well-proven and matured, and they make data science more accessible to a wider range of people.
Snowflake
OCTOBER 23, 2023
Python’s popularity has more than doubled in the past decade¹ and it is quickly becoming the preferred language for development across machine learning, application development, pipelines, and more. One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Engineering at Meta
OCTOBER 24, 2023
Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs.
KDnuggets
OCTOBER 25, 2023
Machine Learning is one of the most exciting fields in computer science today. In this article, we will take a look at the five best yet free books to learn machine learning in 2023.
Confluent
OCTOBER 26, 2023
Read updates and improvements in Apache Flink 1.18, including dynamic fine-grained rescaling via REST API, Java 17 support, and faster rescaling & batch performance improvements.
ArcGIS
OCTOBER 26, 2023
In October 2023, elevation layers have been updated with high-res datasets of France, New Zealand, USA, Italy along with global bathymetry.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Tweag
OCTOBER 25, 2023
Tweag is excited to announce the first release of nixtract 0.1.0 ! This is our first step towards a broader effort to make Nix the best tool to tackle tomorrow’s challenges of the Software Supply Chain. In order to understand why we need nixtract , let me tell you about the “secret” value of Nixpkgs. Is it a bird? A plane? It’s a graph! The Nix language allows you to define the “recipe” to build anything into a package, like the sources and the steps to make the package, but also the dependencie
KDnuggets
OCTOBER 25, 2023
What are the top 5 cloud machine learning platforms in the market today. Our list will help provide some vital insights into which platform might best cater to your specific machine learning needs. See what KDnuggets recommends.
Engineering at Meta
OCTOBER 23, 2023
Meta has a very large monorepo, with many different programming languages. To optimize build and performance, we developed our own build system called Buck , which was first open-sourced in 2013. Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below are five interesting facts you might not have known about Buck2.
databricks
OCTOBER 25, 2023
Predictive Optimization intelligently optimizes your Lakehouse table data layouts for peak performance and cost-efficiency - without you needing to lift a finger.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Precisely
OCTOBER 23, 2023
When most people think of master data management, they first think of customers and products. This is logical, as the core mission of any company is to develop products and services, find the right customers, and consistently deliver excellence. But master data encompasses so much more than data about customers and products. It includes information about suppliers, employees, and target prospects.
Pinterest Engineering
OCTOBER 24, 2023
Each year, we host Makeathon, our annual internal version of a hackathon, where employees from across the business collaborate for three days to bring their dream passion projects to life. The ideas they pitch have a goal to improve our product, culture, internal processes or a combination of the three. This year, Makeathon was hosted from August 7–August 11.
KDnuggets
OCTOBER 27, 2023
This week on KDnuggets: Go from learning what large language models are to building and deploying LLM apps in 7 steps • Check this list of free books for learning Python, statistics, linear algebra, machine learning and deep learning • And much, much more!
Knowledge Hut
OCTOBER 23, 2023
Cybersecurity is a method of safeguarding networks and devices from external attacks. The cybersecurity trend shows a growing emphasis on protection, leading to an increased need for Cyber Security specialists. They are hired by businesses to secure secret information, preserve staff productivity, and boost customer trust in products and services. Cyber security is governed by the industry standard of confidentiality, integrity, and availability, or CIA.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
ThoughtSpot
OCTOBER 26, 2023
To our entire partner ecosystem, I want to personally thank each of you for your incredible contributions over the past year. Our partners play a vital role in driving ThoughtSpot’s mission of becoming a more fact-driven world. Together, we help organizations leverage AI and natural language search to discover insights and make data-driven decisions for their businesses.
Cloudera
OCTOBER 25, 2023
The world faces multiple environmental sustainability challenges — from the climate crisis and water scarcity to food production and urban resilience. Overcoming these hurdles offers opportunities for innovation through technology and artificial intelligence. That’s why Cloudera and AMD have partnered to host the Climate and Sustainability Hackathon.
KDnuggets
OCTOBER 22, 2023
The article delves into a comprehensive methodology that sheds light on how to accurately estimate the carbon footprint associated with AI applications. It explains the environmental impact of AI, a crucial consideration in today's world.
databricks
OCTOBER 24, 2023
"I can't think of anything that's been more powerful since the desktop computer." — Michael Carbin, Associate Professor, MIT, and Founding Advisor, MosaicML A.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Knowledge Hut
OCTOBER 27, 2023
A certified Six Sigma Black Belt expert is a professional who knows and can explain and implement the Six Sigma principles and philosophies. These include tools and supportive systems. A Black Belt professional must have impeccable leadership skills and understand team dynamics. They work in a collaborative manner to assign team members and give them roles and responsibilities.
ThoughtSpot
OCTOBER 27, 2023
Every business that analyzes their operational (or transactional) data needs to build a custom data pipeline involving several batch or streaming jobs to extract transactional data from relational databases , transform it, and load it into the data warehouse. In this post, we show how you can leverage Amazon Aurora zero-ETL integration with Amazon Redshift and ThoughtSpot for GenAI driven near real-time operational analytics.
KDnuggets
OCTOBER 25, 2023
Let’s take a closer look at Hybrid AI, how you can take advantage of it, and how Snapdragon brings hybrid AI to apps at the edge.
Snowflake
OCTOBER 24, 2023
Founded nearly 70 years ago, Werner Enterprises is a North American transportation and logistics leader that operates a fleet of almost 8,300 trucks and 30,000 trailers out of 16 terminals across the United States. The company generates a massive amount of data on the constantly changing, real-time location of each of its assets. Collecting and analyzing this geospatial data is vital for smart decision-making.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Knowledge Hut
OCTOBER 27, 2023
In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. To further amplify your skillset, consider enrolling in Programming training course to leverage online programming courses from expert trainers and grow with mentorship programs.
Netflix Tech
OCTOBER 27, 2023
How Netflix’s Container Platform Connects Linux Kernel Panics to Kubernetes Pods By Kyle Anderson With a recent effort to reduce customer (engineers, not end users) pain on our container platform Titus , I started investigating “orphaned” pods. There are pods that never got to finish and had to be garbage collected with no real satisfactory final status.
KDnuggets
OCTOBER 27, 2023
This article gives a high-level overview of how LLMs work and their attendant limitations with accessible explanations and anecdotes throughout the piece. We also present advice on how people can introduce them into their workflows.
Snowflake
OCTOBER 25, 2023
AI. It’s on everyone’s mind—and marketers are no exception. You’ve likely heard about it from co-workers, vendors and peers, and if you had a nickel for every AI mention you heard … well, you get the point. With the release of ChatGPT late last year, OpenAI supercharged the conversation around large language models (LLMs), marking 2023 as “the year of AI.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Expert insights. Personalized for you.
162 
Let's personalize your content