KDnuggets
MAY 5, 2022
This brief overview of the concept of Hypothesis Testing covers its classification in parametric and non-parametric tests, and when to use the most popular ones, including means, correlation, and distribution, in the case of one sample and two samples.
Cloudera
MAY 4, 2022
Artificial intelligence (AI) has been a focus for research for decades, but has only recently become truly viable. The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. AI has the potential to transform operations by improving three fundamental business requirements: process automation, decision-making based on data insights, and customer interaction.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
MAY 1, 2022
Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. Yotpo has been on a journey to evolve and scale their data platform to continue serving the needs of their organization as it increases the scale and sophistication of data usage. In this episode Doron Porat and Liran Yogev explain how they arrived at their current architecture, the capabilities that they are optimizing for, an
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
MAY 5, 2022
Artificial intelligence is not all that intelligent. While today’s AI can do some extraordinary things, the functionality underlying its accomplishments has very little to do with the way in which a human brain works to achieve the same tasks.
Cloudera
MAY 2, 2022
As digital transformation accelerates, and digital commerce increasingly becomes the dominant form of all commerce, regulators and governments around the world are recognizing the increased need for consumer protections and data protection measures. The European Union has been at the vanguard for some time (most recently having reached provisional agreement on the Digital Services Act ) but from Australia to Brazil , from South Africa to California (the rest of the US hasn’t quite caught on yet!
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
MAY 1, 2022
Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. Matillion was an early innovator of that approach and in this episode CTO Ed Thompson explains how they have evolved the platform to keep pace with the rapidly changing ecosystem. He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the ma
KDnuggets
MAY 4, 2022
Check out all the necessary steps to successfully structure your data science projects leveraging data science templates.
Cloudera
MAY 5, 2022
Fraud, waste, and abuse (FWA) in government is a constant, multi-billion dollar issue that challenges agency leaders at all levels and across all sectors, from healthcare to education to taxation to Social Security. The scope and scale of public spending — federal outlays alone were approximately $6.6 trillion in fiscal year 2020 according to the Congressional Budget Office — make FWA an inherently difficult problem to solve.
Confluent
MAY 3, 2022
The recent release of Confluent Cloud and Confluent Platform 7.0 introduced the ability to easily remove Apache Kafka® brokers and shrink your Confluent Server cluster with just a single command. […].
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
DataKitchen
MAY 2, 2022
The post Podcast: Storytime for DataOps first appeared on DataKitchen.
KDnuggets
MAY 4, 2022
In this article, we’ll look at what Convolutional Neural Networks are and how they work.
Cloudera
MAY 3, 2022
This month we are proud to spotlight Lynne Montalbo, senior business systems analyst from Santa Clara, California, who volunteers as a professional development mentor with Braven. Braven’s mission is to empower promising, underrepresented young people—first-generation college students, students from low-income backgrounds, and students of color—with the skills, confidence, experiences, and networks necessary to transition from college to strong first jobs, which lead to meaningful careers and li
Confluent
MAY 4, 2022
It is no exaggeration that a lot is going wrong in commercial buildings today. The building and construction sector consumes 36% of global final energy and accounts for almost 40% […].
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Monte Carlo
MAY 6, 2022
I’m honored to share that Monte Carlo was just named a Best Place to Work in the Bay Area for 2022 by the San Francisco Business Times and the Silicon Valley Business Journal, placing 6th in the small business category. This recognition is especially meaningful to our leadership team because the results are based directly on employee feedback, collected anonymously from a third-party researcher.
KDnuggets
MAY 2, 2022
Learn Python programming, statistics, and machine learning online from one of the world’s top universities.
Teradata
MAY 6, 2022
The value provided by a powerful data fabric is key for a successful digital transformation. Find out why.
Rockset
MAY 5, 2022
I spent the spring of my junior year interning at Rockset , and it couldn’t have been a better decision. When I first arrived at the office on a sunny day in San Mateo, I had no idea that I was about to meet so many systems engineering gurus, or that I was about to consume immensely good food from the festive neighboring streets. Working with my talented and resourceful mentor, Ben (Software Engineer, Systems), I’ve been able to learn more than I ever thought I could in three months!
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
dbt Developer Hub
MAY 5, 2022
Continuous Integration (CI) sets the system up to test everyone’s pull request before merging. Continuous Deployment (CD) deploys each approved change to production. “Slim CI” refers to running/testing only the changed code, thereby saving compute. In summary, CI/CD automates dbt pipeline testing and deployment. dbt Cloud , a much beloved method of dbt deployment, supports GitHub- and Gitlab-based CI/CD out of the box.
KDnuggets
MAY 5, 2022
The free book is a combination of SQL cheat sheets and practical database examples. It provided bite-size information about every SQL function and attribute with coding samples.
Teradata
MAY 3, 2022
Less than 20% of retailers on track to meet sustainability pledges. Granular, integrated data is the key to move from reporting to action. Read about our framework for profitable sustainability.
Rockset
MAY 3, 2022
There are two major problems with distributed data systems. The second is out-of-order messages, the first is duplicate messages, the third is off-by-one errors, and the first is duplicate messages. This joke inspired Rockset to confront the data duplication issue through a process we call deduplication. As data systems become more complex and the number of systems in a stack increases, data deduplication becomes more challenging.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Eventbrite Engineering
MAY 2, 2022
Background At Eventbrite, we identified in our 3-year technical vision that one of our goals is to enable autonomous dev teams to own their code and architecture so as to be able to deliver reliable, high quality and cost effective solutions to our customers. However, this autonomy does not mean that our team has to … Continue reading "Packaging generated code from protobuf files for gRPC Services" The post Packaging generated code from protobuf files for gRPC Services appeared first on E
KDnuggets
MAY 2, 2022
These are the six top paying companies for data scientists. I’ve looked at absolute salary, but I’ll fill you in on other factors you should consider as well when it comes to picking a data science job for money.
Elder Research
MAY 2, 2022
The post Why Does Elder Research Need a Chief Scientist Committee? appeared first on Elder Research.
Pipeline Data Engineering
MAY 2, 2022
In this interview series we’ll share some of the stories that Daniel and I get to watch unfold at Pipeline Academy. Check out what our graduates have to say about the course, how they’ve tackled its challenges and what they are doing now with their new data engineering superpowers. Peter: Can I ask you to please introduce yourself to the readers of Pipeline Academy’s blog?
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
dbt Developer Hub
MAY 2, 2022
dbt Cloud is a hosted service that many organizations use for their dbt deployments. Among other things, it provides an interface for creating and managing deployment jobs. When triggered (e.g., cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. dbt Cloud provides a REST API for managing jobs, run artifacts and other dbt Cloud resources.
KDnuggets
MAY 5, 2022
After learning the basics of data science, you can start to work on real-world problems. But how do you showcase your work? In this article, we are going to learn a unique way to create a data science portfolio.
Ripple Engineering
MAY 2, 2022
In part one of our two-part series, we heard from RippleX engineers that are ideating, creating and executing on new applications using cutting-edge blockchain and crypto technology. Now, we’ll explore how the RippleNet engineering team is building the foundational payments infrastructure on the XRP Ledger that will allow value to move as easily as information moves today.
DataKitchen
MAY 2, 2022
The post DataKitchen Noted For DataOps Thought LeaderShip first appeared on DataKitchen.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content