KDnuggets
OCTOBER 14, 2022
Leveraging sparse matrix representations for your data when appropriate can spare you memory storage. Have a look at the reasons why, see how to create sparse matrices in with Python, and compare the memory requirements for standard and sparse representations of the same data.
The Pragmatic Engineer
OCTOBER 10, 2022
Part of this article was originally published in The Scoop #27 , for subscribers of The Pragmatic Engineer Newsletter last week. I decided to publish this section for everyone to read after the Business Insider article claiming that 15% of Facebook employees - 12,000 people - may lose their jobs started to spread within the media. The Business Insider article was not specific to software engineers but still spread heavily within tech circles.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Teradata
OCTOBER 12, 2022
ClearScape Analytics provides robust functionality giving people across the organization the ability to efficiently execute their roles in the analytics process on a common platform.
Data Engineering Podcast
OCTOBER 9, 2022
Summary For any business that wants to stay in operation, the most important thing they can do is understand their customers. American Express has invested substantial time and effort in their Customer 360 product to achieve that understanding. In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern sy
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
OCTOBER 14, 2022
Something is missing in artificial intelligence – trust.
DataKitchen
OCTOBER 14, 2022
The post Podcast: Scaling DataOps first appeared on DataKitchen.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
OCTOBER 9, 2022
Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use. In order to make the data lakehouse available to a wider audience the team at Iomete built an all-in-one service that handles management and integration of the various technolo
KDnuggets
OCTOBER 14, 2022
Check out this free ebook covering the fundamentals of mathematics for machine learning, as well as its companion website of exercises and Jupyter notebooks.
Cloudera
OCTOBER 11, 2022
A recent VentureBeat article , “4 AI trends: It’s all about scale in 2022 (so far),” highlighted the importance of scalability. I recommend you read the entire piece, but to me the key takeaway – AI at scale isn’t magic, it’s data – is reminiscent of the 1992 presidential election, when political consultant James Carville succinctly summarized the key to winning – “it’s the economy”.
AltexSoft
OCTOBER 13, 2022
Take a look at the featured image above. Beautiful, isn’t it? The interesting thing is, it isn’t a painting drawn by some famous artist, nor is it a photo taken by a satellite. The image you see has been generated with the help of Midjourney — a proprietary artificial intelligence program that creates pictures from textual descriptions. Neural nets can create images, video, and audio content that not every person can.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Netflix Tech
OCTOBER 12, 2022
A conversation between Travis Brooks, Netflix Product Manager for Experimentation Platform, and George Khachatryan, OfferFit CEO Note: I’ve known George for a little while now, and as we’ve talked a lot about the philosophy of experimentation, he kindly invited me to their office (virtually) for their virtual speaker series. We had a fun conversation with his team, and we realized that some parts of it might make a good blog post as well.
KDnuggets
OCTOBER 12, 2022
Do you want to learn PyTorch for machine learning and deep learning? Check out this 24 hour long video course with accompanying notes and courseware for free. Did I mention it's free?
Elder Research
OCTOBER 13, 2022
The post Machine Learning Engineers & Operations: Where DevOps Meet Data Science appeared first on Elder Research.
Rockset
OCTOBER 13, 2022
The climate changed and everyone quickly noticed how expensive Snowflake is. How Snowflake fails - Benn Stancil Why is Snowflake so expensive - Stas Sajin Snowflake performance challenges - Slim Baltagi Ok, so Snowflake is expensive. But what do I do about it? Avoid frequent updates Optimize for cost-per-query with apps running 24x7 Tune slow queries Reduce auto-suspend to 1 or 2 minutes Build Snowflake chargeback dashboards Try third-party cost analyzers Set resource monitors and spend threshol
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Propel Data
OCTOBER 13, 2022
Developers looking to learn about event-driven architectures.
KDnuggets
OCTOBER 11, 2022
A Data Science Enablement Team consists of people from various departments like marketing, sales, product development, etc. They are responsible for providing the necessary tools and resources to help the data scientists do their job more efficiently.
Picnic Engineering
OCTOBER 13, 2022
We’re excited to announce that Picnic’s Error Prone Support project is now open-source! Last week, we already shared an in-depth overview of how Picnic has adopted Google’s Error Prone static analysis tool for Java code. In short, it allows us to: Improve the consistency and quality of our Java codebases. Introduce custom checks for code (anti-)patterns we value.
U-Next
OCTOBER 13, 2022
Introduction . The Product Manager is the visionary and leader of the product, who leads a team of designers, engineers, and other stakeholders to build a great product interaction design. It is estimated that companies could increase their profits by more than 34 percent when their Product Manager is “fully optimized.” . The role of a Product Manager goes beyond simply managing requirements and specifications.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
phData: Data Engineering
OCTOBER 12, 2022
So you’ve heard all the talk around dbt , but now you’re working to determine if you should go with dbt Core or dbt Cloud and you’re wanting to know what advantages dbt Cloud has over the free dbt Core offering. Upon a quick trial and look at dbt Cloud, the primary things you might notice are the IDE as well as the ease of managing deployments. However, dbt Cloud offers you much more than that.
KDnuggets
OCTOBER 11, 2022
This article serves as a beginner’s guide to web scraping using Python and looks at the different frameworks and methods you can use, outlined in simple terms.
Striim
OCTOBER 12, 2022
I am excited to share with you that Striim is a proud participant in the Microsoft Intelligent Data Platform partner ecosystem as announced at Microsoft Ignite 2022. We have a history of working with Microsoft to help provide our mutual customers with access to enhanced data insights in real time, allowing them to make decisions the moment data is created.
Lyft Engineering
OCTOBER 12, 2022
Pain Points of Traditional Automated UI Tests Creating a great modern-day software product requires a shift-left approach to testing by ensuring faster, more frequent, and earlier testing. Shift-left testing is an approach to software testing and system testing in which testing is performed earlier in the lifecycle (i.e., moved left on the project timeline).
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Pinterest Engineering
OCTOBER 12, 2022
Khubi Shah | (former) Software Engineer Intern, Shopping Content Mining This summer, I had the incredible opportunity to intern at the one and only Pinterest from the new engineering hub in Toronto! I am a final year undergraduate student from the University of Waterloo, majoring in Computer Science with an AI specialization. Growing up, Pinterest was always my go-to social media platform, as it inspired me with new ideas for food, fashion, design, or anything creative!
KDnuggets
OCTOBER 13, 2022
In NLP we must find a way to represent our data (a series of texts) to our systems (e.g. a text classifier). As Yoav Goldberg asks, "How can we encode such categorical data in a way which is amenable for us by a statistical classifier?" Enter the word vector.
DataKitchen
OCTOBER 12, 2022
Can you draw a map of all the paths data takes from source systems to production insight delivery? How many tools, technologies, configurations, and paths do your data take during its production process? What is the ‘run-time lineage’ of data in your organization? The post Map and Monitor Your Data Journey first appeared on DataKitchen.
WeCloudData
OCTOBER 11, 2022
Objectives This tutorial is one part of a containers series of tutorials that will walk the reader through installation of tools that can run applications in containers. By the end of these tutorials the reader will be able to Install services (container engines) that can run containers using tools such as LXD/LXC, Docker, or Podman. […] The post Install and Run Containers on Linux Virtual Machines – LXD/LXC appeared first on WeCloudData.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Rockset
OCTOBER 11, 2022
Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. The flow of data often involves complex ETL tooling as well as self-managing integrations to ensure that high volume writes, including updates and deletes, do not rack up CPU or impact performance of the end application.
KDnuggets
OCTOBER 10, 2022
What does a day in the life as a machine learning engineer look like for you?
Monte Carlo
OCTOBER 11, 2022
Precise endeavors must be done to exacting standards in clean environments. Surgeons scrub in, rocket scientists work in clean rooms, and data scientists…well we try our best. We’ve all heard the platitude, “garbage in, garbage out,” so we spend most of our time doing the most tedious part of the job: data cleaning. Unfortunately, no matter how hard we scrub, poor data quality is often too pervasive and invasive for a quick shower.
WeCloudData
OCTOBER 11, 2022
Objectives This tutorial is one part of a containers series of tutorials that will walk the reader through installation of tools that can run applications in containers. By the end of these tutorials the reader will be able to Install services (container engines) that can run containers using tools such as LXD/LXC, Docker, or Podman. Launch simple applications packaged in containers from template container images.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content