Dr. Daniel Sokolowski

Dr. Daniel Sokolowski
prg-grp@d.sokolowski.org
dsoko
0000-0003-2911-8304
Soko2D
DSoko2
https://dsoko.de

I am enthusiastic about software engineering and programming language technology for modern distributed systems and cloud applications. I focus on advancing Infrastructure as Code (IaC) to achieve reliable deployments for modern DevOps organizations. For more information, please consult my private webpage.

I joined the Programming Group at TU Darmstadt in Mai 2019 and moved to the University of St. Gallen in November 2021, where I graduate with a PhD in September 2024. Before I was part of the Software Factory 4.0 project at the Technical University of Darmstadt.

Publications

TSE
ProTI

Automated Infrastructure as Code Program Testing

Daniel Sokolowski, David Spielmann, Guido Salvaneschi

IEEE Transactions on Software Engineering 50 (6), 2024

Abstract PDF Supp Code Website

Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort. To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT in ProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6081 programs from GitHub and artificial benchmarks shows that ProTI can directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.
MSR

The PIPr Dataset of Public Infrastructure as Code Programs

Daniel Sokolowski, David Spielmann, Guido Salvaneschi

In Proceedings of the IEEE/ACM 21st International Conference on Mining Software Repositories, MSR, 2024

Abstract PDF

With Programming Languages Infrastructure as Code (PL-IaC), developers implement IaC programs in popular imperative programming languages like Python and Typescript. Such programs generate the declarative target state of the deployment, i.e., they describe what to set up, not how to set it up. Despite the popularity of PL-IaC, which has grown more than ten times from 2020 to 2023, we know little about how developers apply it and how IaC programs differ from other software. Such knowledge is essential to effectively use existing software engineering techniques and develop new ones for PL-IaC. To shed light on PL-IaC in practice, we present PIPr, the first systematic PL-IaC dataset. PIPr is based on 37712 public IaC programs on GitHub from August 2022 and includes initial analyses, assessing the programming languages, testing techniques, and licenses of the IaC programs. Beyond the metadata and analysis results of all IaC programs, PIPr contains the code of all 15504 IaC programs whose licenses permit redistribution. PIPr sets the ground for future in-depth investigations on PL-IaC in practice.
ICSE Companion
ProTI

Unleashing the Giants: Enabling Advanced Testing for Infrastructure as Code

Daniel Sokolowski, David Spielmann, Guido Salvaneschi

In Companion Proceedings of the 46th International Conference on Software Engineering, ICSE Companion, 2024

Abstract PDF

Infrastructure as Code (IaC) programs are written in imperative programming languages like Python or TypeScript while declaratively defining the target state of software deployments, which the IaC solution then sets up, e.g., Pulumi and AWS CDK. Through a repository mining study and analysis, we noticed that testing IaC programs poses a dilemma: current techniques are either slow and expensive or require prohibitively high development effort. To solve this issue, we introduce Automated Configuration Testing (ACT), enabling efficient testing with low development effort. ACT automates the tedious aspects of unit testing IaC programs and is extensible through a plugin system for test generators and oracles. ACT is already effective with simple type-based plugins, and leveraging existing giants, i.e., advanced test generation and oracle techniques, in new plugins will further boost its effectiveness.
ProTI
µs

Reliable infrastructure as Code for Decentralized Organizations

Daniel Sokolowski

Doctoral dissertation, Universität St. Gallen, 2024

Abstract PDF

IT must be reliable for organizations to thrive and quickly adaptable for their swift reaction to their environment. Agility is vital, and DevOps achieves these goals by empowering independent cross-functional teams in decentralized organizations and automating the entire software pipeline. Infrastructure as Code (IaC) is the critical tool to automate software operations, including infrastructure provisioning, application deployment, and configuration. Beyond simple IaC scripts, developers implement IaC programs in programming languages like TypeScript and Python. Such IaC programs are software, and their reliability is crucial to the functionality and security of the deployed systems. Still, techniques for the rapid development of reliable IaC programs are missing, limiting organizations’ agility. Specifically, developers lack automation for deployment coordination and updating and quality assurance tools for, e.g., testing and verification.
We surveyed 134 IT professionals, finding that coordination across deployments is commonly needed and often requires manual coordination, even though IT professionals believe automated coordination yields better agility. However, automated approaches are centralized, limiting team independence and agility in decentralized organizations. To solve this issue, we propose automating coordination across deployments in a decentralized fashion through μs ([mju:z] “muse”), a novel IaC solution. With μs, teams have separate IaC programs, which express and jointly automate the correct order of operations across deployments. We further show how μs enables safe updating through IaC programs, preventing updates from breaking distributed transactions or workflows.
Beyond automating the coordination of IaC programs, we address the reliability of IaC program code. To unblock studies, we built a dataset of 37 712 public IaC programs. In initial analyses, only a vanishing fraction implements tests. We identified that available testing techniques are either slow and resource-intensive or require excessive development effort. To solve this dilemma, we propose ACT, an extensible automated unit testing approach that enables testing IaC programs quickly in hundreds of configurations, often without writing additional testing code.
This dissertation studies the coordination and testing of IaC programs. Empirically motivated, we present μs for safe deployment coordination and updating in decentralized setups and ACT for efficient testing of IaC programs. Our contributions nurture future research and reliable deployments in decentralized organizations, ensuring agility.
CONFLANG
ProTI

Creed for Speed: Comprehensive Infrastructure as Code Testing

Daniel Sokolowski, David Spielmann, Guido Salvaneschi

Presentation at the 2nd Workshop on Configuration Languages (CONFLANG), 2023

Abstract PDF

With Programming Languages Infrastructure as Code (PL-IaC), developers implement imperative IaC programs in one of many general-purpose programming languages, e.g., TypeScript, Python, or Go, to declaratively describe deployments. Using these languages provides access to quality assurance techniques and tools developed for traditional software; however, programmers routinely rely on prohibitively slow integration testing – if they test at all. As a result, even simple bugs are found late, tremendously slowing down the development process.
To improve the velocity of PL-IaC development, we propose ProTI, an automated unit testing approach that quickly tests PL-IaC programs in many different configurations. ProTI mocks all cloud resources, replacing them with pluggable oracles that validate all resources configurations and a generator for realistic test inputs. We implemented ProTI for Pulumi TypeScript with simple generator and oracle plugins. Our experience of testing with ProTI encourages the exploration of more sophisticated oracles and generators, leading to the early detection of more bugs. ProTI enables programmers to rapidly prototype, explore, and plug in new oracles and generators for efficient PL-IaC program testing.
SPLASH Companion
ProTI

Extensible Testing for Infrastructure as Code

David Spielmann, Daniel Sokolowski, Guido Salvaneschi

In Companion Proceedings of the 2023 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion, 2023

Abstract PDF

Developers automate deployments with Programming Languages Infrastructure as Code (PL-IaC) by implementing IaC programs in popular languages like TypeScript and Python. Yet, systematic testing—well established for high-velocity software development—is rarely applied to IaC programs because IaC testing techniques are either slow or require extensive development effort. To solve this dilemma, we develop ProTI, a novel IaC unit testing approach, and implement it for Pulumi TypeScript. Our preliminary experiments with simple type-based test case generators and oracles show that ProTI can find bugs reliably in a short time, often without writing any additional testing code. ProTI’s extensible plugin architecture allows combining, adopting, and experimenting with new approaches, opening the discussion about novel generators and oracles for efficient IaC testing.
FIST
ProTI

Towards Reliable Infrastructure as Code

Daniel Sokolowski, Guido Salvaneschi

In Companion Proceedings of 2023 IEEE 20th International Conference on Software Architecture, ICSA Companion, 2023

Abstract PDF

Modern Infrastructure as Code (IaC) programs are increasingly complex and much closer to traditional software than to simple configuration scripts. Their reliability is crucial because their failure prevents the deployment of applications, and incorrect behavior can introduce malfunction and severe security issues. Yet, software engineering tools to develop reliable programs, such as testing and verification, are barely used in IaC. In fact, we observed that developers mainly rely on integration testing, a slow and expensive practice that can increase confidence in end-to-end functionality but is infeasible to systematically test IaC programs in various configurations—which is required to ensure robustness. On the other hand, fast testing techniques, such as unit testing, are cumbersome with IaC programs because, today, they require significant coding overhead while only providing limited confidence.
To solve this issue, we envision the automated testing tool ProTI, reducing the manual overhead and boosting confidence in the test results. ProTI embraces modern unit testing techniques to test IaC programs in many different configurations. Out of the box, ProTI is a fuzzer for Pulumi TypeScript IaC programs, randomly testing the program in many different configurations for termination, configuration correctness, and existing policy compliance. Then developers can add specifications to their program to guide random-based value generation, test additional properties, and add further mocking, making ProTI a property-based testing tool. Lastly, we aim at automatically verifying IaC-specific properties, e.g., access paths between resources.
IEEE Software
µs

Decentralizing Infrastructure as Code

Daniel Sokolowski, Pascal Weisenburger, Guido Salvaneschi

IEEE Software 40 (1), 2023

Abstract PDF

Infrastructure as Code (IaC) automates deployments for single teams, falling short of decentralized deployments across groups. We need mature IaC solutions that embrace and consolidate software engineering principles to enable testing and automation advances for decentralized organizations.
ESEC/FSE DS
µs

Infrastructure as Code for Dynamic Deployments

Daniel Sokolowski

In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE, 2022

Abstract PDF

Modern DevOps organizations require a high degree of automation to achieve software stability at frequent changes. Further, there is a need for flexible, timely reconfiguration of the infrastructure, e.g., to use pay-per-use infrastructure efficiently based on application load. Infrastructure as Code (IaC) is the DevOps tool to automate infrastructure. However, modern static IaC solutions only support infrastructures that are deployed and do not change afterward. To implement infrastructures that change dynamically over time, static IaC programs have to be (updated and) re-run, e.g., in a CI/CD pipeline, or configure an external orchestrator that implements the dynamic behavior, e.g., an autoscaler or Kubernetes operator. Both do not capture the dynamic behavior in the IaC program and prevent analyzing and testing the infrastructure configuration jointly with its dynamic behavior.
To fill this gap, we envision dynamic IaC, which augments static IaC with the ability to define dynamic behavior within the IaC program. In contrast to static IaC programs, dynamic IaC programs run continuously. They re-evaluate program parts that depend on external signals when these change and automatically adjust the infrastructure accordingly. We implement DIaC as the first dynamic IaC solution and demonstrate it in two realistic use cases of broader relevance. With dynamic IaC, ensuring the program’s correctness is even harder than for static IaC because programs may define many target configurations in contrast to only a few. However, for this reason, it is also more critical. To solve this issue, we propose automated, specialized property-based testing for IaC programs and implement it in ProTI.
ICSE

Change Is the Only Constant: Dynamic Updates for Workflows

Daniel Sokolowski, Pascal Weisenburger, Guido Salvaneschi

In Proceedings of the 44th International Conference on Software Engineering, ICSE, 2022

Abstract PDF Supp

Software systems must be updated regularly to address changing requirements and urgent issues like security-related bugs. Traditionally, updates are performed by shutting down the system to replace certain components. In modern software organizations, updates are increasingly frequent—up to multiple times per day—hence, shutting down the entire system is unacceptable. Safe dynamic software updating (DSU) enables component updates while the system is running by determining when the update can occur without causing errors. Safe DSU is crucial, especially for long-running or frequently executed asynchronous transactions (workflows), e.g., user-interactive sessions and order fulfillment processes. Unfortunately, previous research is limited to synchronous transaction models and does not address this case.
In this work, we propose a unified model for safe DSU in workflows. We discuss how state-of-the-art DSU solutions fit into this model and show that they incur significant overhead. To improve the performance, we introduce Essential Safety, a novel safe DSU approach that leverages the notion of non-essential changes, i.e., semantics preserving updates. In 106 realistic collaborative BPMN workflows, Essential Safety reduces the delay of workflow completions, on average, by 47.8% compared to the state of the art. We show that the distinction of essential and non-essential changes plays a crucial role in this reduction and that, as suggested in literature, non-essential changes are frequent: at least 60% and often more than 90% of systems’ updates in eight monorepos we analyze.
ESEC/FSE
µs

Automating Serverless Deployments for DevOps Organizations

Daniel Sokolowski, Pascal Weisenburger, Guido Salvaneschi

In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE, 2021

Abstract PDF Supp Code

DevOps unifies software development and operations in cross-functional teams to improve software delivery and operations (SDO) performance. Ideally, cross-functional DevOps teams independently deploy their services, but the correct operation of a service often demands other services, requiring coordination to ensure the correct deployment order. This issue is currently solved either with a central deployment or manual out-of-band communication across teams, e.g., via phone, chat, or email. Unfortunately, both contradict the independence of teams, hindering SDO performance – the reason why DevOps is adopted in the first place.
In this work, we conduct a study on 73 IT professionals, showing that, in practice, they resort to manual coordination for correct deployments even if they expect better SDO performance with fully automated approaches. To address this issue, we propose μs ([mju:z] “muse”), a novel IaC system automating deployment coordination in a fully decentralized fashion, still retaining compatibility with DevOps practice – in contrast to today’s solutions. We implement µs, demonstrate that it effectively enables automated coordination, introduces negligible definition overhead, has no performance overhead, and is broadly applicable, as shown by the migration of 64 third-party IaC projects.
ESEC/FSE DS
µs

Deployment Coordination for Cross-Functional DevOps Teams

Daniel Sokolowski

In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE, 2021

Abstract PDF

Software stability and reliability are the core concerns of DevOps. They are improved by tightening the collaboration between developers and operators in cross-functional teams on the one hand and by automating operations through continuous integration (CI) and infrastructure as code (IaC) on the other hand. Ideally, teams in DevOps are fully independent. Still, their applications often depend on each other in practice, requiring them to coordinate their deployment through centralization or manual coordination.
With this work, we propose and implement the novel IaC solution µs ([mju:z] “muse”), which automates deployment coordination in a decentralized fashion. µs is the first approach that is compatible with the DevOps goals as it enables truly independent operations of the DevOps teams. We define our research problem through a questionnaire survey with IT professionals and evaluate the solution by comparing it to other modern IaC approaches, assessing its performance, and applying it to existing IaC programs.
AISTA

Impact of Programming Languages on Machine Learning Bugs

Sebastian Sztwiertnia, Maximilian Grübel, Amine Chouchane, Daniel Sokolowski, Krishna Narasimhan, Mira Mezini

In Proceedings of the 1st ACM International Workshop on AI and Software Testing/Analysis, AISTA, 2021

Abstract PDF

Machine learning (ML) is on the rise to be ubiquitous in modern software. Still, its use is challenging for software developers. So far, research has focused on the ML libraries to find and mitigate these challenges. However, there is initial evidence that programming languages also add to the challenges, identifiable in different distributions of bugs in ML programs. To fill this research gap, we propose the first empirical study on the impact of programming languages on bugs in ML programs. We plan to analyze software from GitHub and related discussions in GitHub issues and Stack Overflow for bug distributions in ML programs, aiming to identify correlations with the chosen programming language, its features and the application domain. This study’s results enable better-targeted use of available programming language technology in ML programs, preventing bugs, reducing errors and speeding up development.
SuperCompCloud
ScalaLoci

Leveraging Hybrid Cloud HPC with Multitier Reactive Programming

Daniel Sokolowski, Jan-Patrick Lehr, Christian Bischof, Guido Salvaneschi

In Proceedings of the 3rd IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies, SuperCompCloud, 2020

Abstract PDF

The advent of cloud computing has enabled large-scale availability of on-demand computing and storage resources. However, these benefits are not yet at the fingertips of HPC developers: Typical HPC applications use on-premise computing resources and rely on static deployment setups, reliable hardware, and rather homogeneous resources. This hinders (partial) execution in the cloud, even though applications could benefit from scaling beyond on-premise resources and from the variety of hardware available in the cloud to speed up execution.
To address this issue, we orchestrate computationally intensive kernels using a high-level programming language that ensures advanced optimization and improves execution flexibility-enabling hybrid cloud/on-premise HPC deployments. Our approach is based on multitier reactive programming, where distributed code is defined within the same compilation unit and computations are placed explicitly using placement types. We adjust placement based on performance characteristics measured before execution, apply our approach to a shortest vector problem (SVP) solver from cryptanalysis, and evaluate it to be effective.
OOPSLA

Language-Integrated Privacy-Aware Distributed Queries

Guido Salvaneschi, Mirko Köhler, Daniel Sokolowski, Philipp Haller, Sebastian Erdweg, Mira Mezini

Proceedings of the ACM on Programming Languages 3 (OOPSLA), 2019

Abstract PDF

Distributed query processing is an effective means for processing large amounts of data. To abstract from the technicalities of distributed systems, algorithms for operator placement automatically distribute sequential data queries over the available processing units. However, current algorithms for operator placement focus on performance and ignore privacy concerns that arise when handling sensitive data.
We present a new methodology for privacy-aware operator placement that both prevents leakage of sensitive information and improves performance. Crucially, our approach is based on an information-flow type system for data queries to reason about the sensitivity of query subcomputations. Our solution unfolds in two phases. First, placement space reduction generates deployment candidates based on privacy constraints using a syntax-directed transformation driven by the information-flow type system. Second, constraint solving selects the best placement among the candidates based on a cost model that maximizes performance. We verify that our algorithm preserves the sequential behavior of queries and prevents leakage of sensitive data. We implemented the type system and placement algorithm for a new query language SecQL and demonstrate significant performance improvements in benchmarks.
REBLS
ScalaLoci

Multitier Reactive Programming in High Performance Computing

Daniel Sokolowski, Philipp Martens, Guido Salvaneschi

Presentation at the 6th Workshop on Reactive and Event-based Languages and Systems (REBLS), 2019

Abstract PDF

High Performance Computing (HPC) is crucial in a number of sectors, including weather forecasts, particle simulations and fluid dynamics. Existing programming frameworks for HPC expose developers to low-level details such as message passing and explicit memory management, which are hard to program and error-prone.
In this paper, we present ongoing work on increasing the level of abstraction for HPC. We tackle this problem with an approach based on a combination of multitier programming and reactive programming which enables the development of complex processor configurations in a uniform way using event streams as communication pattern. We report our experience with LULESH, a well known HPC benchmark, and we outline our research roadmap.