Andrew Sazonov

ESS DMSC

Andrew Sazonov

ESS DMSC

A Unified Framework for
Data Processing

Andrew Sazonov

ESS DMSC

Data processing workflow

Data processing workflow

Acquisition
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
Reduction
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
Reduction
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
Reduction
Analysis
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
Reduction
Analysis
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Data processing workflow

Acquisition
Reduction
Analysis
DOI: 10.1126/science.1238932 — Diffraction Results from Mars

Diffraction

Diffraction

Programs

FullProf Suite • Jana2020 • GSAS-II • SHELX • MAUD • FOX • Mag2Pol • CrysPy Editor • TOPAS • TOPAS-Academic • Materials Studio Reflex • HighScore

Libraries

CrysFML • CCSL • pyobjcryst (ObjCryst++) • CCTBX • CrysPy • GSAS-II*

Existing issues

Existing issues

➊ Functionality

Existing issues

➊ Functionality

➋ Usability

Existing issues

➊ Functionality

➋ Usability

➌ Maintainability

Site susceptibility

CrysPy & CCSL only • Needed for MAGIC@ESS

2D Rietveld

Modified GSAS-II only • Needed for DREAM@ESS

Graphical interface complexity

FullProf Suite • Many elements, many windows

Input files complexity

FullProf PCR • Short names, complex structure

Input files complexity

FullProf PCR • Short names, complex structure

Extra dependencies

FullProf • Needs XQuartz on macOS

Not cross-platform

Jana2006/2020 • Windows only

Code complexity

CCSL • Fortran77, short names, goto

Code complexity

CCSL • Fortran77, short names, goto

Libraries-based approach

Libraries-based approach

Programs

FullProf Suite • Jana2020 • GSAS-II • SHELX • MAUD • FOX • Mag2Pol • CrysPy Editor • TOPAS • TOPAS-Academic • Materials Studio Reflex • HighScore

Libraries

CrysFML • CCSL • pyobjcryst (ObjCryst++) • CCTBX • CrysPy • GSAS-II*

Libraries-based approach

Programs

FullProf Suite • Jana2020 • GSAS-II • SHELX • MAUD • FOX • Mag2Pol • CrysPy Editor • TOPAS • TOPAS-Academic • Materials Studio Reflex • HighScore

Libraries

CrysFML • CCSL • pyobjcryst (ObjCryst++) • CCTBX • CrysPy • GSAS-II*

Libraries-based approach

CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

Libraries-based approach

Unified interface (Python) CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

Libraries-based approach

EasyDiffraction​Library Unified interface (Python) CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

Libraries-based approach

Scripting (Python) Jupyter Notebook EasyDiffraction​Library Unified interface (Python) CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

Libraries-based approach

Graphical Interface (QML) Scripting (Python) Jupyter Notebook EasyDiffraction​Library Unified interface (Python) CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

Libraries-based approach

EasyDiffraction​Application Graphical Interface (QML) Scripting (Python) Jupyter Notebook EasyDiffraction​Library Unified interface (Python) CrysPy (Python) CrysFML (Fortran) Other libraries External Libraries

GUI design evolution

GUI design evolution

2019 • Initial mockup

GUI design evolution

2020 • Improved design

GUI design evolution

2021 • Simpler. Cleaner. Lighter

GUI design evolution

2023 • Even more improvements

GUI design evolution

UX expert approval

GUI design evolution

UX expert approval

Linear workflow

Linear workflow

Acquisition
Reduction
Analysis

Linear workflow

Analysis

Linear workflow

Analysis
Sample Model

Linear workflow

Analysis
Sample Model
Experiment

Linear workflow

Analysis
Sample Model
Experiment
Analysis

Linear workflow

Analysis
Sample Model
Experiment
Analysis
Summary

Linear workflow

Analysis
Project
Sample Model
Experiment
Analysis
Summary

Linear workflow

Project
Sample Model
Experiment
Analysis
Summary

Linear workflow

Project
Sample Model
Experiment
Analysis
Summary

Linear workflow

Project
Sample Model
Experiment
Analysis
Summary

Linear workflow

Project
Sample Model
Experiment
Analysis
Summary

Linear workflow

Project
Sample Model
Experiment
Analysis
Summary

Other features

Other features

Free and open-source
https://github.com/EasyScience

Other features

Free and open-source
https://github.com/EasyScience
Cross-platform
Windows, macOS, Linux (Snap Store), Source

Other features

Free and open-source
https://github.com/EasyScience
Cross-platform
Windows, macOS, Linux (Snap Store), Source
Everything is included
All-in-one installer (app) & Python package (lib)

Other features

Free and open-source
https://github.com/EasyScience
Cross-platform
Windows, macOS, Linux (Snap Store), Source
Everything is included
All-in-one installer (app) & Python package (lib)
Different user groups
Basic GUI, Advanced GUI & Python/Jupyter

Other features

Free and open-source
https://github.com/EasyScience
Cross-platform
Windows, macOS, Linux (Snap Store), Source
Everything is included
All-in-one installer (app) & Python package (lib)
Different user groups
Basic GUI, Advanced GUI & Python/Jupyter
Multi-functional
Multiple calculation and minimization engines

Other features

Free and open-source
https://github.com/EasyScience
Cross-platform
Windows, macOS, Linux (Snap Store), Source
Everything is included
All-in-one installer (app) & Python package (lib)
Different user groups
Basic GUI, Advanced GUI & Python/Jupyter
Multi-functional
Multiple calculation and minimization engines
and more...

Modular structure

Modular structure

Do everything in one project?

Modular structure

Divided responsibilities and raised stability

Modular structure

https://easyscience.org

Modular structure

https://easyscience.org

EasyDiffraction EasyDiffractionLib EasyDiffractionApp

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp EasyBackend EasyCore EasyCrystallography

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp EasyBackend EasyCore EasyCrystallography EasyFrontend EasyApp

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp TechniqueIndependantBkg Generic EasyBackend EasyCore EasyCrystallography EasyFrontend EasyApp

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp TechniqueIndependantBkg Generic EasyBackend EasyCore EasyCrystallography EasyFrontend EasyApp

Modular structure

https://easyscience.org

EasyDiffractionBkg Diffraction EasyDiffraction EasyDiffractionLib EasyDiffractionApp TechniqueIndependantBkg Generic EasyBackend EasyCore EasyCrystallography EasyFrontend EasyApp EasyReflectometryBkg Reflectometry EasyReflectometry EasyReflectometryLib EasyReflectometryApp

Unified data analysis interface?

Unified data analysis interface?

Same look and feel for different techniques

Unified data analysis interface?

• EasyDiffraction analysis
Andrew Sazonov

Unified data analysis interface?

• EasyDiffraction analysis
Andrew Sazonov

• EasyReflectometry analysis
Piotr Rozyczko

Unified data analysis interface?

• EasyDiffraction analysis
Andrew Sazonov

• EasyReflectometry analysis
Piotr Rozyczko

• EasyDynamics analysis
Henrik Jacobsen

Unified data analysis interface?

• EasyDiffraction analysis
Andrew Sazonov

• EasyReflectometry analysis
Piotr Rozyczko

• EasyDynamics analysis
Henrik Jacobsen

• EasyImaging analysis
Christian Vedel

Unified data processing interface?

• EasyDiffraction analysis
Andrew Sazonov

• EasyReflectometry analysis
Piotr Rozyczko

• EasyDynamics analysis
Henrik Jacobsen

• EasyImaging analysis
Christian Vedel

• EasyTexture reduction
Oleksandr Koshchii, Jens Walter

Unified data processing interface?

• EasyTexture reduction
Oleksandr Koshchii, Jens Walter

Organisation page

Organisation page

Organisation page

https://easyscience.org

Organisation page

https://easyscience.org

Organisation page

https://github.com/easyscience

Organisation page

https://github.com/easyscience

Common Project Management

Common Project Management

Organization-level
Architecture Decision Records

Common Project Management

Organization-level
Architecture Decision Records
Shared templates (Copier) to enforce organization-wide standards

Common Project Management

Organization-level
Architecture Decision Records
Shared templates (Copier) to enforce organization-wide standards
Common code-quality rules (Ruff) for linting and code/docstring formatting

Common Project Management

Organization-level
Architecture Decision Records
Shared templates (Copier) to enforce organization-wide standards
Common code-quality rules (Ruff) for linting and code/docstring formatting
CI/CD pipelines for code and package testing using GitHub workflows and actions

Common Project Management

Organization-level
Architecture Decision Records
Shared templates (Copier) to enforce organization-wide standards
Common code-quality rules (Ruff) for linting and code/docstring formatting
CI/CD pipelines for code and package testing using GitHub workflows and actions
Automated releases (GitHub/PyPI) and documentation builds (MkDocs)

Live demo?

Live demo?