User:Tim.Jenssen/ProjectStorage Architecture

ProjectStorage Architecture

Overview

ProjectStorage is the SQLite-backed replacement for the old QMLJS code model. It provides a unified source of truth for QML type information (types, imports, properties, annotations, prototypes, extensions) and serves it via the NodeMetaInfo API to all views.

Benefits (persistent DB + incremental updates)

• **Faster** startup/reloads
• **Consistent** cross-view data
• **Clear separation** of filesystem → parsing → storage

At runtime ProjectStorage

• Parses QML / QML types (QmlDocumentParser, QmlTypesParser)
• Synchronizes into SQLite (IDs, links)
• Serves fast **ID-based** queries
• Tracks file changes (watcher + status cache)
• Notifies dependents on change

Architecture

+-----------------------------------------------------------------+
|                   Qt Design Studio Frontend                     |
|(Navigator, FormEditor, PropertyEditor, States, Timeline, etc.)  |
+--------------------^--------------------------------------------+
                     | NodeMetaInfo / Model API (IDs)
                     v
            +--------------------------------------+
            |       QmlDesigner::ProjectStorage     |
            |  - synchronize(SynchronizationPackage)|
            |  - type/import/alias/property APIs    |
            +--------------------------------------+
                     |                   ^
                     |                   |
                     v                   |
    +----------------+--------+    +-----+--------------------+
    | QmlDocumentParser       |    | QmlTypesParser           |
    | (.qml -> Types, Imports)|    | (.qmltypes -> Exports)   |
    +--------------------------+    +--------------------------+
                     |
                     v
            +--------------------------------------+
            |  SQLite (IDs, Types, Imports, Links) |
            +--------------------------------------+
                     ^
                     |
    +----------------+----------------------+
    | FileStatusCache + FileSystem          |
    | + ProjectStorageUpdater + PathWatcher |
    +---------------------------------------+

Runtime Flow

1) Initialization

• In-memory SQLite by default
• Common built-ins and module metadata preloaded (CommonTypeCache, ModulesStorage).

2) Parse → Synchronize

• `QmlDocumentParser::parse(...)` → Storage::Synchronization::Type + Imports.
• `QmlTypesParser::parse(...)` → ExportedTypes.
• Synchronize in one call (example):

  projectStorage.synchronize(package);
  

• During sync: upsert types/aliases/defaults, link prototypes/extensions, clean removed items, notify observers.

3) Incremental updates

• ProjectStorageUpdater listens to watcher events, computes deltas, and rebuilds a minimal SynchronizationPackage before calling sync.
• Only affected sources are reparsed and stored.
• Duplicates in exported types are removed to keep the DB canonical.

4) File status tracking

• FileStatusCache::modified(SourceIds) compares cached size/mtime with current filesystem state; returns changed SourceIds.
• Missing entries are added and considered modified on first sighting; results are sorted and de-duplicated.

5) Watching

• Directory-level watching; events compressed to avoid storms; parent directories considered where needed.
• Watcher computes changed entries using cached FileStatus vs. fresh FileStatus and emits:
  • `pathsChanged(SourceIds)`
  • `pathsWithIdsChanged(std::vector<IdPaths>)` (grouped by project part and source type)

6) Query & access

• Views/tooling query via IDs (TypeId, ImportId, PropertyDeclarationId …).
• Names are for display/debugging; hot paths avoid string lookups.

Synchronization Package (inputs)

The updater constructs a rich delta object and passes it to ProjectStorage::synchronize:

• **Imports** + **updatedImportSourceIds**
• **ExportedTypes** + **updatedExportedTypeSourceIds**
• **Types** + **updatedTypeSourceIds**
• **FileStatuses** + **updatedFileStatusSourceIds**
• **ProjectEntryInfos** + **updatedProjectEntryInfoSourceIds**
• **Module dependencies** + **updatedModuleDependencySourceIds**
• **ModuleExportedImports** + **updatedModuleIds**
• **PropertyEditorQmlPaths** + **updatedPropertyEditorQmlPathDirectoryIds**
• **TypeAnnotations** + **updatedTypeAnnotationSourceIds**

Behavior

A) Updated-ID hygiene & de-duplication

• **Duplicate exported types are removed** before sync:

removeDuplicates(package.exportedTypes);

Sorted + unique → clean set passed to storage.

• **Updated file statuses** are filtered against “not updated” sets to avoid false positives:

package.updatedFileStatusSourceIds = filterNotUpdatedSourceIds(...);

B) Filesystem truth via FileStatusCache

• modified():
  • Intersects known cache with incoming SourceIds; if size/mtime differ → push to modified.
  • Adds any unseen SourceId (first encounter counts as modified) and merges into cache.
  • Returns **sorted** list to keep downstream set operations stable.
• find() and updateAndFind(): Lower-bound lookup by SourceId; insert/update lazily; both are traced for debugging.
• remove() (by directory or sources): greedy-difference against cached entries; keeps only “not removed” file statuses.

C) QMLDIR + directory scanning

• **QMLDIR components** become exported types using normalized paths and module IDs; version is converted to storage format:

  addExportedTypesFromQmldir(..., moduleId, ...);
  

• **Directory `.qml` files** populate exported types for PathLibrary modules:

  addExportedTypesFromDirectory(qmlFileNames, directoryId, pathModuleId, ...);
  

• **Module exported imports** from QMLDIR are transformed into canonical “exported imports” rows for both QML and C++ module kinds (auto-version handled).

D) ProjectEntryInfos & removed sources

• When a directory changes, updater:
  • Pushes updatedExportedTypeSourceIds for the directory
  • Computes **removed** QML documents by set-difference (stored entries vs. current files) and marks their **import/type SourceIds** as updated to force cleanup:

    addRemovedImportAndTypeSourceIds(...);
    

  • Emits updatedProjectEntryInfoSourceIds for directory metadata refresh.

E) PropertyEditor QML paths

• Updater scans designated resource roots and adds per-module editor QML mappings; directory changes trigger updates of updatedPropertyEditorQmlPathDirectoryIds.

F) Type annotations (.metainfo)

• Recursive directory scan for `*.metainfo`; parses each file and appends TypeAnnotations into the package:

  updateTypeAnnotation(directory, filePath, sourceId, dirId, package);
  

• **UpdatedTypeAnnotationSourceIds** are computed per directory by merging changed and stored SourceIds; directories with state change aggregate both stored-changed and newly updated:

  updateTypeAnnotationDirectories(package, notUpdated, updatedSrcDict);
  

G) Watcher → Notifier contract

• Watcher groups changed entries by ProjectPartId and SourceType into IdPaths, then notifies:

  m_notifier->pathsChanged(watchedSourceIds);
  m_notifier->pathsWithIdsChanged(changedIdPaths);

Entries are **sorted** and **unique** per group to stabilize the UI/model updates.

H) Filesystem implementation details

• File sizes and timestamps come from `std::filesystem::directory_entry`; errors are tolerated (size set to 0 with error_code).
• Directory enumeration uses Qt’s QDir/QDirIterator; SourceIds are obtained through the SourcePathCache.

Practical Guidance

Minimal updater call

ProjectStorageUpdater::Update u;
u.qtDirectories = {/* Qt module dirs */};
u.projectDirectory = /* project root */;
u.propertyEditorResourcesPath = /* optional resource dir */;
u.typeAnnotationPaths = {/* zero or more roots with *.metainfo */};
updater.update(std::move(u));

Typical call flow

File change detected
    ↓
ProjectStoragePathWatcher::pathsChanged()
    ↓
ProjectStorageUpdater::update()
    ↓
  parse QML / QMLTYPES / metainfo
    ↓
Build SynchronizationPackage
    ↓
ProjectStorage::synchronize()
    ↓
SQLite updated, Observers notified

Debugging & Tracing

• Build with ENABLE_PROJECT_STORAGE_TRACING → Perfetto/Nanotrace records for: update(), file-status checks, QMLDIR scan, directory traversal, annotation parsing, and synchronization.
• Persist DB (set env var QDS_STORE_PROJECTSTORAGE_IN_PROJECT) to inspect with DB Browser for SQLite.

Design Decisions (recap)

• **IDs over strings** in all hot paths; displayName() is UI-only.
• **No fully qualified type names** stored in nodes; qualification through imports.
• **Versions** live in imports, not per node.
• **Subcomponent manager removed**; ProjectStorage owns project .qml components.