Commands Reference

Incremental Sync

Incremental synchronization in the DID Spaces SDK provides an efficient way to manage large datasets by only transferring changes between your local system and DID Spaces. This approach significantly reduces transfer times and bandwidth usage compared to full synchronization. For a broader understanding of synchronization, refer to the Sync Commands section.
Core Concepts#At the heart of incremental synchronization is the IncrementalSyncBaseCommand class, which establishes the foundational logic and shared properties for both push and pull operations. This base command defines common configurations and behaviors:
Default Concurrency: DEFAULT_CONCURRENCY is set to 4 concurrent operations.
Maximum Concurrency: MAX_CONCURRENCY is dynamically set to the number of CPU cores available on the system.
Default Retry Count: DEFAULT_RETRY_COUNT is 3 for failed operations.
Maximum Retry Count: MAX_RETRY_COUNT limits retries to 10 attempts.
Shared functionalities include initialization of command inputs, debug information printing, and a robust retry mechanism for network requests.
Object Comparison Logic#The getWaitSyncObjects method is crucial for determining which objects need to be synchronized. It compares local and remote object metadata to identify PUT (upload/update) or DELETE actions based on several criteria:
Local exists, Remote doesn't: If an object exists locally but not remotely, it's marked for PUT (upload).
Local doesn't exist, Remote does: If an object exists remotely but not locally, and strictSync is enabled, it's marked for DELETE.
Size Mismatch: If both exist but have different sizes, it's marked for PUT.
Local Not Fresher: If both exist and the local object's lastModified timestamp is older or equal to the remote's, no action is needed.
Hash Mismatch: If both exist, sizes match, but content hashes differ, it's marked for PUT.
This logic ensures that only necessary changes are processed, optimizing sync operations.
No
Yes
Yes
No
No
Yes
No
Yes
No
Yes
Yes
No
No
Yes
Start Object Comparison
Object exists locally?
Object exists remotely?
Is strictSync enabled?
Action: DELETE
No Action
Object exists remotely?
Action: PUT
Local Size == Remote Size?
Action: PUT
Local Last Modified <= Remote Last Modified?
No Action
Local Hash == Remote Hash?
Action: PUT
No Action
End

Helper Methods#shouldBeBackup(absolutePath: string): Determines if a file at the given absolute path should be considered for backup, checking for existence and excluding temporary files.
errorCount: Returns the total number of errors encountered during the sync process.
completedCount: Returns the total number of objects successfully processed during the sync process.
totalSize: Returns the total size of objects processed during the sync process.
totalCount: Returns the total number of objects initially identified for synchronization.
destroy(): Clears the internal queue and removes temporary directories used during sync.
Incremental Push Synchronization#The IncrementalSyncPushCommand is specifically designed to push local changes to your DID Space. It leverages the base command's features to efficiently upload new or modified files and delete remote files that no longer exist locally (if strictSync is enabled).
Synchronization Process (request method)#The request method orchestrates the incremental push process:
Get Object Maps: It first retrieves the map of local objects and remote objects.
Determine Sync Objects: It then uses the getWaitSyncObjects logic to identify which objects need to be PUT or DELETE.
Execute Sync: The sync method is called to perform the actual operations.
DID Spaces APIPQueueIncrementalSyncBaseCommandIncrementalSyncPushCommandDID Spaces APIPQueueIncrementalSyncBaseCommandIncrementalSyncPushCommandalt[if action is PUT][if action is DELETE]loop[For each syncObject]alt[On Error]request()throwErrorIfAborted()getLocalObjectsMap()getRemoteObjectsMap()getWaitSyncObjects(local, remote)initializeSyncObjectRuntimeMap(syncObjects)sync(syncObjects)addAll(actions for PUT/DELETE)Execute actionthrowErrorIfAborted()copyObjectToTmpDir(syncObject)emit SYNC_BEFORE eventPutObjectCommand(tempFile)DeleteObjectCommand(remoteKey)Update completedCount/totalSizeprintDebugInfo()emit SYNC_AFTER eventdestroyObject(syncObject)onIdle()Return IncrementalSyncPushCommandOutputretryRequest(error)retryRequest(error)clear() (queue)sleep()Call request() again

Input Parameters (IncrementalSyncPushCommandInput)#Name
Type
Description
source
Object[]
An array of local objects (Object type from @did-space/core) to be synchronized.
metadata
ObjectMetadata
Optional metadata to be associated with the objects during the sync.
target
string
The target path in DID Spaces where objects will be synchronized.
concurrency
number
Optional. The number of concurrent operations to perform during synchronization. Defaults to 4.
retryCount
number
Optional. The number of times to retry a failed operation. Defaults to 3.
strictSync
boolean
Optional. If true, objects existing remotely but not locally will be deleted from the remote. Defaults to false.
tmpDir
string
Optional. A temporary directory used for intermediate files during synchronization.
debug
boolean
Optional. If true, enables detailed debug logging.
filter
(object: Object) => boolean
Optional. A function to filter which local objects are included in the synchronization.
onProgress
(input: OnProgressInput) => void | Promise<void>
Optional callback function triggered during the sync process to report progress. OnProgressInput contains completed, total, key.
onAfterUpload
(input: OnAfterUploadInput) => void | Promise<void>
Optional callback function triggered after an object has been successfully uploaded. OnAfterUploadInput contains completed, total, key.
Output (IncrementalSyncPushCommandOutput)#Name
Type
Description
statusCode
number
The HTTP status code of the operation.
statusMessage
string
Optional. A message describing the status of the operation.
stack
string
Optional. The error stack trace if an error occurred.
data
{ count: number; errorCount: number; duration: number; size: number }
An object containing detailed statistics about the sync operation: count (total objects processed), errorCount (number of failed operations), duration (total time in seconds), size (total data size in bytes).
Example: Performing an Incremental Push Sync#This example demonstrates how to perform an incremental push synchronization of a local directory to a DID Space. It configures the client and initiates the IncrementalSyncPushCommand with a list of local files.
import { SpaceClient } from '@did-spaces/client';
import { join } from 'path';
import { readdir, stat } from 'fs/promises';

async function getLocalFilesRecursively(dirPath, baseDir) {
  let files = [];
  const entries = await readdir(dirPath, { withFileTypes: true });

  for (const entry of entries) {
    const fullPath = join(dirPath, entry.name);
    const relativePath = join(baseDir, entry.name).replace(/\\/g, '/'); // Normalize path for DID Spaces
    if (entry.isDirectory()) {
      files = files.concat(await getLocalFilesRecursively(fullPath, relativePath));
    } else {
      const stats = await stat(fullPath);
      files.push({
        key: relativePath, // Path relative to the target in DID Spaces
        name: entry.name,
        isDir: false,
        size: stats.size,
        lastModified: stats.mtimeMs,
        absolutePath: fullPath,
      });
    }
  }
  return files;
}

async function performIncrementalPushSync() {
  const client = new SpaceClient({
    endpoint: 'YOUR_DID_SPACE_ENDPOINT',
    signer: async (content) => {
      // Implement your DID signature logic here
      // This is a placeholder, use a real DID signer
      return { 
        sig: 'YOUR_SIGNATURE',
        pk: 'YOUR_PUBLIC_KEY'
      };
    },
  });

  const localSourceDir = './local_data'; // Your local folder to sync
  const targetDidSpacePath = 'my-space-folder/'; // Path in your DID Space

  try {
    // Gather local files recursively into the expected Object[] format
    const localObjects = await getLocalFilesRecursively(localSourceDir, '');

    const command = client.getIncrementalSyncPushCommand({
      source: localObjects,
      target: targetDidSpacePath,
      concurrency: 8,
      strictSync: true, // Delete remote files that don't exist locally
      debug: true,
      onProgress: (progress) => {
        console.log(`Syncing: ${progress.key} (${progress.completed}/${progress.total})`);
      },
      onAfterUpload: (result) => {
        console.log(`Completed: ${result.key}`);
      },
    });

    console.log(`Starting incremental push sync from ${localSourceDir} to ${targetDidSpacePath}`);
    const result = await command.request();

    if (result.statusCode === 200) {
      console.log('Incremental push sync completed successfully:');
      console.log(`  Total files processed: ${result.data.count}`);
      console.log(`  Errors: ${result.data.errorCount}`);
      console.log(`  Duration: ${result.data.duration} seconds`);
      console.log(`  Total size: ${result.data.size} bytes`);
    } else {
      console.error('Incremental push sync failed:');
      console.error(`  Status Code: ${result.statusCode}`);
      console.error(`  Message: ${result.statusMessage}`);
      console.error(`  Stack: ${result.stack}`);
    }
  } catch (error) {
    console.error('An unexpected error occurred during incremental push sync:', error);
  }
}

performIncrementalPushSync();This example first defines a helper function getLocalFilesRecursively to collect file information in the format expected by the SDK. It then initializes SpaceClient, specifies local and remote paths, and executes the IncrementalSyncPushCommand with custom onProgress and onAfterUpload callbacks for real-time updates. The strictSync option is enabled to ensure that remote objects not present locally are deleted.
Incremental Pull Synchronization#The provided information focuses on the push synchronization. An incremental pull synchronization command would follow a similar pattern, utilizing the IncrementalSyncBaseCommand to compare local and remote objects and determine which files need to be downloaded or deleted locally to match the remote state. Specific implementation details for IncrementalSyncPullCommand are not available in the current documentation.

Incremental synchronization is a powerful feature for maintaining up-to-date copies of your data efficiently. By understanding the core comparison logic and utilizing the IncrementalSyncPushCommand, you can manage large datasets with ease. For more details on the specific actions performed, refer to the Put Object and Delete Object commands, or explore the Sync Push and Sync Pull commands for full folder synchronization capabilities.

Name	Type	Description
`source`	`Object[]`	An array of local objects (`Object` type from `@did-space/core`) to be synchronized.
`metadata`	`ObjectMetadata`	Optional metadata to be associated with the objects during the sync.
`target`	`string`	The target path in DID Spaces where objects will be synchronized.
`concurrency`	`number`	Optional. The number of concurrent operations to perform during synchronization. Defaults to `4`.
`retryCount`	`number`	Optional. The number of times to retry a failed operation. Defaults to `3`.
`strictSync`	`boolean`	Optional. If `true`, objects existing remotely but not locally will be deleted from the remote. Defaults to `false`.
`tmpDir`	`string`	Optional. A temporary directory used for intermediate files during synchronization.
`debug`	`boolean`	Optional. If `true`, enables detailed debug logging.
`filter`	`(object: Object) => boolean`	Optional. A function to filter which local objects are included in the synchronization.
`onProgress`	`(input: OnProgressInput) => void \| Promise<void>`	Optional callback function triggered during the sync process to report progress. `OnProgressInput` contains `completed`, `total`, `key`.
`onAfterUpload`	`(input: OnAfterUploadInput) => void \| Promise<void>`	Optional callback function triggered after an object has been successfully uploaded. `OnAfterUploadInput` contains `completed`, `total`, `key`.

Name	Type	Description
`statusCode`	`number`	The HTTP status code of the operation.
`statusMessage`	`string`	Optional. A message describing the status of the operation.
`stack`	`string`	Optional. The error stack trace if an error occurred.
`data`	`{ count: number; errorCount: number; duration: number; size: number }`	An object containing detailed statistics about the sync operation: `count` (total objects processed), `errorCount` (number of failed operations), `duration` (total time in seconds), `size` (total data size in bytes).

Overview

Quick Start

Core Concepts

Commands Reference

Utilities

Incremental Sync

Core Concepts#

Object Comparison Logic#

Helper Methods#

Incremental Push Synchronization#

Synchronization Process (`request` method)#

Input Parameters (`IncrementalSyncPushCommandInput`)#

Output (`IncrementalSyncPushCommandOutput`)#

Example: Performing an Incremental Push Sync#

Incremental Pull Synchronization#

Pull

Backup