An overview of how Datamaton’s Digital Manager works

This is a high level overview of how Datamaton’s Digital Manager works. This app (called Knit) lets you search, organize, backup and use all your digital content from one place. You don’t really need to know how it works to use it, but its nice to know a little more about an app you might use every day.

Knit is an indexer, just like Google & Bing search engines . . .

Similar to how Google and Bing index websites, Knit indexes your storage locations. That is, it makes an inventory of your content while leaving everything undisturbed in its original place. Here are some examples of the kinds of information Knit saves in its index files:

• Emails: email sender, receiver, subject, date, size, message text, attachment info and email attributes.
• Text Messages: sender, receiver, message text, date, attachment info and the channel or group it belongs to.
• Appointments: start & end times, recurrence, subject, description, category, attachment info etc.
• Contacts: name, email address, phone numbers and attributes. Knit automatically harvests contact names and email addresses from email messages it indexes.
• All files: file and folder name, file extension, size, creation & last modification dates and file attributes.
• Photo, music and video files: album name, title, artist, genre, composer, play duration, camera, GPS information, thumbnails photo and other such standard information (e.g. EXIF data).
• Cloud files: user tags/comments and file sharing permissions, in addition to “normal” file properties.
• Compound files like ZIP, ISO and Microsoft Outlook PST files: Knit collects the compound file’s info, as well as info for files/emails embedded inside (as listed above).

Knit lets you search for your content using any of the attributes listed above. You might have a photo inside a ZIP file which is an attachment inside an email that’s been forwarded around a few times. Knit will let you search for the photo based on what you remember about it – e.g. an approximate date range when it was taken.

. . . But it doesn’t save anything on our servers

Unlike traditional search engines, Knit uses your computer to do the work. It saves the index files only on your computer. Even on your computer, it encrypts the index files using a password you provide, so only you can decrypt them. The index files typically take up less than 1% of the original content size they index. You have full control over what it indexes and how often.

A huge advantage of this approach is that nothing about your content ever touches our servers! We can’t use, sell or leak your personal information since we never had it in the first place! Searches are blazingly fast because they directly query index files stored on your computer. You can search your cloud based accounts even when you are not connected to the internet, or search a USB drive or phone that is not currently connected to your computer.

Knit is much more than just a search engine

Knit‘s objective is to “simplify your digital life”. Helping you find your content quickly is just one part of it. Knit is much more than an indexer. It is also an aggregator that lets you search, organize, manage and use all your digital content. There’s a powerful Knit capability that enables this – the ability to create arbitrary content selection rules. A content selection rule specifies:

• a) which storage locations it operates on
• b) which content type it operates on
• c) which content properties it operates on and
• d) how it tests the properties.

Here’s an example: a) From all storage locations b) select emails c) such that email date d) is within the range Jan 1, 2010 and Feb 1, 2010. Another example: a) From C:\, D:\, my Yahoo email accounts and my Google Drive accounts b) select photos c) such that the “camera used” property d) contains the word “iphone”. You can combine together multiple such rules using “AND” and “OR” operators to create very powerful searches.

Here’s how Knit combines indexing and content selection rules to support different capabilities:

• Search: A search is just a combination of one or more content selection rules that may be combined with AND or OR operators. When you issue a search, Knit directly searches the index files it created on your hard drive – this is why Knit searches are so fast!

• Organize: Knit lets you assign tags to organize any type of content stored anywhere. In addition, it supports very powerful Virtual Folders that can contain any type of content, stored anywhere. A virtual folder is simply a collection of content selection rules, though you can also drag-drop specific content into it. For example, the Knit virtual folder “Global Inbox” displays emails and attachments in the folder “Inbox” from all your email accounts. It is simply a collection of two rules:

  • a) From all Data Sources b) select emails c) whose “folder name” property d) contains “Inbox”.
  • a) From all Data Sources b) select files c) whose “I am embedded inside” property d) contains “Inbox”.

  Knit evaluates these rules dynamically, so it will automatically include new/future emails. Since a virtual folder contains only rules and links to the actual content, it takes up very little space and can be easily created and deleted without affecting the actual content that it points to.

• Backup: Knit lets you move or copy any type of content from anywhere to anywhere, manually or via automatic backup tasks. A backup task is actually just a collection of content selection rules paired with a backup destination and backup frequency. For example, you can create a task: “From C:\ select documents whose content contains the word “market” and copy them to Google Drive folder “marketing” at 10am every morning”. Knit runs each such backup task as separate background processes.
• Use: Knit is an email client, calendar manager and file manager rolled into one app. It lets you view all your content and do the usual day-to-day operations on it. You can read, reply or forward email messages. You can move, copy, download, upload and share files in cloud storage accounts. You can create, update, search and delete appointments in local and remote calendars. This is obviously much more than what a search engine like Google or Bing would do.

Knit doesn’t force you to stop using your current apps

You don’t have to stop using the apps and tools you currently use to benefit from Knit. For example: when Knit indexes your email accounts, it will NOT download messages locally and remove them from the server. It will NOT cause an unread email to be marked as read. Thus, you can continue to use your phone or browser to access your email.

For example, a new email message that arrives in your Inbox will start off as “unread”. When Knit indexes it, it will stay “unread” and appear as “unread” on your phone and browser too. If you read that message using your phone app, the message will transition to “read”. Knit will track this and show it as “read” too. You can use either Knit or your phone app to mark it “unread” again (e.g. if you use the “unread” indicator as a “to-do” reminder). The message will appear as “unread” again in both places. Similarly, you can continue to use vendor specific apps from Slack, Google, Microsoft, Dropbox etc. and use Knit only to search, organize or backup that content.

Knit is a collection of applications that run in the background

Knit is actually a collection of many different processes and libraries.

Different Knit components work together to do the overall job:

• The user interface process ( Knit.exe ) is typically the only program you interact with directly. It lets you view, organize, backup and use your data, but it doesn’t actually perform those tasks itself. Other background processes do that, so you can safely close the user interface anytime without affecting anything.
• Background process Cataloger.exe indexes and periodically re-indexes your storage locations.
• Background process Refresher.exe listens for dynamic insertion of CDs, USB drives, phones etc. so they can be indexed as soon as you plug them into your computer. It also makes sure that periodic re-indexing and backup tasks run when needed.
• A disk indexer ( DiskIndexer.exe or FileMonitor.exe ) tracks changes to local hard drives so that newly created or modified files are processed quickly.
• An email indexer ( EmailIndexer.exe or ImapMonitor.exe ) tracks your email accounts so new email is immediately indexed.
• TaskManager.exe runs your backup tasks in the background.
• SpaceMgr.exe periodically runs in the background to delete temporary or cached files created by Knit.
• A separate background process ( KnitUpdater.exe or DMInstallMgr.exe ) periodically checks for Knit updates.
• Recrypt.exe lets you set and change the passwords used to encrypt the Knit index files stored on your computer.

Knit starts and schedules these background processes automatically when needed. You need to interact directly only with the Knit.exe user interface.