SharePoint Data Connector

The BigPanda Unified Data Connector (UDC) syncs SharePoint content through the Microsoft Graph API to provide context and insights for AI Incident Assistant (Biggy), AI Incident Prevention, and AI Detection and Response.

You can sync three kinds of SharePoint content, in any combination:

Lists: List item metadata and selected scalar list columns.
Site Pages: Page metadata, and optionally the readable page body text.
Documents: Files from a document library, and optionally text extracted from those files.

Ingested data is securely stored and made available in the IT Knowledge Graph, powering analytics, trend analysis, and downstream operations workflows.

When to use this connector

SharePoint lists are the source of truth (change calendars, KB indexes, runbook trackers, approval lists, etc.).
Your team keeps runbooks or knowledge on Site Pages or in a document library, and you want that text available alongside list metadata.
You need scheduled, incremental sync into the IT Knowledge Graph.

When not to use this connector

You need attachment binaries or raw page HTML in BigPanda. The connector extracts readable text, not binaries or markup.
Your primary knowledge lives in Confluence or ServiceNow KB. Use those connectors instead.
You run SharePoint on-premises farms (non-Microsoft 365). These are not supported through this Graph-based connector.

Authentication

The SharePoint connector uses OAuth 2.0 client credentials against Microsoft Entra ID (Azure AD) and calls Microsoft Graph. BigPanda refreshes credentials before sync requests, so scheduled runs continue without manual re-authorization. Auth strategy cannot be changed when editing an existing connection.

Microsoft Graph prerequisites

Before BigPanda can configure the connector, your organization must register an application in Microsoft Entra ID that grants the BigPanda application (not delegated) read access to the SharePoint list data you want to sync. The application requires the following admin-consented Microsoft Graph application permission:

Sites.Read.All

Provide the client ID, the client secret, and the following connection settings to your BigPanda account team, who will complete the authorization and set up the connector.

Setting	Value
`instance_url`	`https://graph.microsoft.com` instance_url is always the Graph API base URL. The SharePoint site path is configured separately on the pipeline as site_url.
Auth strategy	OAuth 2.0 client credentials
Application (client) ID	From your Entra app registration
Client secret	The secret Value (not the Secret ID)
`oauth2_token_url`	`https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token`
`scope`	`https://graph.microsoft.com/.default` (used by default when omitted)

Configure the SharePoint connector

Provide the following configuration to your BigPanda account team. Configure at least one of lists, pages, or documents. Each entry in those maps produces one output table, and table names must be unique across all three maps.

Field	Required	Default	Description
`site_url`	Yes	—	SharePoint site path in Graph format: `hostname:/sites/....` For example: `contoso.sharepoint.com:/sites/MySite.` Do not include https://.
`lists`	No	—	Map of output table name and list definition. Each list produces one output table. Configure at least one of `lists`, `pages`, or `documents`.
`pages`	No	—	Map of output table name to Site Pages definition. Each entry produces one output table.
`documents`	No	-	Map of output table name to document-library definition. Each entry produces one output table.
`start_date`	Yes	—	The start of the sync window, in `YYYY-MM-DD` format. Sets the initial sync window and incremental cursor baseline. Required when creating a pipeline. Must be today or earlier.
`end_date`	No	-	The end of the sync window, in `YYYY-MM-DD` format. Applies to pages and documents.
`cron_schedule`	Yes	—	Cron expression for scheduled sync runs (for example, every 15 minutes).
`timezone`	No	UTC	Timezone for schedule interpretation.
`page_size`	No	100	Items per Graph request ($top). Maximum 999.
`rate_limit`	No	20	Maximum requests per minute.
`rate_limit_timeout_ms`	No	1000	Milliseconds to wait when the local rate limiter throttles requests.
`request_timeout`	No	60	Seconds before a Graph request times out.

Lists

Each lists entry accepts the following fields.

Field	Required	Default	Description
`list_id`	Yes	—	The Microsoft Graph list GUID.
`field_names`	No	All columns	The specific list columns to include, using their internal SharePoint names. Omit this field to load all list columns. The active cursor field remains available for incremental tracking even if it isn't included in `field_names`. The identity fields (`site_id`, `site_name`, `list_id`, and `list_name`) and `_udc_resource_type` are always included on every row, regardless of `field_names`.
`cursor_field`	No	—	Sets the sync cursor for that list. Accepts only `lastModifiedDateTime` or `Modified`. Any other value fails configuration validation. Omit to inherit the pipeline-level default.

Pages

Each pages entry accepts the following fields.

Field	Required	Default	Description
`include_content`	No	`true`	When `true`, the connector writes the page body text into `content_text`. When `false`, the connector syncs page metadata only.

Documents

Each documents entry accepts the following fields. Identify the library with at most one of drive_id, list_id, or library_name. Omit all three to use the site default library. Setting more than one library selector fails configuration validation.

Field	Required	Default	Description
`drive_id`	No	—	The library drive ID. Use at most one library selector.
`list_id`	No	—	The library list ID. Use at most one library selector.
`library_name`	No	—	The library display name. Use at most one library selector.
`include_content`	No	`true`	When `true`, the connector extracts file text into `content_text`. When `false`, the connector syncs file metadata only.
`folder_path`	No	—	Limits the sync to a folder within the library, for example `Runbooks/Prod`.
`file_extensions`	No	All	The file extensions to include, lowercase and without a leading dot, for example `docx`.
`max_file_size_mb`	No	25	The maximum file size to extract content from, in MB. Allowed range is 1–250. Files over this size still sync metadata, with `content_extraction_status` set to `skipped_too_large`.

Unknown keys are rejected

A documents entry that includes an unrecognized key fails configuration validation rather than ignoring the key. Check field names against this table if a run fails to validate.

Example configuration

{
  "cron_schedule": "*/15 * * * *",
  "start_date": "2024-01-01",
  "timezone": "UTC",
  "site_url": "contoso.sharepoint.com:/sites/MySite",
  "lists": {
    "tasks": {
      "list_id": "00000000-0000-0000-0000-000000000001",
      "field_names": ["Title", "Status", "Priority"]
    }
  },
  "pages": {
    "site_pages": {
      "include_content": true
    }
  },
  "documents": {
    "runbooks": {
      "library_name": "Documents",
      "folder_path": "Runbooks/Prod",
      "file_extensions": ["docx", "pdf"],
      "max_file_size_mb": 25,
      "include_content": true
    }
  },
  "page_size": 100,
  "rate_limit": 20,
  "rate_limit_timeout_ms": 1000,
  "request_timeout": 60
}

Finding a list GUID

In SharePoint, open the List > Settings > List settings. The list GUID appears in the URL as List=%7B<guid>%7D, or use Microsoft Graph to enumerate lists for the site.

Output schema

The connector creates one output table per entry across the lists, pages, and documents maps. Table names must be unique across all three maps.

Lists

Each list table uses id as its primary key. The sync cursor is Modified by default, or lastModifiedDateTime if configured through cursor_field for that list.

Field	Description
`id`	Unique identifier of the list item. This is the primary key for the table.
`createdDateTime`	The date and time the item was created.
`lastModifiedDateTime`	The date and time the item was last modified, as reported by Microsoft Graph.
`Modified`	The SharePoint `Modified` column. The connector uses this column as the sync cursor unless `cursor_field` is set to `lastModifiedDateTime` for that list.
`webUrl`	A link to the item in SharePoint.
`_udc_resource_type`	The type of SharePoint resource the row represents. Always `list` for list tables.
`site_id`	The Microsoft Graph site ID. Always present.
`site_name`	The site display name. May be empty if the display name cannot be resolved.
`list_id`	The Microsoft Graph list GUID, matching the `list_id` you configured for this list. Always present.
`list_name`	The list display name. May be empty if the display name cannot be resolved.
Selected scalar list columns	Any scalar columns you name in `field_names` for the list. Only scalar values are included. When omitted, all `field_names` are included.

Pages

Each pages table uses id as its primary key and tracks incremental changes on lastModifiedDateTime.

Field	Description
`id`	Unique identifier of the page. This is the primary key for the table.
`name`	The page file name.
`title`	The page title.
`webUrl`	A link to the page in SharePoint.
`_udc_resource_type`	The type of SharePoint resource the row represents. Always `page` for pages tables.
`site_id`	The Microsoft Graph site ID. Always present.
`site_name`	The site display name. May be empty if the display name cannot be resolved.
`pageLayout`	The page layout type.
`createdDateTime`	The date and time the page was created.
`lastModifiedDateTime`	The date and time the page was last modified. This is the sync cursor.
`publishingState`	The publishing state of the page.
`content_text`	The readable page body text. Present only when `include_content` is `true`.

Documents

Each documents table uses id as its primary key and tracks incremental changes on lastModifiedDateTime. When content is requested, each row also reports the extraction outcome.

Field	Description
`id`	Unique identifier of the file. This is the primary key for the table.
`lastModifiedDateTime`	The date and time the file was last modified. This is the sync cursor.
`webUrl`	A link to the file in SharePoint.
`_udc_resource_type`	The type of SharePoint resource the row represents. Always `document` for documents tables.
`site_id`	The Microsoft Graph site ID. Always present.
`site_name`	The site display name. May be empty if the display name cannot be resolved.
`drive_id`	The Microsoft Graph drive ID of the document library.
`content_text`	The extracted file text. Present only when `include_content` is `true` and extraction succeeds.
`content_extraction_status`	The outcome of content extraction for the row. Present when content is requested.
`content_char_count`	The number of characters extracted. Present when content is requested.

content_extraction_status takes one of the following values: ok, empty, unsupported_format, skipped_too_large, error, or metadata_only.

Sync behavior

Ongoing sync is incremental. Lists track changes on the SharePoint Modified column (or lastModifiedDateTime when set through cursor_field). Pages and documents track changes on lastModifiedDateTime.

Initial / backfill: On the first run (or after a cursor reset), the connector loads items modified on or after start_date.
Subsequent runs: The connector loads items modified on or after the stored cursor from the previous successful run.
Scheduling: cron_schedule controls how often incremental runs execute.

Filtering uses SharePoint modified column

The connector filters on the SharePoint Modified field rather than the Graph lastModifiedDateTime field, because Microsoft Graph does not support filtering on lastModifiedDateTime for list items.

Pages and documents filter client side

Microsoft Graph cannot filter Site Pages or document-library files by lastModifiedDateTime, so the connector enumerates the full collection and then drops records outside the start_date and end_date window on its own. Expect longer runs and more Graph traffic on large sites and libraries, even for a narrow date window.

Complete list pagination

List pagination now continues until Microsoft Graph returns no next-page link.

Request and performance controls

You can tune the following controls to manage paging and request behavior.

Control	Default	Description
`page_size`	`100`	The number of items requested from Microsoft Graph in a single call. This maps to the Graph `$top` page size.
`rate_limit`	`20`	The maximum number of requests sent per minute.
`rate_limit_timeout_ms`	`1000`	Wait time when the connector’s rate limiter throttles outbound requests.
`request_timeout`	`60`	How long the connector waits, in seconds, before timing out a request.

When Microsoft Graph returns HTTP 429, the connector honors the Retry-After header before retrying.

Content extraction adds Graph requests

When include_content is true, the connector makes one extra Graph request per page and per extractable file. On large sites or libraries, tune rate_limit and page_size, and use folder_path, file_extensions, and max_file_size_mb to keep a document sync bounded.

Troubleshooting

If a sync run fails, review the items below.

Symptom	What to check
Authentication failures	Entra app client ID, client secret Value, `Sites.Read.All` application permission, and admin consent are valid. Token URL uses the correct tenant_id.
Configuration validation failure	At least one of `lists`, `pages`, or `documents` is set. Table names are unique across all three maps. Each `documents` entry sets at most one of `drive_id`, `list_id`, or `library_name`, and uses only recognized keys.
Wrong API endpoint	Connection instance_url is https://graph.microsoft.com, not the SharePoint site hostname. Site path belongs in pipeline `site_url`.
Site resolution failures	`site_url` uses Graph site path format (`hostname:/sites/...`) without `https://`. Site exists and the app can read it.
List not found / empty table	`list_id` is the correct Graph list GUID for that site. App has read access to the site and list.
Wrong library	Exactly one of `drive_id`, `list_id`, or `library_name` is set, or all three are omitted to use the site default.
Missing or wrong columns	`field_names` use internal SharePoint column names (for example Title, not a display label). The identity fields (`site_id`, `site_name`, `list_id`, and `list_name`) and `_udc_resource_type` are always present and cannot be removed with `field_names`, so a restrictive `field_names` list does not drop them.
Empty `site_name` or `list_name`	This is expected, not a failed run. `site_name` and `list_name` are best-effort display names, and the connector leaves them empty when the display-name lookup cannot resolve them. `site_id` and `list_id` are always populated.
Missing rows	For lists, pagination continues on empty pages, so confirm the cursor and date window. For pages and documents, the connector lists the full collection and drops records outside the `start_date` and `end_date` window client-side.
Slow or throttled runs (429)	Check `include_content`, library size, `file_extensions`, `folder_path`, and `max_file_size_mb`. Lower `rate_limit` or increase `rate_limit_timeout_ms`. The connector already respects the Graph `Retry-After` header.
Unexpected date range	`start_date` is `YYYY-MM-DD`, not in the future, and reflects the backfill window you intend.

FAQs

Why does the connection use graph.microsoft.com instead of our SharePoint site URL?

The connection authenticates to Microsoft Graph. The SharePoint site is configured separately on the pipeline as site_url.

Can this connector ingest Word or PDF files from document libraries?

Yes. Configure a documents entry with include_content set to true. Built-in extraction covers common Office Open XML and plain-text or HTML files. PDF extraction depends on an optional runtime dependency, and legacy binary Office files stay metadata-only. Files over max_file_size_mb sync metadata only.

Can this connector sync Site Pages?

Yes. Configure a pages entry. Set include_content to true to write the page body text into content_text, or false for page metadata only.

Do we need Files.Read.All?

No for this connector. Sites.Read.All (application) is sufficient for list metadata.

In this section: