Migration Custom Source Plugin

There are lots of tools built into Drupal 8 (D8) and Drupal 9 (D9) to assist with migrating from a Drupal 7 (D7) website. There is a whole suite of source and destination plugins that allow you to take data from any D7 node, file or user and migrate it into whatever D8 or D9 entity you want. But Drupal can’t account for every single data source you might have, so at some point in your migration you may hit a snag and need to write your own custom migration source plugin. Luckily, Drupal makes this straightforward.

If you don’t already have a custom migration module built, you can follow Mauricio Dinarte’s tutorial. Once you have that set up, go to your custom migration module and create the following nested folder structure for your custom source plugin:

your_custom_module/
├─ src/
│  ├─ Plugin/
│  │  ├─ migrate/
│  │  │  ├─ source/
│  │  │  │  ├─ CustomSourcePlugin.php

Then create a new PHP file in the source folder.

Now we can write our plugin. If you’ve never written a source plugin before, you can use the <a href="https://api.drupal.org/api/drupal/core%21modules%21node%21src%21Plugin%21migrate%21source%21d7%21Node.php/8.9.x">d7_node</a> source plugin for reference (this is the Drupal core source plugin for migrating nodes from a Drupal 7 database). Set up your namespace and underneath it add use Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase;. Then create a class that extends DrupalSqlBase with an @MigrateSource definition commented above it like so:

<?php

namespace Drupal\your_module\Plugin\migrate\source;
use Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase;

/**
 * 
 * @MigrateSource(
 *   id = "custom_source_plugin",
 * )
 */
class CustomSourcePlugin extends DrupalSqlBase {

Note that your class name should be in CamelCase as usual, but the ID should be in snake_case. The ID is how you will reference your source plugin in a migration YAML file.

Inside your custom source plugin, you will need to create four public functions: public function query(), public function initializeIterator(), public function fields(), and public function getIds(). <br/> <br/>

<strong>public function query()</strong>

Use this to query your source database (such as an old D7 website or whatever external database you are pulling data from). You will build your query using Drupal’s <a href="https://www.drupal.org/docs/8/api/database-api/dynamic-queries">Dynamic Query API</a>. In this example, I have a D7 website that uses an unlimited text field called field_footnote to add footnotes to a node. But in the new D8 website, I want each footnote to be its own entity, while making sure they each stay in the correct order on the correct page. This means I need to process footnotes one by one even though a single node can have several. I also need each footnote to know which node it came from and its order on the page, so the footnotes don’t get scrambled.

To handle this, the query is grabbing the “footnote” text field off all the nodes in the Drupal 7 source database with the entity_id, delta, and field_footnote_value fields selected:

$query = $this->select('field_data_field_footnote', 'f')->fields('f', [
  'entity_id',
  'delta',
   'field_footnote_value',
]);
$query->condition('entity_type', 'node', '=');
return $query;

This means that I process my old Drupal 7 field_data_field_footnote table row by row, turning each footnote into its own entity while keeping track of its parent (entity_id), order (delta) and value (field_footnote_value). This can be the trickiest step to get right because the whole source plugin depends on how the data is queried here. <br/> <br/>

<strong>public function initializeIterator()</strong> <br/> The initializeIterator function is necessary to run the query and start the iterator. All you need is:

$results = $this->query()->execute();
$results->setFetchMode(\PDO::FETCH_ASSOC);
return new \IteratorIterator($results);

If you need to set any constants for the migration, you can do that at this point as well: see <a href="https://api.drupal.org/api/drupal/core%21modules%21file%21src%21Plugin%21migrate%21source%21d7%21File.php/8.9.x">d7_file</a>. <br/> <br/>

<strong>public function fields()</strong> <br/> The fields function lets you state the queried fields from the source table and provide labels:

return [
  'entity_id' => $this->t('Entity ID'),
  'delta' => $this->t('Delta'),
  'field_footnote_value' => $this->t('Footnote'),
];

<strong>public function getIds()</strong> <br/> In the getIds function, you define which data field or fields will be used as unique identifiers. In my footnote example, multiple footnotes can be on the same node, so I need more than just the node ID to uniquely identify them. I also need to add the delta (the footnote’s order on the node).

return [
  'entity_id' => [
    'type' => 'integer',
  ],
  'delta' => [
    'type' => 'integer',
  ],
];

The entity_id and delta fields will get translated as sourceid1 and sourceid2 respectively in the migrate_map table. <br/> <br/>

That’s all you need to create your custom source plugin. Now you can write a migration to use it. Remember that in your migration YAML file you will use the @MigrateSource ID that you set in the comment above the class, not the class name.

If you want some extra credit, you can add a fifth public function, prepareRow. This allows you to make data manipulations on each row before it gets sent to the rest of the migration. If I wanted to set a character limit on my footnotes I could use prepareRow to trim or flag any footnotes that are too long. This can also be done as a hook in your module file, hook_migrate_prepare_row.