Buying the developer license allows you to install and use the module on an unlimited number of sites.

• DataGrab developer license

The license has been introduced for an experimental period. I am hoping it is something that EE developers will find useful - several people have already requested it. The price of the license is also on a trial basis and may change in the future, most likely upwards.

Currently the license is only available on brandnewbox.co.uk, but if it works here I’ll introduce it on devot-ee too.

On a side point, in recent months DataGrab has been available on this site at a reduced price. This “sale” will soon end and prices on this site and devot-ee will be kept approximately inline (exchange rates permitting).

I’d be interested in any comments you have regarding developer licenses and their pricing.

Andrew

This is a holding page to add notes regarding importing data into a Matrix field.

Preliminary support for Pixel & Tonic’s Matrix is available in DataGrab 1.7.0 and above. Please consider this support as a Beta until further notice.

Version details

• DataGrab version 1.7.1
• Matrix version 2.2.4 Earlier versions will not work.

Configuring the import

The results

Matrix cell types

The beta version currently supports text, date and file cells. Support for other cell types to follow.

Date fields can be formatted in any recognisable date format (YYYY-MM-DD HH:MM:SS is recommended)

Images/files should be uploaded manually to an existing File Upload area and the filename in the data source should be prefixed with the Upload area variable (e.g., {filedir_1}filename.png if filename.png is in the upload area with the id of 1)

Supported data types

CSV and XML imports are supported. I’ll document these better as the beta progresses. Feel free to contact me in the meantime if you need help.

Example data format are:

XML

ie, repeating item elements can be imported into a Matrix. Select the first item in the configuration pulldown, and DataGrab will add the rest

CSV with Matrix rows on separate lines of the file

In the settings assign a unique id (eg, product id) to a custom field. Use that field to check for duplicates and check the Update existing entries setting.

CSV with matrix rows as comma delimited values in a cell

Note, that the 4th to 7th columns each have 2 comma-separated values. Each value will be assigned to a new matrix row.

Known problems

Please consider this Beta level software. Features may not work and may change in future releases.

How to find the current memory limit

You can find the current PHP memory limit from the ExpressionEngine 2 Control Panel

Go to: Tools > Utilities > PHP Info



and search for the line “memory_limit”



the first value (in the “Local” column) will tell you the current memory limit

Increasing the memory limit

Depending on your web hosting, you may be able to increase your memory limits. Some hosts will not allow this, or will limit the maximum amount of memory you can set. I’ve found some hosts are willing to set quite high limits if you explain it will be for a one-off data import and that it can be lowered again afterwards.

Your web hosting service will be able to tell you the appropriate way to increase you memory limit, but here is a summary of the options.

.htaccess

This is my preferred method as it makes it easy to set values of a site-by-site basis, but will not work under all hosting configurations. In a .htaccess file in the root of your website, add the line:

This will give you a memory limit of 128M.

php.ini

If you are able to edit the server’s php.ini file you can modify the the memory limit there. Look for the line:

and adjust the value accordingly. This will apply to all sites.

You may also be able to create/modify a php.ini in the website’s root folder.

This version has made a number of changes to the underlying code to make future development easier.

New features include:

• Preliminary support for Matrix fields
• Multiple category group support
• Improved compatibility with other field types (e.g., Playa, multi select checkboxes, files)
• Authors can be assigned from custom member fields
• Added basic inline help. First stages of documentation improvements
• Fixed numerous bugs

The latest version can be download from the Downloads section.

Url Title

DataGrab’s default behaviour is to generate an entry’s url title from the entry’s Title field (as it would if you were publishing an entry manually). If you need the entry to have a specific url title then you can specify which field of the data source to use.

Note, that URL Titles cannot be made entirely from numbers. If your titles are numeric, then you can use the Publish Page Customization options under the Channel Preferences to add a URL Title Prefix.

Date

DataGrab will attempt to read your date in whatever format you provide. For best results use YYYY-MM-DD hh:mm:ss. Try to avoid ‘ambiguous’ formats, such as DD/MM/YY or MM/DD/YY.

Duplicate Entries

DataGrab can check whether entries already exists and if they do it can skip or attempt to update the existing entry.

To check for an existing or duplicate entry, DataGrab will test whether the potential new entry has a field value that matches an existing entry.

For example, an RSS feed often has a guid field that uniquely identifies a feed element, or a store product usually has a stock id number.

If you save this value into a custom field, you can then, on subsequent imports, check this field for an existing entry and update it or ignore it as appropriate.

You can use the Use this field to check for duplicates setting to select which field to check. If you leave this field blank it will always add a new entry.

if you check the Update existing entries option then DataGrab will attempt to update the entry’s values from the new record. Otherwise, it will ignore it (ie, not update it, but not add a new entry).

Add a timestamp to this field

DataGrab can add a timestamp to a field that it adds or updates. This can help in tracking which entries have been updated or deleting old entries.

Delete old entries

If you check this, DataGrab will delete any entries in the channel that have not been added or updated by this import. This can be useful for maintaining an up-to-date product list, for example, but must be used with care.

Default Author

This allows you to set the author of any new entries.

Author

You can also assign the author from a field in the data source. The Author Field Value tells the system what format the author is stored as (eg, email address, username) and Author specifies which data field it is in.

Status

The Status field allows you to specify a data source field to set the entry’s status or you can use Open, Closed or the channel’s default.

Offset

Sometimes you need to adjust the time to take into account localization. The offset option allows you to add or subtract any number of seconds from the date/time values.

The next stage of development will be some underlying code changes to help support the import of 3rd party field types (such as Matrix). I’m going to try and maintain an informal road-map here to give some idea of the current priorities.

The latest version can be download from the Downloads section.

Some of the new features include:

• Support for Solspace Tags and SEO Lite add-ons
• Ability to set url title and expiry date from data
• Bug fixes and minor optimisations

Matrix support

Once this version is released I will be concentrating on adding Matrix support.

Just as Matrix provides huge flexibility in how you build your site, it makes the configuration of the import difficult - it becomes effectively an import-within-an-import. Coupled with DataGrab’s ability to handle different data types (CSV, XML etc) it makes mapping values from the data source to Matrix fields a challenge.

I have a version of DataGrab that successfully imports into a Matrix field, but the data needs to be formatted quite specifically. To get the development process to the next stage I need real-world examples of what you need to do.

If adding Matrix support to DataGrab would be useful for you, please could you or add a post in the support forum with examples of data sources (eg, CSV, XML files or others) and matrix designs (screenshots would be great) and how you’d like the data to map to the Matrix fields.

This will give me a much better understanding of what the Matrix import process needs to achieve.

There is an example of what would be helpful here:

Documentation

While developing the next stage I also want to improve the module’s documentation. Any feedback would be much appreciated.

Many thanks, Andrew

DataGrab won’t create members for you - it cannot guarantee having all the information it will need - but it will let you dynamically assign authors based on a field in the data source.

In the “Other settings” section of the configuration you can set the following options:

Default Author

This will allow you set the default author that entries will be assigned to.

Author

This will give you a pull-down list of the available fields in your data source. If you choose an option here, the module will attempt to find a member that matches this field and assign the entry to it.

Author Field Value

DataGrab can look in the following fields to see if the Author matches an existing member: member id, screen name, username, email address

An XML file

For this example, the path will be

An RSS feed

The path for this file would be

.

There’s a fantastic line-up of speakers to enjoy (Carl Crawley, Joel Bradbury, John Henry Donovan, Leslie Flinger, Stephen Lewis, Lodewijk Schutte, and Garrett Winder) and a great looking venue.

The conference is being organised by Ian Ebden of DesignKarma and tickets are on sale until Friday 19th August.

You can follow the latest news using the twitter hashtag #eeuk11.

Hope to see you there!

The first import

Firstly, we’ll import our initial set of data into the News channel. Here we are using a CSV file called dummy.csv.

Configure the import

We’ll just fo a simple import, setting the Title, Entry Date and Body fields of the entry

Results of the first import

We now have 3 new entries in the News channel.

The second import

In this import we’ll use an XML file called dummy.xml.  We want to relate these entries to the entries we have just imported (using the Playa field), so the entries in this file all have a field that contains the titles of the entries that they should be related too (eg, Two)

Configure the second import

As with the first we have set up a simple import, but with the addition of the Playa field.

In the Playa field we choose which field in the data source describes the relationship. In this case the field ‘related/title’ contains the name(s) of the titles from the first import.

Note that you can use another field other than the entry’s title to match on. You could use the entry’s id or another custom field if that is a better option (eg, a Product’s unique identifier)

The final results

When we run the second import it creates new entries and adds any Playa relationships it finds.

Upcoming courses include:

• BrilliantRetail & Datagrab
• Advanced ExpressionEngine 2.0
• ExpressionEngine 2.0 for Beginners

Check out the testimonials here.

Here is an example template to display a user’s folders:

The key part of this is the

added to the asset’s link. This creates a default icon class and then an additional class that varies depending on the asset’s extension.

We can then, along with a suitable icon library (I’ve used famfamfam’s silk icon set), use the following CSS:

to give a different icon based on the asset’s file type/extension.

Several new datatypes have already been developed, and until a better place is found to store them, a forum has been created where they can be downloaded and basic support provided.

DataGrab Datatype repository

Datatypes for Excel, SQL, MovableType and twitter have been added at this stage.

If you have any requests for a new datatype that you think will be useful to yourself or the community, or if your project has need of a bespoke form of import, then please get in touch.

As tumblr is a hosted service it means you don’t have to worry about managing the system yourself, but when there are problems you are reliant on tumblr to fix them and to protect you from any data loss.

If you are using ExpressionEngine, then DataGrab can help provide some peace of mind.

All your tumblr data can be accessed using the tumblr API, and returned as XML, by visiting http://mysite.tumblr.com/api/read (where mysite is replaced by your tumblr site’s name).

Then, using the XML import type in DataGrab, you can import this data into your EE channel.

You could import all your data into a single channel storing the @type field to differentiate between the different types of posts that tumblr allows.

Alternatively, you could do separate imports for each type using the API’s GET parameter ‘type’ (eg, http://mysite.tumblr.com/api/read?type=photo would access just the photos in your tumblr blog).

In the next part of this article, I’ll introduce a specialised tumblr import type and explain how it might be used.

The plugin is similar to the Magpie RSS/ATOM Parser plugin but can handle all XML formats, not just RSS and ATOM, so it is ideal for displaying REST-based API results.

For more details see the EE Feed Parser product page.

Each data type is contained in a folder and contains at least 1 file. For example, a twitter data type could have the following structure.

Note: the datatype file has the format dt.[folder_name].php. I’ve prefixed my datatypes with ajw.

To install this datatype, simply copy the folder to your system/expressionengine/third_party/ajw_datagrab/datatypes folder:

Now when you go to the DataGrab module you should have an extra option in the “Create new import” menu.