Data Import

With jmb.core is possible to import data in your project database by processing a xls file or csv line by line and than create or edit your models.

Create do method

First step is declare a class that extends XlsFile or CsvFile (which inherit from Sheet) in which you must overwrite do method where you can process the file line by line. do method use follow parameter:

row

xls row object. Row element can be taken with row.<column_name> where <column_name> is the unicode and case-sensitive head column (at first row).

kw

dictionary of mapping column->model.field (if fields_map and relative model is set kw is automatically fill).

j

row number.

Process cell by cell

Follow code is an example of how to override the do method for import an xls file:

from jmb.core.admin.import_data import ImportData

class MyImportData(XlsFile):
    def do(self, row, kw, j):
        customer = Customer.objects.create(company_name=row.Company_name, vat = row.Vat)

Process with a mapping

If you are importing an xls file where the column coincide with all or with parts model field, you can map column head with model field like this:

from jmb.core.utils.data_import import XlsFile

class MyImportData(XlsFile):
    fields_map = {
        'code' : 'ancode',
        'description' : 'andescr',
        'foo' : 'anfoo',
      }
    def do(self, row, kw, j):
        Contact.objects.get_or_create(code=row.ancode, defaults=kw)

x = MyImportData('anagrafiche.xls', auto=True)
x.read()

In fields_map dictionary the keys are model field name and the values are column head (the space are replaced with _).

Define a custom form

jmb.core provide a default form that already contains a simple form with upload widget. This step is only necessary if you want customize your import form and use cleaned_data in do method. Custom form is simply a form that extend ImportDataForm:

from jmb.core.admin.forms import ImportDataForm

class MyImportForm(ImportDataForm):
    """
    This class is not required, if you not declare a custom form you will use default form ImportDataForm.
    """
    import_option = forms.CharField(
        max_length=50,
        required=False,
    )

Form values will be available in your XlsFile extended class as opt instance attribute

from jmb.core.utils.data_import import XlsFile

class MyImportData(XlsFile):
    def do(self, row, kw, j):
        import_option = self.opts['import_option']
        ....

x = MyImportData('import.xls', auto=True)
x.read()

Define a view

The view must be extend ImportDataView:

class MyImportView(ImportDataView):
    """
    Custom view for import file, for a proper configuration you must set:

        import_class = MyImportData

    where MyImportData is a class that define a do() function.
    If you can set (but not is required) a custom form to use for import:

        form_class = MyImportForm
    """
    import_class = MyImportData  # Mandatory
    form_class = MyImportForm    # Not Mandatory

Configure url

Last step is define a step that call your MyImportView:

urlpatterns = patterns(
    ...
    url(r'import_data/$', views.MyImportView.as_view(), name='import_data'),
)

Use a command

An alternative to form + view solution is a command solution. Below is a simple example of how to create a command to import a customer:

class ImportCustomer(XlsFile):
    def do(self, row, kw, j):
        customer = Customer()
        customer.company_name = row.Company_name
        customer.vat = row.Vat
        customer.save()

class Command(BaseCommand):
    option_list = BaseCommand.option_list + (
        make_option('-f', '--file', dest='filename', help='Read file in argument', default=None),
    )

    output_transaction = True

    def handle(self, *args, **options):
        filename = options.get('filename')
        if filename is not None:
            plants_to_import = File(open(filename, 'r'))
            import_data = ImportPlant(
                file_contents=plants_to_import.read(), auto=True)
            import_data.read()
        else:
            print "please indicate file to import with -f <file_name>"